The type of data to be collected

Evaluations and impact assessments are empirically based efforts. Research designs are thoroughly influenced by the choice of information sources, by the availability of data and by the strategy of collecting the necessary data.

12Own compilation, based on [Oldsman – Hallberg 2002].

2.3.1. Qualitative studies

In most research situations the access to qualitative data is easier and cheaper than to perform a questionnaire based quantitative survey among beneficiary companies. In case of studies are based on qualitative data, inference is based on the analysis of aid delivery documents, on interviews made with owners and managers of affected companies, with officials of regulatory agencies, or conducted with project managers of the aid delivery process. Causal inference is frequently based on the comparison of ―cases", i.e. on the secondary analysis of previous studies made in comparable countries where the same interventions have not been taken, or by analysing the effects of previous comparable interventions in the same country or in other regions. Quite frequently, qualitative studies made by international organisations compare countries: in such cases the implementation and reception of similar or analogous interventions – e.g. subsidy schemes, regulatory reforms, campaigns, etc. – is being compared. In qualitative studies the issue of sampling arises as the proper selection of (a) cases to compare, (b) documents to analyse and (c) interview subjects to visit.

Interviewing some representative members of the target group. Most impact assessment studies rely on a small scale sample of in-depth interviews made at companies affected by the examined policies. Sampling strategy as a rule is restricted to having a quota of at least one or two company of each type in the sample. For example, researchers may collect interviews from companies of various sizes, sectors and legal forms, and firms working in various regions and settlement types should also be represented. The collected empirical material contains important information about company characteristics, strategies and behaviour, about the awareness of companies of the examined policy intervention and their responses to the examined measures, about the opinions of experts and stakeholders about the institutions responsible for SME development. The results of these in-depth interviews are then contrasted with each other, classified by explanatory variables and aggregated. Inference to causal statements is made by asking the interviewed experts, regulators and enterprises about the counterfactual: what is their opinion about ―What would happen (or what would have happened) if the intervention did not take place (had not taken place)". While this method is relatively cheap, flexible and feasible, it can be easily biased by subjectivity or by lack of skills on the side of the interviewers.

2.3.2. Sampling strategies for quantitative impact evaluation

Quantitative impact assessments are prepared with the ambition to make statistically reliable statements about the relation of the cause (i.e. the policy intervention) and its consequences: about their impact exerted on companies. These efforts are always based on business surveys.

The sampling strategy of rigorous quantitative impact assessments of SME development measures consists of the following stages: (a) statistically defining the enterprise population that is the target group of the measure (b) selecting a sample of enterprises affected by the measure and (c) selecting a comparable sample of other companies, that have not been exposed to the measures. If possible, data collection from "treatment" and

―control" companies is repeated both before and after the intervention – this is called a ―before-after design".

The literature of this body of policy research often uses the language of experimental design used for medical research: the companies exposed to the intervention are called the ―treatment group" and a comparable group of companies that has not been exposed to the intervention is called the ―control group". In such studies the statistical inferences result in impact statements that refer to the ―treatment group", but these statements are on based on a comparison between the ―treatment group" and the ―control group".

However, due to lack of data, lack of resources or because of ethical reasons, it is not always possible to involve a control group into these research efforts. In such cases the inference about the causal relationship (i.e. the connection between the measure and its impacts) is weaker.

The following sampling strategies have been routinely applied in quantitative impact assessment designs. These designs have been ordered by the decreasing reliability of the causal explanations that can be based on them. On the upper end of this scale, the random selection of both the treatment group and the control group is considered to yield the most credible impact statements. The other extreme is when there is no control group at all: although such a sampling strategy is perfectly justifiable in case of descriptive studies, but any inference to impacts or causal relations can be made only if it is strongly reinforced by additional qualitative information which has been gathered independently from the quantitative data collected by the survey.

(1) Random assignment. This experimental design is used to measure the observed results of the intervention by comparing (a) a random selection of enterprises having been exposed to the intervention with (b) a random selection of enterprises not having been exposed to it. This sampling strategy is the so-called ―randomist"

approach. Although this approach leads to impact statements that have very high reliability, there are some

ethical, methodological and feasibility considerations against its use. Since regulations and subsidies have well defined target groups, it is not feasible or not ethical to experiment with companies by randomly choosing a

―treatment" and a ―control" group. Researchers are not in the position to decide, by way of random choice, which enterprises should receive subsidies or which companies should be exempted from the force of a regulation.

(2) Matched control group. In this case the control group is not randomly selected, but is constructed to be as similar as possible to the group affected by the intervention. Similarity is attained by selecting a control sample of companies of which the composition by size, sector, region and other important explanatory variables are identical with that of the affected group.

• In most cases this is ensured by using company quotas according to some previously selected explanatory variables such as sector, size and region.

• Another approach to matching is called ―matched area comparison design". In this case the outcomes in the pilot area – i.e. where the intervention is introduced – are compared with a control group that is chosen by picking enterprises of another, comparable, possibly similar area (e.g. region) where the intervention is not implemented.

• In more sophisticated cases a better match can be achieved by computing the so-called ―propensity scores".

Propensity score is the conditional probability that a company will be supported (―treated"), given the set of its values on the explanatory variables. Each company in the supported (―treated") group is matched to a subject that has a similar propensity score in the untreated group. The control group consists of the non-beneficiary companies selected by this method.

(3) Treatment group and control group selected by convenience . In the practice of SME development policy evaluation, company samples are very often determined by the willingness or capability of companies to respond to the questions of evaluators.

Examples of this approach are those research designs where the responses of a group of responding beneficiary companies are compared to the responses of those applicants who were applying unsuccessfully to the same subsidy scheme. This research design is somewhat biased for two reasons. (a) Voluntary respondents do not necessarily represent properly the target group. (b) Companies that have applied to the subsidy but were rejected by the selection committee have performed weaker during the application process and are not comparable to the beneficiary group. Consequently, by aggregating their responses the researchers cannot directly respond to the counterfactual question ―What would have happened without the intervention?" ¹³

(4) Research design without control group . In many situations of SME research interviews are made with - or questionnaires are collected from - only the companies affected by the intervention. In other words, the research design involves no control group. Certain causal statements with weaker reliability can be formulated even in such cases, by comparing internal sub-groups of the target group. This type of inference can be facilitated by the following sampling strategy: a sample with the possibly highest variability should be compiled. A high variability can be attained if the sample includes companies from various sectors, regions, size classes and levels of exposure to the (planned) intervention. ¹⁴

For example, let’s suppose that some previous qualitative information supports the hypothesis that the positive impact of the examined intervention is proportional to the size of the beneficiary company. If the empirical findings show that the expected positive changes do not occur among small companies, but do occur among medium sized enterprises, and even more so among bigger companies, then this might be interpreted as a reinforcement of the hypothesis.

2.3.3. Compromises and pitfalls in sampling

The range of data to be collected depends strongly on the financial resources available for the evaluation or impact assessment project. Research projects can save significant costs by applying weaker principles to sampling strategy. Such compromises might be unavoidable, but in such cases the report must (a) unanimously point out the lower reliability of the causal statements that it has arrived at and (b) should refer to the direction of the possible bias. The following methodological pitfall demonstrates how poor sampling can lead to biased impact statements.

13 E.g. Case Study C in this document.

14 E.g. Case Study A in this document.

the members of the target group of the intervention. This mistake is committed if the researcher examines the change that was brought about by a previously implemented intervention only in that group of companies in which the positive effect of the intervention was able to exert its influence.

Table 2.1. Box 1.

Example of selection on the dependent variable Bankruptcy regulations in Hungary

An example of bankruptcy research should illustrate the consequences of selecting the sample on the dependent variable. In the early 1990s in Hungary the transformation crisis has lead to an unusually high number of indebted companies. Long chains of non-payments have evolved, whereby each company in the chain was indebted to the next one. One of the causes of this anomaly was that the country lacked a well functioning bankruptcy law. Consequently a series of laws and regulations were issued with the aim of ensuring orderly bankruptcy procedures, giving chances of recovery to the indebted companies, but at the same time defining fair rules of liquidation in order to assure that the creditors – their clients, the state, other companies and banks - will be compensated by the assets of the indebted companies.

Two years later an evaluation was made about the interventions implemented in favour of those companies that got into troubles. The evaluation was based on a survey among surviving companies and among companies being still under liquidation procedure at the time of the survey. Only a handful of those companies were included in the sample that have been already liquidated between the bankruptcy regulations and the survey date. The reason for this omission was that at this time only a few managers could be reached who were able to speak about the liquidated companies.

The evaluation report clarified that (a) the sample of the survey had been selected on several of the dependent variables, i.e. the survival of companies and the length of the liquidation procedure; (b) the impact statements were somewhat biased, because the results of the examination could not be generalised to the full population of enterprises that were under the impact of the law.

The consequences of the lack of quantitative data. Researchers preparing quantitative impact assessment studies always face resource and the data constraints: surveys are expensive and data obtained from official statistics are in most cases outdated or irrelevant. In most research situations the group of enterprises affected by the examined intervention constitutes an aggregate of very specific character which does not correspond to any one of the widely used statistical categories of economic sectors, company size classes or geographic regions. For this reason, many researchers are compelled to adopt ad hoc methods by combining the use of qualitative data with the available quantitative data obtained from statistical offices, by doing a secondary analysis of previous surveys and by doing interviews by using a project-specific questionnaire.

In most cases it is the lack of available data that reduces the applicability of quantitative evaluation research methods ¹⁵ rather limited in their actual working practice. Due to a chronic lack of statistical data, impact assessments and evaluations related to the entrepreneurial sector do, in practice, depend on document analysis, on administrative and company interviews which are complemented by business surveys and by official statistical data in favourable cases only. Most guidelines of existing impact assessment and evaluation cultures do not count on the availability of relevant time series or survey data.