Follow-up

Eventually, four years later, in 2005 Croatia has partially harmonised its legislation with the New and Global Approach directives, and LVD was transposed as a part of these measures. The country has adopted legislation aiming at transposing the Acquis on lifts, machinery, gas appliances, low voltage equipment, pressure equipment and simple pressure vessels. However, at the end of 2007 further amendments, horizontal legislation and institutional development of notified bodies still were needed to achieve full alignment with the Acquis.

Chapter 5. 5. Conclusions and recommendations

The status of evaluation and impact assessment in development policies. Evaluation and impact assessment efforts are applied social research activities which have been adapted and institutionalised according to the needs of donor organizations and public agencies responsible for financing and implementing development policies. For this reason, evaluation and impact assessment efforts must take into consideration (a) the principles of research design in social science, with special respect to methods of data collection and inference (b) the particular organizational traditions of the donor organization or government agency which finances or implements the assessed development measure (c) the nature, aims and instruments of the particular policy area, with special respect to the beneficiary target group.

Planning and implementing evaluation and impact assessment projects. Evaluation and impact assessment efforts should be embedded into a wider managerial and administrative process: these efforts should be part of the policy debate and the resulting studies should lead to policy change. Evaluation and impact assessment activities should be organized as projects and should be planned in advance. Policy makers and donors responsible for planning SME development initiatives should facilitate the feasibility of evaluation and impact assessment efforts by rendering the policy initiatives as transparent as possible and by committing the resources necessary for performing the evaluation or impact assessment. Evaluators must collect data about (a) the conditions prevailing in the SME target group before the measures were taken (baseline data) (b) and about the impacts of the measures during and after the implementation of the measures. Evaluation and impact assessment efforts should be based on appropriate research design, which must be prepared before the measures are taken.

The research design which has been developed during the planning phase, should be the basis of any data collection and analysis during the implementation phase.

SME development context. Evaluation and/or impact assessment projects should start with the careful analysis of the assessed initiative/measure. In particular, the nature, targets, intended outcomes, target groups, instruments and professional content of the initiative should be clarified, with special respects to its novelty features as compared with previously existing SME development measures. Moreover, the details of the implementation of the planned measure must be clarified, with special respect to the institutional framework of its implementation.

Focusing on SME development instruments. If SME development measures are to be assessed, the research design of evaluation and impact assessment must be tailored to the specific instruments applied on behalf of SMEs. These instruments may be (a) the introduction of enterprise-friendly regulations facilitating business creation, employment and the access to permits and markets, (b) financial assistance (e.g. microcredit provision, tax relief or subsidy for start-ups), (c) supported business development services (e.g. training, education and provision of advice and consultancy in the field of general entrepreneurship, marketing, export, technology, e-business, in developing business organization or business co-operation patterns such as subcontracting and outsourcing) or (d) institution development (e.g. the creation of science parks, incubators, enterprise zones, cluster organizations or other types of business networks). Each component of the research design of the evaluation and impact assessment effort must correspond to the peculiar features of the above mentioned SME development instruments, of the project organization and of the target group. In particular,

• the research question must be relevant to the assessed measure and must be formulated in terms of the assessed public policy;

• a theoretical hypothesis should be formulated about the impact mechanism (in case of impact assessments) or about the success criteria (in case of evaluations) of the assessed measure;

• the strategy for data collection and sampling must correspond to the organizational framework of project/programme implementation, and must reflect the willingness of the target group members to respond to survey/interview questions; and finally

• when making inferences, the collected data and the applied analytical methods should prepare the grounds of supporting or falsifying the above mentioned theoretical hypothesis about the impact mechanism (in case of impact assessments) or about the success criteria (in case of evaluations) of the assessed SME development measure.

and impact assessment of SME development measures. Within each country, the rates of enterprise creation, growth and bankruptcy differ markedly across regions and localities. These rates are determined by the natural endowments and entrepreneurial traditions of the region, and by local factors of demography, labour and education. Small business development can be an important driving force in developing disadvantaged or deprived areas. However, evaluators and impact assessors must always take into consideration the specific impediments to entrepreneurship in these communities such as the lack of demand on services, lack of experiences, an under-developed culture of co-operation and a high reliance on social benefits. Projects implemented in localities lagging behind the national average may need a special design and delivery mechanism. In well endowed regions (e.g. in areas with many tourist attractions) the impact and success of SME development measures must be evaluated by properly taking into consideration the local context.

Distinctive features of impact assessment and evaluation. Both evaluation and impact assessment are empirically based comparative analytical exercises used for improving development policies. However, the researchers must establish a clear difference between

• impact assessment on the one side, which should be a value-free activity aimed at finding causal relationships by demonstrating the methodological rigor of positivist analysis and objective social science and

• evaluation on the other side, which cannot be value-free, because the shorter and longer term interests of various stakeholders (e.g. donors, project managers and beneficiaries) can be confronted only if the evaluators rely on certain principles of good governance, and ethics.

Impact assessment is essentially the demonstration of causality between the assessed measure and its impacts on the companies of the target group, a verification of impact mechanisms and the ruling out of alternative explanations. Impact assessments should include an analysis of the counterfactual situation, i.e. the analysis must respond to the question: what will happen (or what would have happened) to the target group if the measure was not (would not have been) taken. Although the counterfactual scenario cannot be observed, indicator values about the counterfactual scenario can be obtained with rigorous methods, e.g. by the following experimental design: by randomly assigning support to a sub-group of eligible applicant SMEs and by randomly denying support to another eligible sub-group of applicant SMEs. However, in most cases of impact assessment, due to lack of data and scarce resources, the counterfactual scenario is assessed by less reliable qualitative methods, e.g. by asking beneficiaries in in-depth interviews about "What do you think, what would happen to your business under the impact of the planned regulation" or "What would have happened to your business if you had not received the subsidy". Impact assessment is a comparative effort in that sense that the indicator values about the actual observed impacts of the measure must be compared with the indicator values describing the impacts under the counterfactual scenario.

Identifying impacts of SME policy measures. In typical circumstances it is extremely difficult if not impossible to isolate the net impact of small business policies, because there are various macroeconomic policies that can influence of entrepreneurship and small business performance in a country. Decisions of public policy on taxation, consumer protection and environmental regulation, immigration and on the institutional framework and procedures of enforcement may strongly influence the birth, survival and growth of SMEs. It is the task of SME policy makers to identify these policy influences on entrepreneurship. Evaluation and impact assessment projects should be key instruments in identifying these interactions between various policy areas and in developing co-operation patterns with other departments of the government.

Evaluation efforts qualify the SME development measure; they determine the value of a project, a programme or a policy, by delivering value judgements about its design and implementation, by analysing the course of SME development actions according to previously defined evaluation criteria (e.g. relevance, efficiency, effectiveness, etc.) Evaluation is a comparative effort in the sense that the actual observed or calculated impacts of the measure must be compared to the expectations of the evaluators which in turn are based on the aims of the measure and on the results of analogous, comparable projects or programmes.

The standards of evaluation and impact assessment activities have been defined by donors and governments, the recommended or prescribed methods are part of these institutional cultures. The EU alone has several dozens of evaluation and impact assessment guidelines and methodologies. Policy makers and donors should continuously work on the establishment and institutionalisation of unified, compatible standards, guidelines and methodologies for obtaining feedback about policy measures, in particular about those affecting SMEs. These standards should rely on good evaluation and impact assessment practices. Researchers should be encouraged to

prepare comparative reviews of evaluations and impact assessments of which the subjects are analogous, comparable SME development measures in various countries.