2. Evaluation of SME development projects, programmes and policies
2.4. Policy level evaluations
SME development policies. In case of policy evaluation the subject of the assessment is an entire policy field including its aims, strategy and its instruments. In particular, SME development policies can be characterised by the range and volume of services supported by them, by the portfolio of preferential credits and subsidies provided for small enterprises and by all accompanying legislative, political and institution development measures.
Template of policy evaluation reports. An important methodological decision is the choice of the structure of policy evaluations, i.e. the definition of its title and of its subtitles. Choosing the template of the report is an
important decision because the evaluators demonstrate by this act their opinion about what the real building blocks of the respective policy are. Frequently, the structure of evaluation reports follows a template that has been determined by some guideline issued by the donor agency. Evaluation reports of policies can be structured according to the following principles.
• By policy aims. If the evaluators want to structure their report according to the declared aims of the programme, the report takes one-by-one the aims of the particular SME development programme as they were previously defined in a key document. The attainment level of these aims may be analysed by responding to the following questions. (a) What are the subordinated aims, objectives, preferences into which the main aims can be broken down? (b) What are the instruments that have been utilised for realising these aims and subordinated aims? (c) What are the circumstances and conditions that facilitate and what are the obstacles of reaching the given aims? (d) What results have been obtained in the given period of time? (e) What are the tasks to be solved next.
• By policy instruments. In this case the report evaluates one by one the following policy tools: projects, grant schemes, credit schemes, regulations, measures of institutional development, etc. Frequently the evaluators want to structure the report according to the tools of the examined policy. In this case different chapters of the report are devoted to different implementing institutions, to different steps of legislation or to various co-operation forms with entrepreneurial interest representations, to various support schemes etc. 25 Compared to the other approaches, this approach is more comfortable for the evaluators, because most source documents are devoted to the implementation of some subsidy scheme or grant scheme or credit scheme. Similarly, it is easy to find interview subjects who are knowledgeable in the provision and administration of some of the above mentioned schemes.
• By evaluation criteria (such as relevance, efficiency, etc.). In this case the policy is treated as if it was a single project and it is being evaluated with the methodology described in the previous chapter.
• Combined structure. It is also possible to structure the report according to a combination of the above possibilities.
The following boxes demonstrate the application of some of the above mentioned template alternatives.
Example for policy evaluation structured by policy aims:
Table 3.22. Box 6.
Evaluation of the implementation of the decisions of the European Charter of Small Enterprises in the West Balkan states
In 2003 Albania, Bosnia-Herzegovina, Croatia, Macedonia, Serbia and the UN governed province of Kosovo have signed an agreement with the EU aimed at the development of small enterprises. The content of the agreement was to realise the decisions as put down in a key document called „Charter for Small Enterprises" a. This document has defined the following goals for developing the SMEs of the region:
1. Education and training of entrepreneurs
2. Cheaper and simpler establishment of companies 3. Better legislation and regulation
4. Developing the labour force possessing the necessary professional knowledge 5. Electronic taxation and electronic company registration
6. Better utilisation of the advantages of the Common Market
7. Improving the fiscal environment and accessibility to finance for SMEs
8. Improvement of technological capabilities of SMEs
25For example: [UNIDO 1996]
Evaluation of the implementation of the decisions of the European Charter of Small Enterprises in the West Balkan states
9. Spreading o successful e-business models and improving the quality of enterprise support 10. Stronger and more efficient representation of entrepreneurial interests
In 2006 a comprehensive evaluation was applied using an up-to-date methodology in order to assess the small enterprise development policies of these countries and in particular their results in achieving the above aims during the years 2003-2006 in the countries of the West Balkan area. b
The process of evaluation was organised as follows. In the first wave the West Balkan states have prepared self evaluating national reports by using a template structured by the above 10 aims. These reports were forwarded to independent consultants who have assessed these reports and attached a preliminary numerical rating to the attainment level of each of the 10 aims in case of each participating country. Finally, the governments of these countries were given an opportunity to comment upon the results of the preliminary rating. The ratings were finalised by experts of the European Community and other international organisations, those who had ordered the making of the report.
a [EC 2000]
b [EC-OECD 2007]
The structure of the evaluation report often simply follows the structure of the key document of the programme evaluated. This is frequently the case if the evaluation precedes the implementation of the programme, i.e. in case of ex ante evaluations. Here the overriding aim of the evaluators is to demonstrate the expected results and impacts of the evaluated SME development measures.
Table 3.23. Box 7.
Evaluation of the SME support measures of the Economic Development Operative Programme for Hungary of 2006
The Economic Development Operational Programme a is one of the executive documents of the New Hungary Development Plan b. This key document is a plan to spend a subsidy portfolio worth 2 and a half billion Euros of supporting projects. The document defines, for the period between 2007 and 2013, the structure of the European Community’s co-financing supports which can be used for enhancing the competitiveness of the Hungarian economy. The document defines a series of development priorities. To each of these priorities the document devotes a specific chapter and allocates a specific sum to it.
A particular development priority serves the development of SMEs in an explicit way. The name of this priority is: „Priority No. 2: A complex development of enterprises, with special respect to SMEs".
The Economic Development Operational Programme document underwent, before being accepted, a so-called
„ex ante evaluation, in order to obtain answers to the following questions: c
• Is the aid delivery strategy defined in the Programme able to respond to the major challenges facing the regions of Hungary and the sectors of the economy?
• Is the document in compliance with the relevant policies of the European Union , the Hungarian government and the regions of Hungary ?
• Are the goals and priorities defined in the Programme clear and attainable?
• What will be the effects of achieving these aims in quantitative terms?
• Does the document contain adequate indicators?
• Are appropriate executive orders, institutions attached to the Programme ?
In order to respond to the above questions, the evaluators have assessed the programme documents and made
Evaluation of the SME support measures of the Economic Development Operative Programme for Hungary of 2006
interviews with officials of the aid delivery organisations as well as with experts working in the governmental apparatus.
The text of the evaluation report exactly follows the structure of the evaluated document, in other words its chapters tackle the chapters of the Economic Development Operational Programme one by one.
In case of „Priority No. 2: A complex development of enterprises, with special respect to SMEs", both the Operational Programme Document and the Ex Ante Evaluation document analyse the following measures:
• Supporting technological modernisation of enterprises
• Creating working places in disadvantaged areas by way of technological modernisation
• Developing the organisation of companies by spreading up-to-date management methods
• Introducing quality systems and supporting the application of standards
• Introducing e-commerce and other electronic services.
The document analyses the above fields of intervention against the following criteria:
• Are the target groups of the planned support and the professional content defined concretely enough in the document?
• Will the support measure bring significant effects in terms of development policy, employment, the attraction of capital, and innovation?
• Is the support measure scheme tailored to the needs of micro-, small and medium size companies?
• Is the support scheme designed to a large quantity of smaller projects, or rather a fewer number of more costly and more complex projects? Which option is better for the promotion of the effectiveness of the support?
• What conclusions can be drawn from the operation of analogous supporting systems?
a [GOP 2006]
b [UMFT 2007]
c [GOP EX ANTE 2007]