2. Evaluation of SME development projects, programmes and policies
2.1. Project and programme level evaluations
Donor organisations, whether public or private, collect a wide body of information in order to decide, which project they are going to support, and likewise, these organisations must receive reliable feedback about the programmes and projects that have been financed by them. Both the World Bank 910 and the European Union 11 have implemented strict policies about the mandatory preparation of a priori and a posterior evaluation of supported projects and programmes. These organisations have published guidelines on how to collect data for evaluation and define those criteria against which the projects and programmes should be evaluated. The application of some of these guidelines has a mandatory nature, while others are issued as recommendations to be followed by evaluators.
Enterprise support projects aim to enhance the competitiveness and the creditworthiness of companies or to develop a particular function of the companies such as innovation, human resource management or marketing.
For this reason, the evaluation of enterprise support projects has much in common with traditional business consultant activities where the competitiveness or the creditworthiness of SMEs [bib_7] is analysed or various functions of companies are assessed [bib_8]. However a major difference between such descriptive studies on the one side and the evaluation of subsidised projects on the other side is that while the genre of evaluation always involves the elements of causation and impact, this is not necessarily the task in case of diagnostical analyses.
9[WB 2007]
10[WB 2003]
11[EAR 2005]
What is a project. It is reasonable to take a look at the definition of the notion ―project", since in most cases this is the entity which is evaluated by the analysts. Moreover, the process of evaluation is just one of the phases in the life cycle of a project, i.e. the evaluation of projects is embedded into the wider activity called ―project cycle management". Within the aid delivery methodology of the European Union 12 the concept of a project is defined as a system of activities, which has a clear and well defined purpose, a planned duration, a budget, target groups, furthermore it has co-ordination-, management- and finance mechanisms. As a rule, projects are selected as building blocks to accomplish a political programme of a government or an international organisation.
What is a programme. According to the conceptual framework used in EU documents, the notion of the programme has a somewhat wider meaning than that of a project: it is a set of homogenous projects co-ordinated by a particular policy aim. In case of programme evaluation the subject of evaluation goes beyond the individual projects and a wider portfolio of support projects having common objectives or being implemented by the same institution is evaluated in a comprehensive way. For example, in the practice of the EU aid delivery, the so-called operative programs have to be evaluated before, during and after their realisation. Within the policy area of SME development a wider enterprise-supporting programme can consist of several projects, each of which offers subsidised training or consultancy services for eligible small enterprises. In this framework, eligibility can be defined according to the overarching aim of the particular policy which is served by the programme, e.g. in case of innovation policy, only innovative SMEs can be eligible to receive these subsidised services.
In the standard aid delivery methodology of the European Union the criteria of project evaluation are as follows.
• The relevance criterion is appropriate for the analyst to ask, whether the design of the project and its implementation corresponds to the challenges facing the target group, whether it is harmonised with the relevant goals of public policies, and whether the project can be / has been properly adapted to the existing institutional structure.
• The efficiency criterion is needed to assess whether the project adequately transforms the existing resource inputs to outputs. In particular, evaluators ask here about the quality of time management, cost efficiency and the level of inter-organisational and intra-organisational co-operation.
• The effectiveness criterion needs to be applied in order to assess the contribution of project outputs to the direct goals of the policy. In particular, in this phase of the evaluation, researchers have to ask about the actual achievements of the project.
• The impact criterion is appropriate for assessing the contribution of outputs to a wider range of economic and social aims, including local and regional externalities, such as organisational learning or the improvement of entrepreneurial culture.
• The application of the sustainability criteria is needed to check the long term viability of the results of the project, in particular the extent of its donor dependence.
Table 3.1. Box 2.
Evaluating the Phare Microcredit Programme in Hungary
In Hungary , following the political changes of 1989, a wide range of small enterprise development projects were launched by various public bodies and non-profit organisations. These projects and programmes were frequently co-financed by the European Union , directly or indirectly. All of them were several times evaluated with the standard evaluation methodology of the EU. A major result of these subsidised projects is a well functioning network of local enterprise development foundations which were established by the end of the 1980s. a Subventions of the EU have financed the launching of micro-credit and other preferential credit schemes as well. The Phare Microcredit Scheme was called into being by the fact, that one of the major barriers for development of the young Hungarian small enterprise sector was the undercapitalisation of these firms b. In 1992 the Microcredit Fund was established and soon the county-level c enterprise-development foundations started to disburse these credits for start-up and growth-oriented small enterprises. This preferential credit was given to those promising small firms, which were not creditworthy enough to receive credits from
12[EC 2004c]
Evaluating the Phare Microcredit Programme in Hungary
the commercial banks. During the existence of Hungarian micro-crediting a major question was whether the programme would / will be viable in real market conditions or it is only prospering with governmental support, in a ―donor-dependent" way. d During its existence the Micro Credit Scheme was several times evaluated [bib_9].
In 1999 the Microcredit Programme was evaluated by independent consultants [bib_10]. Their report begins by demonstrating the Phare Microcredit Scheme as a financial product: its interest rate, guarantees, maturities, the financial sources of this credit scheme, which beneficiaries are eligible to receive microcredit, what kind of appraisal mechanism is used to select beneficiaries. Statistical information follows: the volume of the disbursed and repaid credits, the composition of the beneficiaries by sector and county.
The core of the evaluation report contains the opinions of the evaluators according to the selected evaluation criteria, expressed in numerical scores and justified in text. The 5 evaluation criteria were given in advance by the donor organisation and it was the task of the evaluators to transpose these criteria into operational
evaluation questions. The main evaluation questions were as follows:
• Relevance . What kind of sources other than microcredit were available to satisfy the needs for capital of SMEs? Were the parameters of the microcredit scheme (e.g. the interest rate and the limit of disbursable credit for one beneficiary) relevant with the goals of subvention and with the needs of the SME sector? Was there sufficient demand for the microcredit offered?
• Efficiency . Were there delays in the implementation of the programme? If yes, why? How much time did it take for the beneficiaries from their application to the disbursement of the credit? What was the level of transactional and handling costs of micro credit, and from which sources were these charges covered? Do the regional enterprise-support foundations charge reasonable price for managing the programme? Is there a good co-operation between the various organisations active in the implementation of the programme?
• Effectiveness : Has the programme succeeded in reaching the originally targeted SME sector? What types of investments, developments were accomplished which could not have been financed without the micro credit? Has the microcredit programme offered equal opportunities across the various sectors of the economy, such as services, trade, industry and agriculture? Did SMEs in all of these sectors receive sufficient micro credit? Was microcredit provision decentralised across the regions?
• Impact: Has the microcredit scheme succeeded to attract further capital from other sources to finance the SME sectors? Did the micro-credit scheme stimulate the commercial banking sector to provide credits for SMEs? Did the programme increase the prestige and legitimacy of the county–level enterprise development foundations? Did the availability of the microcredit scheme motivate SMEs to reduce their participation in the informal economy? Did the scheme strengthen the local ties; facilitate the local ties of SMEs?
• Sustainability. What are the characteristics of credit repayment and credit default among beneficiary SMEs?
Can the Micro Credit Fund, which is the source of micro credit provision, sustain its operations from the repaid capital and from the interests of micro-credits? Are the selection, control and motivation systems appropriately defined in order to increase the proportion of repaid credits? Is there any appropriate fund involved to cover the respective credit risk?
The answers to the above questions need to be confirmed by available statistics, project management documents as well as by interviews made with project managers and grantees.
a[Bateman 2000]
b[Szirmai 2003]
cThe territory of Hungary is admnistratively divided into 19 counties plus the capital, Budapest.
d[Kállay 2003]
Logical Framework . Several donor organisations such as the USAID and the EU have decided to apply a standardised version of the so-called ―Logical Framework" method to co-ordinate the process of project planning, implementation, monitoring and evaluation. The method compares the inputs and outputs of the project and summarises them in the so-called Logical Framework Matrix, which is developed in the project planning phase and used a major document of project implementation, continuous monitoring and periodical evaluation. It defines the hierarchy of goals and risks of the project, furthermore it specifies the so-called
objectively verifiable indicators, which are supposed to measure the achievements of the project in quantitative terms or prove its success with the help of empirical facts.
The life cycle of projects financed or co-financed by the EU consists of the following phases:
• Programming. In the phase of project identification and specification the representatives of the donor agency and the supported government make decisions about the main priorities, the type of subvention and sort of financing, analysing the viability of the project and the sustainability of its results. This is a planning activity which precedes the selection of the beneficiaries and the launching of the projects.
• In the implementation phase of the project the previously accepted activities are performed. During the lifetime of the projects the supporting organisation obtains monitoring reports which give feedbacks about how the implementation of the project is proceeding. In case of failures the project plan and the implementation of the project can be modified, or in extreme cases the project can be stopped.
• Monitoring is a continuous activity which goes into more detail in terms of feasibility and efficiency than evaluation. Monitoring experts may help the day-to-day activities of project managers by concentrating on the inputs and results of project activities.
• In the phases of preliminary, interim and ex post evaluation independent experts examine the project against the criteria of relevance, efficiency, effectiveness, impact and sustainability. These criteria have been used by the EU during the last decade, but other donor organisations may recommend the use of other, more or less equivalent criteria. The evaluators analyse the economic uses and administrative rationality of the project, assess whether the results are sustainable and collect the lessons to be learned from the project. Depending on their findings, the donors may modify some project instructions, may decide to take corrective actions.
• In the audit phase auditors are controlling whether the relevant laws, the provisions of the support contract, the financial, accounting, purchasing and reporting rules of the subsidy scheme have been observed. Auditors examine the procurement procedures and decide, whether the expenditures were proportional to the purchased products and services during project implementation. While evaluators do not control the legitimacy of project procedures, decisions and payments, auditors do exactly this type of work. The main information sources of audit work are the contract documents and the bills and invoices issued and received during the project.
The main information sources of evaluation are as follows.
• Documents of project planning and management such as invitations to tender and monitoring reports.
Evaluators must always pay attention to account for eventual biases contained in source documents with special respect to withheld information.
• Interviews made by the evaluators with the persons responsible for project design and implementation, such as the employees of co-ordinating and support agencies, representatives of the beneficiary institution and if applicable, the final beneficiaries. Evaluators must always pay attention to account for the hidden agenda of interviewed persons.
• Quantitative surveys. Large or medium sized sample surveys are very informative; in fact the statistical evaluation of the responses can convey the best and most objective picture about the success or failure of a project. However, due to its extensive costs, evaluators can rarely rely on a questionnaire based survey conducted among beneficiaries and with the members of an appropriate control group. Within the questionnaires of evaluation surveys the particular questions correspond to one of the above mentioned five evaluation criteria. The major task of questionnaire design is to operationalise these criteria and to transform them to relevant micro-level questions. Good questionnaires cover all relevant issues of the project, including (a) the evaluated support scheme, (b) the institutionalisation of the support and (c) the characteristics of the target group.
Table 3.2. Box 3
Evaluation of PHARE and Transition Facility subsidies
An example for project evaluation methods is given by the standard template of the PHARE Interim
Evaluation of PHARE and Transition Facility subsidies
Evaluation Reports. In Hungary during the years 2003-2008 several dozens of interim evaluation reports have been prepared about the planning and implementation of programmes and projects co-financed by the PHARE Fund of the European Union and the Hungarian government. Each of these reports was devoted to one single
―sector". In the terminology of PHARE support, a sector is a policy are such as agriculture, SME development, justice and home affaires, environment, etc.
For evaluating the aid absorption capacity of such a sector, a sample of projects is taken and evaluated. The main chapters of each report are as follows.
• Sector Background: Major events that happened in the sector during the time period evaluated, including its institutional environment and the major challenges to aid delivery.
• List of projects, beneficiaries, tables of the allocated and disbursed finances.
• Description of aims, activities and corrective actions listed by projects.
• Evaluation results are sorted as follows: the five criteria (e.g. relevance, efficiency, etc.), and under each criterion the individual projects are evaluated according to the selected criterion.
• Conclusions and Recommendations.
An application of the above template can be illustrated by the evaluation of the following specific project.
Table 3.3. Box 4.
Evaluating an SME-support project for developing e-business services
„E-business as a tool of SME-development" was a European Community co-financed project within the framework of the Hungarian PHARE program, on which the donors have jointly spent 4 million Euros during the time span of the project, between 2003 and 2005. The activities financed for the beneficiary SMEs were training courses for groups of entrepreneurs and customised counselling services for individual firms to support the development of company level ICT strategies. These activities were performed on 20 different locations of Hungary , altogether for about 800 SMEs. The beneficiary SMEs were able to publish their offer on an e-commerce website which was prepared within the framework of this project. These services have provided real added value for the beneficiary companies, because at the time of the project most Hungarian SMEs were not ready to invest into sophisticated e-commerce services or into the relevant training courses under the conditions offered by the open market.
The Sector Background chapter of the document has described the state of ICT applications within the SME sector of Hungary and previous efforts of spreading this technology among small businesses. The most important chapter of the document was the application of the five evaluation criteria on the specific project.
This was operationalised by asking the following research questions in the course of the interviews and document analyses.
• Relevance : Evaluators wanted to know if the project design has met the real needs of SMEs and whether the major issues of entrepreneurial culture and competitiveness were addressed by the project.
• Efficiency : Was the project carried out in due time, with the possibly lowest drop out rate? Was the project successful in establishing good cooperation among the participating organisations such as project managers, trainers, and ICT consultant companies?
• Effectiveness . Was the project successful in reaching a wide enough stratum of SMEs? Did the project create good quality training materials and were these materials published on the Internet? Is it possible to successfully offer, sell, buy and pay products on the e-business website created in the framework of the project?
• Impact. Was the project successful in diversifying the activities of the beneficiary SMEs and their co-operating partners? Did the project enhance the cooperation among the participating firms and those which
Evaluating an SME-support project for developing e-business services are vertically or horizontally were connected with them?
• Sustainability : Following the closure of the project, when support for such activities will stop, will beneficiaries be able to develop their ICT strategies and achieve success in e-business? Will the website created in the framework of the project survive, will it be competitive enough besides the already existing e-commerce websites, and eventually will it be privatisable? Will the training material compiled in the frame of the project be good for use, following the end of the project, too?