Purpose and method of the questionnaire based evaluation

In Hungary the government authorities responsible for innovation policy ²³ have implemented a wide range of innovation related subsidy schemes which have been afterwards evaluated. Small enterprises were found among

21 See the sub-chapter „Methodological aspects of RIA" in this study.

22 [Peredy 2007]

23 During the 1990s: the State Technical Development Committee (OMFB). Afterwards: its legal successors such a sin 2009 the National Office for Research and Technology (NKTH)

the beneficiaries of both the successful and the less successful grants. The next case study describes one of the above evaluations which focus entirely on the innovative activities of SMEs.

In 2000 the Hungarian State Technical Development Committee (OMFB) entrusted a group of independent experts with the task of evaluating the impact of various subsidy schemes that were offered by the Committee as a donor. Three subsidy schemes of the Committee, called „Applied R+D tender", „Competitive products tender", and „Regional innovation tender" with different eligibility and management rules, have allocated support to a wide range of SMEs. The aim was to report, how through these projects the donor organisation has supported SME sector and to evaluate the contribution of the Committee to Hungarian SME development and innovation policy.

The evaluation [bib_17] report consisted of two major parts:

• The institutional overview was based (a) on the analysis of official documents (e.g. the government’s SME- and technological-political documents, the tender documents issued by the donor organisation) and (b) on interviews with the experts of the Committee.

• The small sample business survey relied on questionnaires returned by beneficiaries.

The institutional overview started with the identification of the examined subsidy schemes. The evaluation report characterises the subsidy schemes by describing (a) their dates of implementation (b) the declared aims (c) the administrative procedures of decision making, (d) the financial allocations and disbursements related to these schemes, and (e) the number and other characteristics of the benefited enterprises. The document explains the context of these subsidy schemes within the enterprise supporting system and positions the development of these schemes within the donor organisation.

Following these fact finding exercises, the evaluation switches to its main task, i.e. to give the opinion of the evaluators about these schemes. From the point of view of methodology this means, that a general system of evaluation criteria had to be adapted, applied for the subsidy schemes both (a) in document research as well as (b) during the interviews made. The following list of questions explains how the researchers have transformed the five abstract criteria of evaluation into specific questions concerning the examined subsidy schemes

Relevance :

• Are the evaluated subsidy schemes properly adjusted to the relevant governmental tasks, to the system of goals and methods, as well as institutions, to the corresponding policy fields (SME policy, technology policy), to the innovation activities of small- and medium size enterprises?

• Do the declared goals of the schemes correspond to those principles and internationally tried and tested practices which have been declared by relevant Hungarian and international innovation support organisations?

• Have the target groups of the tenders been chosen in accordance with the declared purposes? (e.g.: innovative small enterprises, enterprises following a modern market niche strategy, new technological enterprises)

• Have these schemes been evaluated earlier? Have the experiences of the earlier evaluations been taken into consideration when shaping, designing the present schemes of support? Would it be possible to use the indicators of previous evaluations with comparative purposes?

Efficiency :

• Is the invitation to these tenders intelligible, transparent and client- oriented enough?

• Are the selection aspects of winning companies announced in advance and are they clear enough?

• Do the documents of individual subsidy schemes consequently and intelligibly communicate the goals to be supported?

• Does the performance of the funded projects, in each of their phases (announcement, admission of the applications, preparation, selection, disbursement, control), follow the procedures of the donor organisation and the rules of the particular subsidy schemes?

Effectiveness :

• Were the supported projects effective, did they produce the expected positive results? Has a system collecting information and feedback been established?

• Was the announcement of the tenders and the selection procedure successful in finding those enterprises that belong to the declared target groups of the subsidy schemes? Was the support distributed among many firms, or rather, concentrated to a few beneficiary enterprises? Which approach has served best the original aims of the relevant policies?

• Did the choice of the financial model of the subsidy well serve the original aims of the relevant policies?

Were the options of (a) non-returnable subsidies, (b) preferential credits and (c) public procurements applied in a segmented, target group specific way?

• Did the subsidy schemes motivate the supported enterprises to take full responsibility for the projects and to assume the respective risks?

Impacts benefiting others than the immediate beneficiaries that extend beyond the actual projects:

• Did the positive impacts of the project extend beyond the benefited enterprises and reach a wider range of stakeholders, such as their clients, subcontractors, co-workers, and the settlements in which these enterprises operate as well?

• What were the proportions between supporting (a) ―soft deliverables", such as developing company knowledge base, knowledge transfer and organisational development on the one hand, and (b) ―hard deliverables" such as physical investments with the purpose of innovations, on the other hand? Were these preferences serving well the underlying aims of the respective policies?

Sustainability :

• Are the supported enterprises able to maintain the results of the projects after the phasing out of financial subsidies?

• Has the donor organisation sufficient administrative capacity to trace the evolution of the supported projects afterwards?

• What legal and institutional guarantees have been built in to make sure that those enterprises which break their contractual obligations in the medium term or in the long run will have to pay back the support?

Small sample business survey. It is a basic principle of international donor organisations such as the World Bank or the EU that for evaluating support schemes, the beneficiaries must be always asked about their opinions and recommendations. The quantitative part of the evaluation of this case study consisted of an analysis of a questionnaire based survey of small sample of supported enterprises which were ready to respond to the disseminated questionnaires.

The purpose of the survey was to obtain responses to the following research questions:

• How do supported companies interpret the concept of innovation?

• What is the attitude of the enterprises towards research and development activities?

• Which are the characteristic features of the innovative activities in small enterprises?

• What results have the supports achieved?

• What are the entrepreneurial attitudes towards innovation oriented subsidies?

• What kind of opinions have been shaped about the supporting organisation in the group of supported enterprises?

Sample and sampling considerations. The enterprises providing the answers had won the support of one of the 3 evaluated OMFB projects during the 5 years preceding the tender research. Surveys based on postal dissemination of questionnaires and voluntary responses of companies are preferred tools of firms conducting

market research and business climate research. However, this model of data collection allows the estimation of tendencies only with some distortions. The main source of bias is the fact that certain groups of enterprises may be overrepresented among the respondents: those motivated by: (a) gratitude for the support (b) hope for a next support (c) public relations considerations (d) an urge to complain about the efficiency of the proceedings of the support tender or about some supposed injustice.

No control group. The impact statements of business surveys are justified much stronger if they are based on a comparison of beneficiaries and not supported companies. However, in this particular case it was not feasible to ask a control group of not supported companies.

The willingness of enterprises to participate was relatively low: although the donor organisation had distributed 300 questionnaires among the supported enterprises, only 57 responses have arrived. Due to this small sample size, this could not be a representative survey, but a sufficient number of representatives of the following company strata were included:

• Companies from various size categories ( from small ones up to bigger firms)

• Companies of various sectors (agriculture, industry, and services, including knowledge based companies)

• Companies owned by Hungarians and by foreigners

• Companies of various legal forms (Limited Liability Company, partnership, Shareholder Company, sole proprietorship or other forms).

Table 14. The composition of responding enterprises by size classes

Size class Number of enterprises which

replied

The average number of the hierarchy steps of the firms belonging to the given size category

Less than 10 employees 29 1, 6

Between 11 and 50 employees 18 3, 0

Between 51 and 250 employees 5 3, 5

251 to 500 employees 0

-Over 500 employees 5 4, 8

Total 57 2, 5

Table 15. The composition of responding enterprises by legal form

Legal form Number of enterprises

Sole proprietorship 2

Deposit partnership 4

Limited liability company 41

Shareholder company 8

State owned enterprise 0

Legal form Number of enterprises

General partnership 1

Co-operative 1

Total 57

Table 16. The composition of responding enterprises by sector of their main activity

Classification of replying enterprises Number of enterprises

Agriculture and catering industry 11

Chemical industry 7

Metal industry 6

Production of machinery 9

Electrotechnic and precision engineering 9

Information technology 7

Telecommunications 2

Light industry 3

Construction industry 1

Trade 1

Protection of the environment 1

Total 57

Table 17. Does the company have a foreign owner or co-owner?

Proportion of foreign ownership Number of replying enterprises

There is no foreign co-owner 46

There is, with 15-50 % ownership 7

There is, with 100 % ownership 3

No answer 1

Total 57

The questionnaire consisted of 74 questions; half of them closed questions, the other half being open questions to which the respondents have answered in writing, with their own words. Such a wide range of open questions has in some respects counterbalanced the small sample size of the survey. These responses were evaluated with the method of content analysis. Due to the high number of the open questions it can be stated that returned responses could be regarded as substitutes or proxies of in-depth interviews. For the above reasons, while the results of the questionnaire based survey had to be handled critically, yet, its data has contributed to the understanding of the impact mechanism of the examined intervention.

In document Evaluation and Impact Assessment of Business Climate Development (Pldal 51-56)