Purpose and method of the evaluation

The main purpose of the applied research [bib_12] was to prepare an ex post evaluation of a specific support scheme implemented by the Public Foundation for Employment (OFA). This Foundation has operated the

„Support programme promoting the development and strengthening of small enterprises" in the years 2001 to 2004 in form of a grant scheme. During this time the Foundation allocated 1,019 billion HUF (approximately EUR 4 million) to grantees supporting altogether 212 SMEs.

The basic research questions of the evaluation were as follows:

• What were the effects of the programme on the labour market position of the supported companies as well as on their competitiveness ?

• What was the level of satisfaction of the applicant SMEs with the support and with the supporting institution?

• What was the opinion of the applicants about Hungarian SME development policy ? The empirical bases of the evaluation were provided by the following sources.

Analysis of documents. As a first step, the researchers examined the most important documents of the grant scheme: the annual announcements about the tender, the lists of the applicants and those of the winners, the criteria used by experts selecting the beneficiaries among the applicants, the project files of the individual winning companies, and the annual reports about the programme.

Interviews with project managers. As a second step, interviews were made with the managers of the programme about the success rate and success factors of subsidised projects, moreover about the lessons learnt and to be applied in subsequent support schemes.

Questionnaire based survey among supported and refused enterprises. As a third step, a survey was made with the applicant enterprises, both with beneficiaries and with those refused by the donor. In April-May of 2005, interviewers contacted 278 such small and medium size enterprises, out of which 146 enterprises had won the demanded support by the OFA, and 132 SMEs had applied but were rejected subsequently. This was a relatively high sample size, since during the years 2001-2004, altogether 742 SMEs applied for the OFA tenders in question.

Due to high sample size the evaluators were able to apply statistical testing of hypotheses. In particular, findings were significant on a 5 % level of significance in which the researchers compared (a) winner and rejected companies, (b) small and bigger firms, (c) companies working in various sectors, (d) firms having applied for subsidy in various years, or (e) firms being located in various regions. One of the main aims of the survey was to compare the supported and the refused enterprises.

The Questionnaires. The interviewers have used two distinct questionnaires: one for the supported enterprises (containing 146 questions) and a slightly different questionnaire for applicant enterprises that have been refused (87 questions). A specific feature of the questionnaires was, that they contained a relatively high number of open questions which could be answered in freely formulated text. This enabled the researchers to learn more about the attitudes and decision mechanisms of the questioned enterprises (e.g.: their satisfaction, expectations, optimism, fears, their motives to employ additional people or their willingness apply a growth oriented strategy for the enterprise). The responses to open questions were assessed with the help of content analysis.

The sample of the survey. The sample of the survey has proportionally represented the target group of the subsidy scheme, because the inner proportions of the sample have approximately corresponded to the inner proportions of the population of SMEs in terms of company size, economic sector, and region of Hungary .

• Size. About half of the SMEs in the survey were micro-companies.

• Legal form. More than half of the companies were limited liability companies, one fifth were sole proprietorships, one sixth was a partnership, but co-operatives, joint stock corporations and companies without legal personality were also represented in the sample.

• Age. The sample has contained enterprises of all ages, ranging from ―older" firms which were established 45 years ago to those founded 2 years ago. The median age of companies was approximately 8 years, both in case of supported and refused enterprises.

• Ownership. In most cases, the responding firms had two owners. Responding co-operatives had a large number of owners. Nearly all the answering enterprises had a Hungarian private person among the owners, more precisely; the proportion of Hungarian ownership was 100 % in nearly each case.

• Premises. About two-thirds of the applicants have worked on the same single location. In the circle of the supported firms the proportion of those with 2 or 3 premises was 30 per cent, while in the case of refused SMEs this proportion was only 20 per cent. Firms with more than 3 premises occurred only very rarely.

The questionnaire based interview was conducted personally with a high ranking manager of the applicant company, in 80 percent of the cases, the owner (or co-owner) of the enterprise. In all other cases the interviewed manager was selected as a competent person, who knew well about the project supported (or rejected) by OFA and/or personally participated in it.

Table 1. Distribution of supported enterprises by firm size and sectorNumber of enterprises

Sector

Company size group (number of full time employees)

Industry Agri-culture Services Trade Total

Micro – 1 (0-5 employees)

9 10 18 5 42

Micro - 2 (6-10 employees)

10 4 7 6 27

Small (11-50 employees)

30 13 13 15 71

Medium sized (51-250 employees)

0 5 0 1 6

Total 49 32 38 27 146

Table 2. Distribution of refused enterprises by firm size and sector Number of enterprises

Sector

Company size (number of full time

employees)

Industry Agri-culture Services Trade Total

Micro - 1 (0-5 20 12 29 17 78

Sector

Company size (number of full time

employees)

Industry Agri-culture Services Trade Total

employees)

Micro - 2 (6-10 employees)

4 2 6 4 16

Small (11-50 employees)

12 1 10 12 35

Medium sized (51-250 employees)

1 1 0 1 3

Total 37 16 45 34 132

Results of the survey

Profitability. In the year of the application about 86 percent of the enterprises were profitable, and less than 7 percent have made losses in that year. There was no difference between the supported enterprises and those which were refused, from the point of view of profitability at the time of the application.

Financial characteristics of the submitted project plans. The companies have applied, as an average, for subsidies in the range of 4 to 6 million Hungarian Forints (16.000 to 24.000 Euro). Supported firms on average had applied for somewhat higher subsidies firms that were subsequently refused. Most firms have offered to co-finance the project from their own sources in order to have the project realised. About 8 % of the applicants have declared that the project was co-financed by a third party, other than the supported firm. Such co-financing was, in most of the cases that of the character of a credit, and the co-financing institutions, in all such cases, were banks. 23 % of the supported enterprises received smaller amounts than they had applied for.

Professional content of submitted project plans. Nearly half of applicant companies wanted to spend the required amount of money on employing new persons. One third of applicants intended to purchase new appliances, another third wanted to embark on building operations. There were many projects aiming at social, educational, innovative purposes as well as projects aiming at the protection of the environment. In many projects, the above aims were attained with the help of consultancy services, which have helped the entrepreneur to implement the project or to establish relationships with other enterprises. Within the group of supported enterprises some two thirds have committed themselves in their applications to at least keeping the existing level of employment in the firm, while within the group of those applications that were refused subsequently, only half of the applicant SMEs have entered such obligations.

Organisational characteristics of supported projects. The time span of the supported projects was 15 months as an average, but as a maximum, projects with a duration of 2 years have also occurred. 46 % of the supported firms have relied on suppliers and subcontractors d, in the course of implementing the project.

Table 3. Purpose of the planned project as stated in the application document Respondents could select more than one option.

Purpose Supported Refused

N= Number of respondents =146 N=Number of respondents=131 Proportion of those who mentioned the given purpose, %

Purpose Supported Refused

N= Number of respondents =146 N=Number of respondents=131 Proportion of those who mentioned the given purpose, %

Hiring new employees 45 50

Acquisition of appliances 36 47

Construction operations 32 37

Introduction of new product or procedure

12 12

Purchase of services 11 3

Acquisition of ISO or other qualification

10 9

Training of employees 6 10

Consultancy or marketing 5 3

Self-evaluation of the programme by the beneficiaries. The majority of the supported enterprises have expressed the opinion that the subsidy was useful and reached its goal both with respect to facilitating employment and competitiveness .

Table 4. Why was the project needed? Content analysis of the responses of applicant enterprises

A content analysis of the various verbal responses given to the above question led to the following results.

One fifth of the responses emphasised elements connected with competitiveness : they mentioned the

introduction of some new product, technology, a system of organisation of work (e.g.: ISO). About one fifth of the emphasised elements were connected with employment. About one tenth of the respondents said that their survival depended entirely on the support.

Characteristic individual answers:

• Focus on human resources. „The subsidy has helped to employ people, it improved the conditions of employment" - „It helped to establish a nice dressing room for our employees." – „It has helped us to retain the working places."

• Focus on competitiveness. „ISO qualification is hardly dispensable in our sector." - „The catering unit needed the subsidy in order to comply by EU requirements." - „We wanted to develop, and development was indispensable for our further operation." - „The subsidy was used for the acquisition of computers and for the creation of an up-to-date information system." - „We badly needed such a machine which we could not have bought without the subsidy. The work of the employees has become easier."

Table 5. What has changed in your enterprise as a consequence of the support?

A content analysis of the verbal responses of supported enterprises.

The content analysis of the verbal responses has led to the following results.

The subsidy had the following effects:

• Increase of capacities;

• improvement of the market situation (mentioned by 37 );

• Improvement of employees’ general feelings (mentioned by 36);

• Improvement of quality (mentioned by 9); Increase in company employment (mentioned by 18);

• No significant effect (mentioned by 10);

• Improvement of the financial situation (mentioned by 10).

Other types of changes, less frequently mentioned: More satisfied clients; ISO qualification; Increased stability of the firm; retaining the staff; survival of the firm; improvement of the qualification of the employees.

Characteristic individual answers:

• Focus on employment. „Two of our employees have obtained a higher level of professionalism." - ―The working conditions of our employees have improved" - ―We did not have to dismiss workers." - ―The social block of the job-shop has been completed; the conditions for workers to douche and dress have become more convenient." - ―We have hired new employees, but later we were compelled to dismiss them."

• Focus on competitiveness. „Our turnover has increased; the general feeling of the employees has improved."

- ―We introduced an ERP software and created a website." - ―Complaints of consumers have decreased drastically." - ―Better equipped, enlarged kitchen, a bigger capacity of the catering establishment." -„We are working with modern machinery" - ―We have become known with the help of our websites, there is no need to explain things, customers can calculate their needs themselves and come to us with the already compiled order." - „We have gained experience in how to apply for further subsidies."

Effectiveness and efficiency. About 90 percent of the supported companies completed all the activities planned in the project in time. However, within the group of the smallest companies one fifth of the supported enterprises have reported a delay in properly implementing the project.

Impacts on employment. As a result clearly attributed to the subsidy, within the group of the supported enterprises 68 percent of the enterprises has employed new persons, on average five employees. Three quarters of the firms who employed new persons, reported that among the new colleagues there had been persons who were registered unemployed beforehand. Besides increasing or retaining the number of previously existing workplaces, in many cases the subsidy scheme improved the qualitative parameters of the working environment, such as the conditions of work, the safety of labour and the qualification of the employees.

Impacts on competitiveness. 73 % of the supported enterprises have reported that the subsidy has improved the competitiveness of the firm. As a result of the subsidy, a significant number of these enterprises has increased its turnover or its profitability or has stabilised its financial standing. In some cases new spheres of activities were added to the old ones, in others innovations were introduced. It is typical, in most of the above mentioned cases, that improvements were the direct consequences of the support given. The subsidy has helped several beneficiaries to purchase their inputs from an enlarged circle of suppliers. Some of the beneficiaries have reported that the support has increased their capacities or has improved their quality strategy.

Table 6. In what ways has the supported project improved your abilities to employ persons?

Respondents could select more than one option. Impacts are listed in decreasing order of the number of times they have been mentioned. Base: supported enterprises, N=143

Impacts of the subsidy on the quality of employment

The proportion of those whose response was „Yes"

Improved the employment possibilities of the enterprise

81

Improved the conditions of labour 65

Improved the safety of labour 54

Eased the overburdening of workers 44

Improved the qualification of workers 32 Improved the qualification of managers 20

Table 7. Changes following the implementation supported project

Base: supported enterprises, N=146

Since the implementation of the supported

project…

Yes, as a result of the subsidy

Yes, but not as a result of the subsidy

There were no such changes

Proportion of those who agreed

Increased turnover? 46 20 31

Improved profitability? 42 16 38

Widened range of

activities? Any new sphere of activity introduced?

34 8 55

Introduced any innovation in the operation of the enterprise?

22 8 67

Table 8. In what ways has the support improved the competitiveness?

Content analysis of the verbal responses given by supported enterprises Number of respondents agreeing with the statement. Respondents could select more than one option.

Statement Frequency

It has not improved competitiveness 31

It has improved competitiveness 123

It has improved competitiveness, in the following ways:

..Increased turnover 21

..More efficient work 20

..Introduced quality assurance (QA) 19

..Improved trust of customers/clients 13

..Survival of the firm 11

..Technological advantage compared with other enterprises

11

..Entry to new market 9

..Marketing and PR advantages (e.g.: the company has become more known)

7

..Cost reduction 7

..Wider range of products 5

Opinions about the supporting institution and about the efficiency of administering the application. The respondents have expressed their level of satisfaction in school marks. The following aspects were rated:

• intelligibility of the tender document,

• provision of the applicants with clear and useful information,

• counselling for applicants,

• helpfulness and professional expertise of the officials of the donor organisation.

Representatives of supported companies were much more satisfied with the work of the donor institution than those of refused companies. Representatives of supported companies have given marks between 4 and 5 for all of the above aspects, while refused companies gave marks between 3 and 4 for the same aspects. The following typical opinions can be highlighted:

• Favourable opinions. Companies which had already applied before for support at other donor organisations have found the evaluated tender more favourable than those of other donors. The following reasons were given: this tender was more fair, more simple, it was available for a wider group of eligible firms, it provided more friendly conditions for enterprises, it was more problem oriented and helped survival instead of forced development.

• Unfavourable opinions. Criticisms concerning the donor institution were related to the slow assessment of the applications, to insufficient justification of refusals, and to the strict requirements of project documentation.

Some 48 percent of the refused enterprises found either the decision of refusal or one or other elements of the proceedings unjust.

Table 9. What would you do differently today in implementing the project?

Content analysis of the responses given by supported enterprises. N=131 responses.

Three quarters of supported enterprises were satisfied with the way they had completed the project. The rest pointed out the following elements which might serve as a lesson in case of applying for an other grant:

• Preparation for application must start in an earlier phase of the realisation of the project,

• Employees and contractors must be chosen with greater care and cautiousness.

• Invoicing to the donor organisation must be prepared with greater care.

• Companies should commit themselves to their funded projects for the long term.

• Companies should employ new persons only in justified cases, after due deliberation.

Characteristic individual responses:

• Controllability. „I would organise the working processes in a more transparent way." - „I would pay more attention to the employment of new employees." – ―I would make the process quicker and would assign it more into the scope of authority."

• Accounts. I would break down the expenses and the deliverables into smaller units." – „One must pay more attention to book-keeping aspects, receipts and invoices of the project."

• Planning. „More time should have been devoted to the planning of the system."

• Keeping contact. „I would like to keep closer connections with the donor organisation."

Impacts pointing beyond the benefited enterprises. Most of the supported enterprises believed that the influences of the project went well beyond the enterprises themselves and affected the whole of the settlement in terms of newly established working places. Others have pointed out that the project had a positive impact on the environmental characteristics of the settlement. Larger enterprises have frequently mentioned the project had a positive impact on the cultural development of the settlement as a whole.

Sustainability of the results . The results of those projects which have facilitated the introduction of new products or new production processes, the construction of buildings or the acquisition of new devices were sustainable in most of the cases. On the other hand, „softer" project deliverables such as training courses, certification, consultancy and marketing services were thought to be sustainable only in two-thirds of the cases.

More than half of those projects where the results have directly improved competitiveness (such as the extension of production, the development of technology or the introduction of quality systems) were thought to be sustainable for more than 5 years. However, projects with employment related results (such as the hiring of new workers) were thought to be sustainable for a shorter period: in two-third of the cases such outcomes were to last only for less than 5 years. Representatives of beneficiary micro firms have considered that the deliverables of their supported projects were less sustainable than what representatives of larger companies thought about their own projects.

Table 10. What would have happened if you had not received the support?

Content analysis of the verbal responses of 143 supported enterprises.

This question is very important for impact assessment reasons because it asks directly about the opinion of the beneficiaries about the counterfactual scenario.

The responses of the beneficiaries demonstrate a significant impact. Only a few respondents stipulated that

The responses of the beneficiaries demonstrate a significant impact. Only a few respondents stipulated that

In document Evaluation and Impact Assessment of Business Climate Development (Pldal 38-49)