The questionnaire

On the basis of the findings of the first round of learner and, more importantly, the first round teacher interviews, a Likert-type questionnaire of 30 statements was developed in which teachers were asked to evaluate each statement on a five-point scale. The online questionnaire was selected as a tool since a large number of teachers with different geographical, educational and socio-cultural backgrounds were to be asked and general trends were to looked for. The Likert-type scale was selected as it is often used to measure feelings, opinions and attitudes of larger samples. Even though a wider range of options, for example a nine-point scale, would have allowed respondents to indicate more discrimination in their responses (Hatch & Lazaraton, 1991), the five-point scale was selected because it is the omnipresent grading system in education in Hungary, thus it seemed to be teacher-friendly.

An odd number of response categories was selected because respondents often invent a middle option or choose the middle two options if they are forced to take a stand with an even number of options (Brown, 2000, p. 28). The questionnaire is not a multi-item scale or summative scale in which the same target content area is addressed using several related and linked questions on the same issue (Likert, 1932; Dörnyei, 2002) but a Likert-type set of statements with single discrete items each eliciting responses on different issues (Clason &

Dormody, 1994) to avoid respondent fatigue.

To illustrate the way in which the online questionnaire built on the interviews that had been conducted prior to the compilation of the questionnaire, let us quote one participant of the teacher study. During the interview, she made the following remark:

Researcher: Alright. How often do you learn something new in relation to teaching?

Participant 4: Do we mean exercises, or … Researcher: Anything, really.

Participant 4: Well, in that sense, quite often… I thumb through resource books and then I collect, and if something works well, I will use it when we are doing the same material, or I adapt it to another topic. So, in that sense, I regularly learn something new.

As the issue of reflective processes emerged from many of the interviews, it was included in the questionnaire. The way this particular participant worded the phenomenon gave the researcher an authentic utterance that was later refined and inserted into the questionnaire as two distinct statements on reflective practices: 1. If an idea works, I use it later, too. 2. If something doesn’t work the way I have planned, I try it another way the next time.

The questionnaire did not contain any real questions. In the first half of it, Section A, statements about the skills and knowledge of English language teachers were listed, while in the second half, Section B, the possible sources of learning were collected. Skills and knowledge domain related statements can be seen as attitudinal items, while statements concerning the ways in which teachers learn can be seen as behavioural items (Dörnyei, 2002, p. 6-7) since the task of the respondents was to categorise the statements and thereby reveal their attitudes and habits. In the last short section, some biodata were collected. The statements were worded so that they still resembled the thoughts reported by participants of the first round of the interview study but remained understandable and relevant for teachers even if they worked in very different circumstances.

Before the questionnaire was administered, the statements had been tested for response validity in different ways. In addition to having asked for two researchers’ expert opinion and a former secondary school teacher to comment on and criticise the first draft of

the questionnaire, an attempt was made to make the questionnaire more comprehensible and valid by trying it out with both a primary school and a higher education teacher. The participating pre-test method was used as recommended by Converse and Presser (1986).

Both teachers were asked to think aloud while they were marking their decisions concerning the statements in the questionnaire and their verbalised thoughts were recorded. This allowed me to observe whether the decisions the two teachers took reflected their considerations. The method also allowed the researcher to identify statements in the questionnaire where prospective respondents might feel uncertain or undecided. Both verbal reports were analysed and the questionnaire was altered accordingly. Many of these alterations were necessary since the first draft of the questionnaire had been written in English and the Hungarian equivalent still displayed traces of the rough translation. However, it was not merely the language that needed improvement. The response categories needed to be refined and some of the statements were overcomplicated. The primary school teacher’s opinion was especially valuable and with her help the statements were made more straightforward and comprehensible. It was partly thanks to her insights that the originally six-point scale was reduced to a five-point one by merging the middle two categories (“Partly agree” vs. “Partly disagree”) since she was unable to differentiate between the two middle categories. In section B, where teachers are asked about the sources of their development, “agree-disagree”

statements were replaced by “true of me/not at all true of me” statements, also on the basis of the primary teacher’s think-aloud, in which she had found it difficult to work with the “agree-disagree” statements in this section. The higher education teacher also contributed substantially to improving the questionnaire. Upon her advice, three complicated statements were turned into six separate simplified ones, as she thought there were too many components mentioned in each.

To illustrate this point, when the statement ‘An EFL teacher needs to have a good command of the language’ was transformed into two: A) An EFL teacher needs to have a good command of English and B) An EFL teacher doesn’t need to have a good command of the students’ mother tongue. This way both target language knowledge and mother tongue knowledge were included in the questionnaire.

Another example of a similar modification was when she suggested transforming the following statement about cooperation into two different statements: An EFL teacher needs to be able to work well with their students and their colleagues. This way two new questions were formulated: A) An EFL teacher needs to be able to work well with their students and B) An EFL teacher doesn’t need to be able to work well with their colleagues. This latter decision is underpinned by expert advice: not “more than one idea should be contained in every question” (Patton, 2002, p. 358) and that “questions with many sub-parts” should be avoided (McDonough & McDonough, 1997, p.177).

In addition, the questionnaire was later piloted on 34 English teacher colleagues and the results were analysed (Soproni, 2007). The pilot version was a paper-based questionnaire with 31 statements with respondents marking their responses on the actual paper. The responses were computerised manually afterwards. As a result of the pilot study analyses, one statement that had appeared to be too general was dropped. “An EFL teacher needs to know what to do in the classroom” was a statement that did not eventually appear in the online version.

In the final online version of the questionnaire, or “opinionaire” as McDonough &

McDonough put it (1997, p. 176), there were two thematic sections. The first 14 statements covered the knowledge domains of an English language teacher while 16 statements covered the possible sources of learning. 11 of the statements were negative, 19 were positive in order to avoid respondents’ ticking the most positive response categories (e.g. Strongly agree). The

online version was first tested by fellow researchers and the data were then removed so that actual data collection could start. An example of a negative statement about teacher skills and knowledge is the following: “An EFL teacher doesn’t need to be able to work well with their colleagues” (Statement No. 14). An example of a positive statement about teacher knowledge is Statement No 10.: “An EFL teacher needs to be able to give clear explanations”. Positive statements about the sources of learning for a teacher included: “I have learnt a lot from my own previous teachers” (Statement No. 18.), or “I can use a lot of ideas from resource books (E.g. Recipes for Tired Teachers, 165 ideas, etc.)” (Statement No. 25.). Negative statements about the sources of learning included: “I think visiting my colleagues’ classes is a waste of time” (Statement No. 17.), or “I have never learnt any new words or expressions from my students” (Statement No. 27). The respondents were asked to indicate the extent to which they agreed with the statement or, to the extent to which it was true of them. (See Appendix C for details of the questionnaire.)

It was anticipated that by attempting to reach as many teachers online as possible, some of the native English speaker teachers and foreign non-native English teachers working in Hungary would be contacted. Accordingly, it was important to ensure that the online questionnaire was available to respondents both in English and Hungarian. The two equivalent versions were both commented on by expert researchers and practising English teachers and tested in the pilot study, and thus respondents were in a position to select the validated version which they would find more comfortable to work with.

The questionnaire may be criticised for not addressing constructs (e.g. learning from colleagues) with several statements. Since the success of data collection depended heavily on teachers’ willingness to provide data on a voluntary basis, the decision was made to compile the shortest possible user-friendly questionnaire. Respondents were thought to be “expert respondents” in the sense that they were familiar with the issues raised in the questionnaire

and had already completed several questionnaires in their lives. The respondents were also expected to be familiar with Likert-type questionnaires as they are widely used in the profession.

The questionnaire might also come in for criticism for using an odd number of choices. Researchers have pointed out the difficulties of interpreting the mid-point (McDonough & McDonough, 1997, p. 176), namely, whether the middle point means that the respondent either has no opinion or is not interested in the topic. Generally speaking, the researcher can never be certain how respondents interpret the statements or questions in a questionnaire, whether their interpretation is similar enough to that of the researcher or those of the other respondents. However, the questionnaire was not the primary research tool in the present investigations, and it was hoped that it would be sufficiently reliable. Reliability statistics presented in Table 9 and Table 10 provide evidence that the tool was reliable.

In the last section of the online questionnaire, factual questions were asked about participants’ mother tongue, age, gender, degree(s), teaching experience and teacher training courses. The response boxes were constructed in a way that they would not limit participants in their choices, which allowed them, for example, to tick more than one kind of institution or add several different courses.