4.5 Instruments
4.5.2 Questionnaire
4.5.1.4 Classroom observations
The interviews were accompanied by two classroom observation sessions during which the teachers’ actual performance in the language classroom was monitored. For the classroom observations, the official teacher trainee observation sheet of the Department of English Language Pedagogy at Eötvös Loránd University was taken as a starting point, which was slightly modified with the assistance of my supervisor, on the basis of the theoretical frameworks of communicative competence consulted previously (Celce-Murcia, Dörnyei & Thurrell, 1995). For more details on the observation scheme, see Appendix E.
4.5.2.1 Initial questionnaire
The initial questionnaire was composed of four main sections. The first section consisted of 29 five-point Likert scale items, and was constructed by the author, based on Celce-Murcia, Dörnyei and Thurrel’s (1995) content specifications for the components of communicative competence. The aim of this part of the questionnaire was to measure the significance learners attribute to the different components of the construct.
The second part of the questionnaire included 30 five-point Likert scale items adopted from Kormos and Csizér’s (2008) motivation study. This section covered the three dimensions of the motivational self-system and some additional motivational variables (Csizér & Dörnyei, 2005a, 2005b; Dörnyei, 2009; Dörnyei & Csizér, 2005), with the purpose of exploring whether students’ attitude to the English language and culture have detectable effects on their beliefs about communicative competence. At that early stage of the research, investigating the relationship between learners’
motivation to learn English as a foreign language and the perceived importance of communicative competence was also a research interest of mine. However, the fact that communicative competence is in itself a complex construct and is already directed from various perspectives in the study lead to the recognition that including a motivational aspect would over-complicate the project. Therefore, the inclusion of the motivational self-system in the questionnaire was decided against.
The third section of the questionnaire contained 17 five-point Likert scale questions exploring the frequency of different types of contact with the target language community and culture. The final part concentrated on participants’ biographical data and language learning experience.
4.5.2.2 Validating the questionnaire
The initial questionnaire was validated by the author in May 2009. First, two secondary school students were invited to think aloud during the completion of the questionnaire. On the grounds of their comments, changes were made to the wording of some questions.
The second step in the pilot study was the administration of the questionnaire to a smaller sample (N=62) of secondary EFL students. Since the first part of the questionnaire was devised by the author, a careful validation of the instrument was of necessary. The responses were computed and analysed using the SPSS 16.0 (Statistical Package for the Social Sciences) software.
On the basis of the results, reliability measures were calculated and correction methods were applied. Based on the calculated reliability coefficients, unreliable items were reconsidered or excluded from the questionnaire, and, in certain scales, additional items were included. It was necessary to rephrase negatively worded items to smooth the progress of statistical analyses. Likewise, double negative items were reworded in order that participants could specify their opinions without any potential misunderstandings.
The initial version of the questionnaire in Hungarian and English can be seen in Appendices G and H, where reworded, reconsidered and additional items are highlighted in grey.
4.5.2.3 Results of the pilot study
The validation of the instrument began with calculating the internal reliability coefficients (Cronbach α-s) for each scale, in order to examine whether the scales
adequately measure participants’ attitude to the components of the construct. This procedure was of crucial importance in the case of the communicative competence scales, since the rest of the scales were adopted from a validated and standardized questionnaire (Csizér & Kormos, 2008).
Table 5 shows the the internal reliability coefficients for each scale. In the previous section I already explained that the initial version of the questionnaire included a section on the motivational self system (Csizér & Dörnyei, 2005a, 2005b;
Dörnyei, 2009; Dörnyei & Csizér, 2005) and some additional motivational variables with the purpose of identifying correlations between motivated learning behaviour and the perceived importance of the components of communicative competence. However, the inclusion of such a complex perspective would certainly have over-complicated the research project; therefore, this section of the questionnaire was excluded from the final instrument.
Table 5 Internal reliability coefficients in the pilot study
SCALES CRONBACH
ALPHA
NO. OF ITEMS COMMUNICATIVE COMPETENCE
Linguistic competence / Vocabulary -.252 4
Linguistic competence / Grammar .757 4
Spoken discourse competence .277 4
Written discourse competence .290 3
Actional competence .736 4
Sociolinguistic competence .619 5
Strategic competence -.001 5
MOTIVATION
Ideal self .748 4
Ought-to self .058 5
Language learning experience .866 4
Motivated learning behaviour .784 4
Cultural interest .615 4
Attitudes towards learning English .736 3 English as an international language .597 4 CONTACT VARIABLES
Direct spoken contact .639 5
Direct written contact .777 3
Indirect contact .438 4
Cultural contact .677 5
In the first part of the questionnaire, the internal reliability indices of the scales demonstrated that two of the scales, Grammar and Actional competence, had relatively high reliability coefficients, .757 and .736, respectively, indicating that the items belonging to the scales adequately tap into learners’ attitudes towards these two components of communicative competence. Considering the Sociocultural competence
scale, the slightly lower Cronbach α value (.619) might be attributed to the small sample size; therefore, no changes were made to the items. However, in the case of the other four scales, the computed reliability coefficients proved to be very low.
Taking a closer look at the Vocabulary scale revealed that the negative Cronbach α value might be attributed to a double negatively worded item (Q15), which could be difficult for the participants to interpret properly. Therefore, the original wording ‘It disturbs me if I don’t know the correct spelling of a word in English’ was rephrased to
‘I think getting the correct spelling in English was crucial.’
Considering the Spoken discourse competence scale, two items turned out to be problematic for different reasons. The problem with Q17 was again negative wording, which was warned against by severalresearch methodologists (Creswell, 1994; Dörnyei, 2007), since it might cause difficulties in the statistical analyses. Consequently, the original phrasing ‘I don’t always understand speakers of English who talk fast’ was changed to ‘I understand even those speakers of English who talk fast’. The other tricky item (Q26) was a double-barrelled question, which again should be avoided, since even if respondents do provide an answer, it is impossible to decide which part of the question it refers to (Dörnyei, 2007). Therefore, the original compound statement ‘It upsets me if I cannot express myself because a word or expression does not come to my mind’ was simplified to ‘It does not upset me when a word or expression does not come to my mind.’
Q23 in the Written discourse competence scale was also a compound item, inquiring about the participants’ opinion about the importance of writing letters AND emails. Thus, the item was split into two separate statements, Q23 (For me it is important to learn how to write a formal letter in English.’) and Q30 (‘For me it is
In the case of the Strategic competence scale, there turned out to be a conceptual problem with Q25. Although the aim of the item was to measure respondents’ attitude to strategic competence, the item ‘When I cannot express something in English, I change topics’ included an avoidance strategy. As a result, despite the fact that the item was positively worded, indicating number 5 as an answer still denoted a negative attitude to communication strategies. Therefore, the item was rephrased to ‘When I cannot express something in English, I try to say it in a different way’.
As explained before, the second part of the initial version of the questionnaire, i.e., items 30-57, were not included in the final version of the instrument. Concerning the third section of the questionnaire, it can be seen that some of the contact variables scales, Indirect contact and Direct spoken contact in particular, got rather low Cronbach α values, .438 and.639, respectively. Despite the relatively low internal reliability coefficients, however, no modifications were done to these scales. The reason for keeping the items unchanged was the fact that the scales were adopted from a standardized questionnaire, and thus it was hoped that in a larger sample they would prove to be more stable.
The findings of the pilot study demonstrated that it was of vital importance to validate a newly constructed instrument. The results of the reliability analysis indicated that unreliable items (and scales) had to be reformulated in order to get a more stable instrument. Besides, it also helped to obtain a more narrow research focus.
4.5.2.4 Final questionnaire
The revised questionnaire had three main sections. As previously mentioned, the first section aimed to explore participants’ attitude to the components of communicative competence. Due to their complexity, two components of the construct, linguistic and
discourse competence, were further divided into vocabulary and grammar, and spoken and written discourse competence, respectively.
For questions 1-30, the respondents were requested to assess on a five-point Likert scale the extent to which a statement concerning learning English is accurate about them. The items intended to cover the concepts detailed below, illustrated with a sample item:
Linguistic competence / Vocabulary (4 items): the knowledge of lexical items and rules of phonology and orthography
sample item: ‘I would like to know as many English expressions as possible.’
Linguistic competence / Grammar (4 items): the mastery of rules of morphology, syntax and semantics
sample item: ‘I think it is essential to know English grammatical rules.’
Spoken discourse competence (4 items): the ability to create cohesive and coherent texts in speaking
sample item: ‘I can easily use the words and expressions I learnt in spoken communication.’
Written discourse competence (4 items): the ability to create cohesive and coherent texts in writing
sample item: ‘For me it is important to learn how to write emails in English.’
Actional competence (4 items): the ability to perform and interpret speech acts and speech act sets
sample item: ‘I think it is useful to know how to make a phone call in English.’
Sociocultural competence (5 items): the knowledge of how to be appropriate in the social and cultural context of communication
sample item: ‘I think it is useful to learn slang expressions in the language class.’
Strategic competence (5 items): the inventory of verbal and non-verbal communication strategies
sample item: ‘When I don’t understand a word, I try to guess the meaning on the basis of the context.’
The second part of the questionnaire, i.e., items 31-47, concentrated on contact variables which might exert influence on learners’ beliefs. Respondents were asked to specify how often they are in different types of contact with the target language:
Direct spoken contact (5 items): students’ meeting and talking to native speakers of English
sample item: ‘How often do you use the English language during your holidays in Hungary?’
Direct written contact (3 items): electronic and traditional correspondence and chatting in English
sample item: ‘How often do you write emails in English?’
Indirect contact (4 items): seeing but not talking to native speakers of the target language
sample item: ‘How often does your English teacher talk about life in an English speaking country?’
Cultural contact (5 items): use of the English-speaking media sample item: ‘How often do you read magazines in English?’
The final section of the instrument included questions regarding certain biographical data of the participants, such as age and gender. Open ended items were also included to reveal the first, second (and third) foreign languages participants studied, and the contexts of learning these foreign languages.