Data collection instruments

In the project a multi-method approach was used in order to increase validity (see Kasper & Dahl, 1991). Foreign language proficiency was measured by a C-test. The main body of data was collected through role-plays, which served as pre- and post-tests in the project. Discourse rating tasks were used after the treatment in order to investigate students’ perception of grammatical and pragmatic violations. During the treatment, classes were visited by both researchers, so as to gain insight about how the treatment tasks were implemented, as well as to investigate general classroom issues. Observation of authentic speech was employed in order to complement the other, more restricted, data collection instruments. Finally, as a follow-up to the treatment program, the students were given questionnaires and all five teachers were interviewed, so we could receive feedback about the treatment and explore the participants’ views on pragmatic competence and general classroom issues, placing pragmatic competence in the larger context of EFL instruction. In the following sections I describe all seven data collection instruments.

4.3.3.1 The L2 proficiency measure

Students’ language proficiency was measured by a C-test administered before the treatment phase (see Appendix A). The C-test is considered a reliable technique to measure overall L2 competence and this particular test was validated among Hungarian EFL learners by Dörnyei and Katona (1992). It was used in the Leeds’ project (Németh &

Kormos, 2001), and we acquired permission to apply it in our study. However, an important adjustment had to be made. The original C-test consisted of three separate texts

of approximately equal length but of increasing difficulty. We decided that the last text would be too difficult and thus frustrating for students in our sample, therefore, we only administered the first two texts. The omission resulted in the decrease of the maximum score from 63 to 42 points. Students’ proficiency measures were calculated by adding up the scores of the two texts in the C-test (21 points each).

4.3.3.2 The pre- and post-test role-plays

The pre- and post-tests comprised role-plays, which is a widespread technique to elicit speech acts (Kasper, 1997a). It was important to have role-plays that resemble real-life situations where openings and closings fit in naturally. Both role-plays included a problem or a conflict that students had to solve, which allowed room for discussion (see Holló & Lázár, 2000). The role-play for the pre-test took place at a rock concert, where one of the students was a rock musician, the other a festival organizer (see Appendix B).

The post-test was an exchange between the owner of a house at Lake Balaton and a prospective renter. The role-plays were closed, that is, they provided guidelines for the students about the steps of the conversation and instructed them to greet and say good-bye to their partners. In this way the rubrics allowed for researching openings, closings, and opting out.

Prior to the experiment the role-plays were piloted twice with secondary-school students not taking part in the study. Initially two pairs of students were asked to perform the role-plays and to make remarks concerning the topic, understandability, and any other aspect of the tasks. Having received their permission, the role-plays and the feedback

session were recorded. Based on the participants’ suggestions, the role-play tasks were rewritten, and the revised versions were again piloted by a different set of students.

Before the pre-test, students were allowed to choose their partners for the role-play. At the post-test, the same pairs were asked to work together in order to create as similar circumstances as possible. Obviously, due to absenteeism, this was not always feasible, but in the majority of cases students worked with the same partner on both occasions. The role-cards were written in Hungarian, in order to avoid comprehension problems. Students were given time for preparation after they received the role-cards, but they could not discuss anything with their partner, see their partner’s role card, or ask questions from the researchers that pertained to the content of the role-plays. Prior to handing out the role-cards to students, we asked for their permission to audio-tape their performance. They were assured that the recordings would not be part of any school-related assessment. All the role-plays were transcribed and checked against the tape before carrying out the analyses. Although students performed the role-plays in pairs, their performance was analyzed individually.

4.3.3.3 The discourse rating task

All students received the discourse rating tasks after the treatment program. The questionnaire comprised eleven short situations that were easy to identify for our sample as they took place at school and contained interactions between school friends or student and teacher (see Appendix C). Some of the items were based on Bardovi-Harlig and Dörnyei (1998). The instructions asked participants to read the dialogues carefully and decide whether the speakers used English correctly or not. We also drew attention to the

fact that there may be mistakes other than grammatical ones. The questions were classified into five categories based on the kind of problem they contained. First, we included items with grammatical mistakes (such as verb tense errors), as in Items 6, 9, and 11. Second, dialogues with general pragmatic violations were present. These comprised stylistic and politeness mistakes, such as Items 1, 3, and 5. As the project focused on openings and closings, some items pertained to these two speech acts: Items 4 and 7 for openings, and Item 10 for closings. Finally, correct items were also included:

Items 2 and 8.

The aim with the DRT was to explore students’ perception to pragmatic and grammatical violations (based on Bardovi-Harlig & Dörnyei, 1998). However, our sample was not asked to rate the utterances on the scale of seriousness, as Bardovi-Harlig and Dörnyei already conducted a large-scale survey on this subject and we did not wish to replicate their study on a smaller scale. Rather the goal was to triangulate the post-test-role-play. The DRT and the role-plays provided data about different aspects of students’

pragmatic competence. Whereas the DRT was geared toward pragmatic awareness and recognition skills (identifying pragmatic violations in a written dialogue), the role-plays elicited active speech act production (performing a dialogue under somewhat stressful conditions).

The questionnaire was piloted twice. First, I asked a native speaker of English to go through the rating. Then, I gave the questionnaires with a different heading to my teacher trainees at Pázmány Péter University asking them to grade the “papers” as if they were written by their intermediate students. Furthermore, they were asked to rate the mistakes according to their seriousness. The follow-up discussions in both pilots provided

valuable insights about the task and prompted some necessary changes. As a side it was interesting to see the teacher trainees rating grammatical mistakes as more serious than pragmatic ones (confirming the findings of Bardovi-Harlig & Dörnyei, 1998).

4.3.3.4 Classroom observation

During the treatment period, each class was observed at least once by one of the researchers. The goals were to observe how the treatment tasks were implemented in the classrooms and to examine class atmosphere, teaching methods, lesson structures, and the like. The observation questions were different in the treatment and the control group classes (see Appendix D). We hoped to be able to justify some of the results with the help of the observation experiences.

4.3.3.5 Observation of authentic speech

I conducted fieldwork in the United States for four years. During this time, I observed openings and closings in authentic conversations in both formal and informal settings and collected field data (Edwards, unpublished). I derived the data mainly from native speakers’ interactions, but I also focused on native-nonnative conversations. I concentrated on speakers’ openings and closings as well as instances of misunderstandings or pragmatic failure between the partners. I also refer to Maisel’s (forthcoming) annotated thesaurus on closings, which is based on observation of authentic speech and student questionnaires. These two sources of authentic speech data provided valuable input when analyzing the pre- and post-test role-plays and comparing native- and non-native speaker speech act production.

4.3.3.6 Student questionnaires

The follow-up questionnaires were administered on the day of the post-test or if there was no time left, they were given to the teachers to distribute later. The aim was to gather background information about the students that was not possible to gain with the other data collection instruments. We asked four questions intending to expose students’

attitudes toward the role-plays, as well as different issues of learning English, such as in what contexts they use English outside the classroom, what problems they struggle with, and how they would describe a successful language lesson (see Appendix E). The language of the questionnaire was Hungarian in order to ensure comprehension and to allow the students to express themselves without limitation.

We had decided not to ask the participants specifically about the treatment tasks for two reasons. For one, since we requested the teachers to incorporate the tasks into their regular teaching, we thought it might be difficult for the students to remember them specifically. We also wanted to avoid “putting words in their mouths” by asking about the importance of such tasks, which may have elicited automatic positive answers from most students. The question pertaining to the treatment concerned the role-plays only, which all the participants had experienced and they were easy to remember because of the circumstances (doing it in pairs with the researchers present, being tape-recorded, etc). However, we hoped that some of the treatment activities would come up at the last question, which asked about a successful language lesson they can remember.

4.3.3.7 Teacher interviews

All five teachers were interviewed separately after the treatment in order to explore their impressions about the experiment and their opinion about teaching pragmatic competence (see Appendix F for the interview protocol). The interview was a semi-structured one: the protocol comprised a list of questions, yet allowed for digressions. The treatment group teachers were asked about the most successful task and any problems, questions, or suggestions they had concerning the treatment. The control group teachers received their package of all the treatment task materials and were given a short description of the experiment. They, too, were invited to share their reactions to the tasks as well as the role-plays that their students participated in. We also investigated all five teachers’ attitude towards teaching communicative and pragmatic competence.

Two interview questions overlapped with the students’ questionnaires: teachers were asked about what contexts their students use English outside the classroom and what problems they struggle with. In the data analysis, I will compare the responses of the students and the teachers in these areas. The interviews were conducted in Hungarian in order to avoid any self-consciousness and to provide free expression. Each interview lasted for approximately 20-30 minutes and, with the interviewees’ permission, was tape-recorded and transcribed.