Pragmatic competence and L2 proficiency: an item analysis of the DRT

In the item analysis of the DRT I give an account of the facility values of the variables as well as analyzing participants’ performance on the various items. The facility value measures the difficulty of an item, namely what percentage of students answered each question correctly. The sample size here is 86, as six students were absent when the tasks were distributed. At this point, the difference between the treatment and control group will not be analyzed. Table 7 shows the five categories the DRT items are classified into: ones with mistakes in grammar, pragmatics, openings, closings, and correct items (see Appendix C and section 4.3.2.2). Facility values are calculated for every item as well as each category combined.

Table 7. Facility values of discourse rating task items and categories Item type Name and number of

item

Facility value of item (%)

Cumulative facility value (%) Item 4: Shopping 79.1

Opening

Item 7: Summer holiday 96.5

87.8 Item 1: Snack bar 65.1

Item 3: Class trip 50 Pragmatic

Item 5: Library 94.2 69.8

Item 6: Homework 44.2 Item 9: Before class 77.9 Grammar

Item 11: Forgetting book 52.3

58.1

Closing Item 10: After school 38.4 38.4

Item 2: Invitation 45.3 No mistake

Item 8: Teacher’s book 23.3

34.3

Students reached the highest score on the items with mistakes in openings. The cumulative facility value for this item type is 87.8%. The item that the largest number of students (more than 96%) answered correctly was Summer holiday, which was an exchange between a student and a teacher. Respondents pointed out John’s lack of politeness and respect towards his teacher. The other item in openings, Shopping, was based on the pragmatic differences in post-openings between English and Hungarian (Edwards, 2003a) and it also yielded a high score (79.1%). Students noted that complaining is inappropriate in this context (“az angoloknál nem illik panaszkodni”) and that topics of health, finances, and politics are taboos in English.

A possible explanation for the high facility values in this category is that the items are not very challenging in nature. They contain mistakes that are easy to identify, such as saying What’s up? to a teacher (Item 7) or responding to How are you? with a lengthy

complaint (Item 4). In the case of closings (Item 10), the facility value is only 38.4%, which shows the difficulty of the item. The mistake to be identified was using Hello! as a leave-taking, which is a typical pragmatic problem of Hungarian EFL learners (Edwards, 2003a) and it apparently caused difficulty for many respondents. In hindsight, I should have included more items on closings in order to provide more data for generalizability.

The items with general pragmatic mistakes contained two kinds of speech acts:

requests (Items 1 and 5) and a refusal (Item 3). In the case of requests the nature of the problem was similar in both cases (using direct utterances such as I want… or Tell me…).

Interestingly enough, the items challenged students to different degrees. Almost all respondents (94.2%) identified the problem in the Library situation, whereas only 65.1%

scored correctly in the Snack bar interaction. The students pointed out the “impoliteness”

of the answers and that they sound more like demands than requests. Some respondents described such answers using quite strong adjectives, such as bunkó or alpári. As for the speech act of refusing, exactly half of the participants identified the item as incorrect.

They pointed out the importance of providing reasons for the refusals in order to be more polite. In the cases where students did not identify the pragmatic mistakes correctly, they mentioned the need to make changes in vocabulary or word order. Hardly anyone considered these items correct.

The category with grammatical mistakes posed various difficulty levels for participants, with scores ranging from 44.2 to 77.9% and a cumulative facility value of 58.1%. The item that students solved the most successfully (77.9%) was Before class (Item 9), where they had to identify the Let us to structure as incorrect. Possibly due to the well-known phrase Let’s go and the shortness of the utterance, this item did not

present any significant difficulty for the students. Surprisingly, only half of the respondents (52.3%) discovered the incorrect past tense (didn’t brought) in Item 11.

Those who answered this item incorrectly either missed the grammatical mistake altogether and wrote “correct” as their answer, or they pointed out a pragmatic problem, for instance that Maria provided too much explanation. The low facility value of this item may be due to the fact that Maria’s response is rather lengthy, therefore participants may have missed the mistake “embedded” in the utterance. Furthermore, I classify using a past tense verb with the auxiliary did as a typical Hungarian EFL mistake, which may still manifest itself at this level of proficiency. Incidentally, students admitted in the follow-up questionnaire that verb tenses are indeed the most challenging area of English grammar for them (see section 4.4.7). Last, fatigue effect may have contributed to the low facility value, as this was the last item on the DRT. The most challenging item in this category is Homework (Item 6), which only 44.2% of the students identified as incorrect.

This item again contained a verb tense mistake (I haven’t been here yesterday). Many students failed to recognize any mistake whatsoever, others suggested vocabulary changes or less explanation.

The category that proved to be the most challenging is that of correct items (Items 2 and 8). The cumulative facility value is 34.3%, meaning that only a third of the students were able to answer these items successfully. The participants found “mistakes” of vocabulary, syntax (verb tense usage), and pragmatic nature (too polite or formal answer). The possible reasons for this are two-fold. The first is what I call “red pen effect”, meaning that when students are asked to correct mistakes in an exercise, they tend to “overcorrect”. I have come across this phenomenon in my teaching experience

both in the case of my intermediate and advanced EFL students and with my English native speaker students in the United States. The second possible reason for the low facility value is that both dialogues in this category contain relatively difficult subordinate clause structures, which could have added to the challenge.

All in all, the item analysis provided much insight into the discourse rating task: it supplied information on the difficulty of each item and the examination of students’

answers revealed reasons why they may have struggled with some items. In some cases the analysis yielded unexpected results, such as students’ different performance on opening and closing items and the varying scores on pragmatic items. The next section explores the relationship between L2 proficiency and students’ pragmatic competence analyzing the results of the C-test, the discourse rating task, and students’ speech act production in the role-play.