when I speak English, I feel
Session 22: playing roles and having a good time
6.1 Results on the general proficiency test
The proficiency test consisted of 5 parts, each of which focussed on a different skill. Table 6.1 below shows the pre- and post-test scores for the chat group and the control group on the five
parts of the test: reading (read), language elements (lang), writing (writ), listening (list), speaking (speak), and the total test score (total).
Table 6.1 Pre- and post-test scores on the proficiency test (all scores are given in percentage, the maximum is 100%)
name n pre -rea d
post-read
pre-lang
post-lang
pre-writ
post -writ
pre-list
post -list
pre-speak
post-speak
pre-total
post-total
chat group total
8 81 87 69 76* 50 71* 80 79 60 65 67 75*
contr ol group total
12 80 77 52 72* 59 62 71 73 61 70* 63 71*
* p < .05
T-tests were used to reveal if the changes between the pre- and post-test results of the two groups were significant. The asterisks mark the mean scores that show a significant positive change in the post-test result compared to the pre-test result. In the case of the chat group, the results of the language elements paper, the writing paper and the total result of the proficiency test were found to be significantly better in the post-test. The control group produced significantly better results in the post-test when compared to the pre-test on the language elements paper, the speaking part and the total score of the proficiency test.
The following sections will discuss the test scores in which there was a significant increase for both groups: Firstly, language elements and the total score are focussed on, then the two skills in which only one group showed development: writing in the chat group and speaking in the control group. In the last part of the discussion, the two skills where no significant development took place: reading and listening are discussed.
6.1.1 Language elements
The t-tests showed that both groups made significant progress on the language elements paper of the proficiency test. In the case of language elements, which include grammar and vocabulary, the increase in scores can be explained by the regular development of the learners’ grammatical competence and vocabulary in the English classes. Elements of grammar and items of vocabulary are present in virtually all types of classroom activity. If learners take part in five English classes a week over an extended period of time, as was the case in both groups, their participation can result in the improvement of their skills in these areas.
As both groups show a significant increase in the language elements paper of the test, the usefulness of chat classes appears to be comparable to the usefulness of ‘traditional’ classes, as far as the acquisition of language elements is concerned. The chat logs produced by the chat group also contain evidence that there were opportunities to learn while performing tasks in chat.
The following chat extract shows an instance of negotiation of vocabulary, which is a regular phenomenon in the chat logs produced in the BHS Chat Project6.
Chat extract, 2 December, Seth & Tom [tom]I've just protect him. I wasn't smuggler.
6 In the total corpus of 69 chat logs, there are 32 instances of meaning negotiation.2
Online chat in the secondary school EFL class Chapter 4 Research Methodology
<seth>What does proteckt mean?
[tom]Defend, guard...
<seth>ok
Sequences in which grammar is negotiated hardly ever occur in the chat logs. In both groups, the participants’ development in grammar can be attributed to the instruction, and in the case of the participants of the chat group, to the regular production of English texts in the chat classes.
6.1.2 Total test score
Besides the language elements paper of the test, both groups made significant increase on the total score as well. The groups’ progress on the total score can also be interpreted as the outcome of the regular English instruction the learners acquired. Making progress between the beginning and the end of a school year is probably what every secondary school language teacher expects from her learners.
There are learners in probably all language groups, like Mitch in the chat group, who do not make progress, or at least not in a way that can be measured by a proficiency test. Not making progress can have several reasons. The learner may lack motivation, or the instruction may not match his learning style. Following the post-test proficiency test I gave the participants their results and conducted a short interview with all of them about how they interpreted the results. My first two questions were:
1 How would you explain the changes in scores?
2 What differences are there between the test in September and this one? Which one did you take more seriously?
Here are the answers Mitch gave to the questions:
LTJ, 30 March, Tuesday
1 ‘I was not interested in the test, I was in a bad mood when writing it.’
2 ’I took neither of them seriously.’
6.1.3 Writing skills in the chat group
The chat group performed significantly better on the post-test writing paper than on the pre-test paper. The writing part was a letter writing task in both cases. The score for the writing paper had three components. The first component was communicative value, which indicated how well the learners completed the task. The second component was accuracy. The third component was a score for the form of the letter.
As the inclusion of chat tasks in the EFL classes entailed that the learners understood the tasks and carried out the instructions precisely, they probably became fairly task-sensitive, and learnt to pay attention to task instructions. The inclusion of chat tasks also involved regular spontaneous writing practice, and the regular correction of the chat logs. This explains why the participants could focus more on accuracy in the post-test. During the regular English classes, we devoted a lesson to letter writing conventions. Reviewing expressions used in letters was also a useful activity that could have been conducive to good results on the writing part of the post-test.
During the interview in which the participants of the chat group individually evaluated their own performance, Martin said he was surprised by the increase in his score, but he also added that he had prepared for the post-test in order to perform better (LTJ, 30 March). Piper, Footie and Tom all mentioned that they attributed their progress in the writing part of the test to the fact that we had talked about letter writing conventions in class. The interview with the control group’s English teacher (see Member checks, 4.3.7.9) revealed that the control group also received
instruction on writing letters. This showed that as far as training in letter writing is concerned, the two group’s situations were similar.
6.1.4 Speaking skills in the control group
The control group performed significantly better on the speaking part of the post-test than on the pre-test. As the control group’s English teacher explained, this positive change can be attributed to the speaking activities done in class. A good part of the free production activities the control group did in class was in speaking, and consequently, the learners developed a certain routine in face-to-face conversations.
The control group’s gain in the speaking part is especially important from the viewpoint of the chat project, because practising speaking skills was a dominant theme in the chat group. A considerable proportion of the free production activities was spent with chat tasks in the chat group, and the time devoted to class chat exceeded the time spent on free production speaking activities, such as role-play or discussion. Four of the eight participants mentioned in the end-project interview that there should have been more speaking in the classes. One of the questions I asked in the end-project interview was:
Was there anything you missed from the English classes? Below are the answers Footie, Dot, Seth and Piper gave to this question.
EPI, Footie, Question 9
Footie: (szünet) Hm. Hát a chatbıl például a szóbeli kommunikáció hiányzott. De ugyanakkor az írásbelit azt fejlesztette. Tehát szóbeli kommunikációban is kellett volna ilyen feladatokat csinálni, de igaz, hogy azok nem maradnak így meg, mint a chat.
[Footie: (pause) Hm. What I missed from chat for example was oral communication. But then again, it developed written communication. So we should have done such tasks in speaking a well, although those cannot be saved like chat.]
EPI, Dot, Question 9
Dot: A többi óráról: talán a beszélgetéseket. Szerepjáték stílusú, két ember a csoportból. Talán beszédbıl volt a legkevesebb.
[From the other classes (Dot means the classes without chat): maybe oral conversation. Role-play-style, between two people in the group. Probably we did the least of speaking.]
EPI, Seth, Question 9
R: Mi volt, amit esetleg hiányoltál az órákról?
Seth: (nevet) Hát talán többet kellet volna így beszélni, hát ilyen párbeszédes feladatokat kellett volna többet oldani.
[R: Was there anything you thought was lacking from the classes?
Seth: (laughs) Well, maybe we should have done more talking, we should have had more conversation tasks. ]
EPI, Piper, Question 9
Piper: Picit több beszédes feladat.
[Piper: A few more speaking tasks.]
As shown above, four of the eight learners in the chat group felt we did too little speaking in class. The lack of focus on speaking is an explanation for the lack of significant progress in speaking skills. However, there is an overall tendency for numerical increase between the pre- and post-test scores in both groups. Significant changes in all fields of a learners’ proficiency can hardly be expected to result from one year’s instruction at school. Depending on the focus of instruction, the learners’ development will vary in the different areas of their proficiency.
Online chat in the secondary school EFL class Chapter 4 Research Methodology
6.1.5 Skills unchanged: reading and listening
There were no significant changes in either groups in their reading and listening skills. There are two possible explanations for this. The first one is that in both groups, the pre-test scores are relatively high, so in order to increase the score, the group members should have performed at a near perfect level on the tests. It should also be taken into consideration that both groups received instruction aimed at developing the productive skills, speaking and writing. Significant
development in reading and writing, without special training focussed on the receptive skills, was not possible.
In spite of the lack of statistical differences between the pre- and post-test scores in these two skills, the chat group’s score in reading increased from 81% in the pre-test to 87% in the post-test.
This positive tendency, which was not characteristic of the control group (see table 6.1), could be a result of the regular reading the participants did in the chat sessions. While performing the chat tasks, the learners were reading and interpreting the messages in real time, and responded to what they had understood. Consequently, they both provided and gained evidence of understanding their partner’s message correctly. Although the length of a chat post is just a fraction of the length of most reading texts learners have to tackle, the quick and spontaneous understanding of these is crucial to the success of the conversation.