Written Proficiency in EFL
3. Research context and questions
In order to assess students’ performance in writing in English and possibly guide them in further studies learning effectively, a writing program has been provided for the students attending the English programs at the BA level at the University of Pécs, Hungary. Each semester, there are five sets of proficiency tests to assess language learners’ performance in writing, grammar, reading, listening and speaking. The level of the exam is set at C1. The main focus in this paper is on students’ writing ability, as documented in the scripts of the essays they wrote in two examination occasions: in 2009 and 2014. (For the test instrument see Appendix A. See a sample essay in Appendix B.) For this purpose, we produced the electronic version of the essays written by keying in the handwritten essays. After checking text authenticity, that is, after we proofread the transcribed essays to ensure they were proper representations of what the students had written, we created our corpus to serve as the objective basis for our quantitative and qualitative analyses. Thus, our corpus, called the Happy Corpus, has been created from 95 student essays gathered from the two sets of proficiency tests: 51 essays (of 19,777 tokens) from the 2009 batch, and 44 essays (of 17,087 tokens) from the 2014 batch, two comparable subcorpora based on identical test instructions, allowing for an acceptable level of valid comparisons.
As can be seen in Appendix A, in the writing section of the proficiency tests given in 2009 and 2014 BA students were asked to choose one of the two tasks provided for them and write a 300-word text. The first task, Don’t worry! Be happy!, was about worry and happiness, in which students had an opportunity to explain their attitudes to people who worry, describe a person who worries too much, write a story related to that person, and share their ideas on whether people can be happy if they do not worry. The second task, Can anything worth learning be learned in the classroom?, was about students’ attitudes to classroom study, their ideas about how classroom activities could be useful by bringing an example and telling a story about a classroom activity which they thought was helpful in the process of language learning, and finally write about whether they agree with learning languages through activities in classrooms or not. The majority of students selected the topic about worries and happiness.
That is why we compiled the data in our Happy Corpus from essays written about students’ expression on how happiness and worries affects people’s life.
We asked two research questions: 1. What differences can be identified in the lexical profiles of the two subcorpora? 2. What can we learn from the way students focused on the narrative component of the task? As we were analyzing our data, we aimed to discover whether and how the content and language used in these texts changed between 2009 and 2014, and what was the level of academic vocabulary used in 2009 and in 2014. We were curious about an important aspect of short-text characterization: the ratio of functional words and content words as well.
4. Results and discussion
Several BA students of the English program in the University of Pécs, who took these proficiency tests 20019 and 2014, appeared to try to use grammatical rules and vocabulary as appropriately as they could. They also did their best to express their opinions and tell their stories as comprehensively as possible.
Thus, we investigated how the content and language that students applied in these scripts have changed between 2009 and 2014.
In order to do the vocabulary analysis and evaluate the lexical density in the two subcorpora, Happy Corpus 2009 and Happy Corpus 2014, and the occurrence of keywords in these subcorpora, we used the tools available on Cobb’s Lextutor website.
After studying the results of word categorization by users as K1 items including proper nouns and other items, we came to the realization that there was no major change in the use of content words in the essays across the two subcorpora as the lexical density was almost exactly the same in essays written in both 2009 (0.43) and 2014 (0.44). We could clearly observe the close percentage of function and content words appearing in the texts in 2009 (F = 56.88% and C = 30.75%) and 2014 (F = 56.12% and C = 31.35%). See Figure 1.
Figure 1. Lexical density in the 2009 and 2014 subcorpora
In our courses that include an element of writing skills development, students have confronted the issue of choosing an appropriate strategy that would result in the use of concrete, appropriate, and accurately chosen vocabulary items.
This is especially important in light of the fact that in a short text, lexical
richness needs to be achieved to a degree that can communicate good focus. To help with this, the proficiency exam approach used in the past several years has always included the recommended use of a thesaurus, with initial training in its use.
When we look at the specific layering of the two subcorpora, we can see that the patterns there, too, are almost identical. Figures 2 and 3 are based on the data extracted from the vocabulary profile tool of Cobb’s Lextutor site. Again, we can very similar profiles in 2009 and 2014. (The off-list count in both sub-coprora originated mostly from misspelled words and Hungarian proper nouns.)
Figure 2. The lexical profile layers of the 2009 subcorpus
Figure 3. The lexical profile layers of the 2014 subcorpus
Thus, we can see that a claim that some have made in term of standards of student proficiency levels being on a decline cannot be substantiated as far as the lexical density and profile character of the essays represented in the two subcoprora are concerned. Obviously, in addition to the objective measures concerning this aspect of L2 performance, we need to investigate further aspects as well.
Although this was not originally intended to be a focus of our study, in the course of the analysis it became obvious that the length of the essays often exceeded the recommended word count, 300 words. Figures 4 and 5 show those results, with a clear trend of the essays becoming longer over the years. The qualitative elements of our analysis brought to light that one reason for this was that in 2014 the narratives about the worried people tended to be longer (maybe indicating that there was more to worry about recently than five years earlier).
Figure 4. Word count of essays in 2009
Figure 5. Word count of essays in 2014
As pointed out earlier, we were also interested in how students dealt with the narrative requirement of the task as one of its content elements: telling a story related to the person worrying too much. We read and interpreted all stories and came to the realization that a few students wrote about tales which seemed logical, especially their stories about their mothers and how their worries
affected their children’s life emotionally and/or mentally, and how some of them described their mothers as their heroes. There were also stories in which students seemed to exaggerate somewhat to draw a picture of situations they wrote about which sometimes appeared fictional narratives, rather than descriptions of events that they experienced personally. Such details add evidence to the need that in courses preparing students for these exams more attention should be paid to issues of originality and credibility of non-fiction narratives.
5. Conclusion
In analyzing exam scripts, in our case, the essays written in a proficiency examination of students whose language skills are expected to be near or on the C1 level, we need to bear in mind the many constraints that students have to deal with in such tests. One among these is the choice of theme. As on both exam occasions presented in this paper the “Don’t worry” theme was chosen by a majority of students, we think we have some evidence that in this case, topic familiarity and willingness to express ideas about it were both sufficiently high, meaning that any features of the essays can be claimed to represent accurately the characteristic of individual and group performance.
Perhaps the most interesting finding of the study is the lack of any clear difference in the lexical profiles of the 2009 and 2014 essays. We did not anticipate either a dramatic drop or increase over the five years, but such a similar picture from both tests was largely unexpected.
Another useful finding for us as testers is related to the length of the essay scripts. As we have seen, the task instructed students to write about 300 words.
The majority of students wrote significantly longer than that, and thus we have objective evidence that raising the word length to 400 would be an adequate decision by the testing team, which will be implemented in the 2016 test.
We hope that this change will not deepen anyone’s worries.
References
Bitchener, J. & Basturkmen, H. (2006). Perceptions of the difficulties of postgraduate L2 thesis students writing the discussion section. Journal of English for Academic Purposes, 5, 4-18.
Cobb. T. (2015). Compleat lexical tutor. http://lextutor.ca
Doró, K. (2014). The lexical richness of undergraduate student essays: The influence of writing experience and essay writing strategies. AFinLA-e, 6, 7-19.
Doró, K. (2015). Changes in the lexical measures of undergraduate EFL students’ argumentative essays. In P. Pietila, K. Doró & R. Pípalová (Eds.), Lexical issues in L2 writing (pp. 57-76). Cambridge Scholars Publishing.
Granger, S. (Ed.). (1996). Learner English around the world. London: Arnold.
Granger, S. (2002). A bird’s eye view of learner corpus research. In S. Granger, J. Hung & S. Petch-Tyson (Eds), Computer learner corpora, second language acquisition and foreign language teaching (pp. 3–36).
Amsterdam: John Benjamins.
Horváth, J. (2001). Advanced writing in English as a foreign language: A corpus-based study of processes and products. Pécs: Lingua Franca Csoport.
Horváth, J. (2013). What are BA and MA students proud of? The case of the International Pride Corpus. Argumentum, 9, 178-185.
Horváth, J. (in press). Lexical profiling of written learner language in EFL: The case of a Hungarian university exam corpus.
Lehmann, M. (2013). The use of lexical bundles in EFL academic writing tasks.
In J. Mihaljevic Djigunovic & M. Medved Krajnovic (Eds.), UZRT 2012:
Empirical studies in English applied linguistics (pp. 131-141). Zagreb:
FF Press.
Mendikoetxea, A. Bielsa, S. M. & Rollinson, P. (2010). Focus on errors:
Learner corpora as pedagogical tools. In M. C. Campoy-Gubillo, B.
Bellés-Fortuno & M. L. Gea-Valor (Eds.), Corpus-based approaches to English language teaching (pp. 180-194). London: Continuum.
Partington, A. (1998). Patterns and meanings: Using corpora for English language research and teaching. Amsterdam: John Benjamins.
Römer, U. (2010). Using general and specialized corpora in English language teaching: Past, present and future. In M. C. Campoy-Gubillo, B. Bellés-Fortuno & M. L. Gea-Valor (Eds.), Corpus-based approaches to English language teaching (pp. 18-35). London: Continuum.
Appendix
Appendix A: The essay writing task of the May 2009 and December 2014 proficiency exams
Test of Writing
In this part of the test, you have a choice. Choose Task A or Task B and write the required essay according to its instructions. Your writing will be evaluated according to the following criteria:
the degree to which you have completed the task;
•" the range and appropriateness of vocabulary;
•" the range and accuracy of structures;
•" the organization and coherence of paragraphs.
For each task, four topics are given. Write about all of them. Treat each topic in a separate paragraph. Write about 300 words altogether.
You have 60 minutes for the test. You may use a thesaurus.
Task A: Don’t Worry: Be Happy Write an essay about worry and happiness. In the text,
• explain your attitude to people who worry;
• describe a person who worries too much;
• tell a story related to that person;
• discuss whether people can be happy if they don’t worry.
Task B: Can Anything Worth Learning Be Learned In The Classroom?
Write an essay about classroom study. In the text,
• explain your attitude to classroom study;
• describe a useful classroom activity;
• tell a story related to that activity;
• discuss whether anything worth learning can be learned in the classroom.
Appendix B: A script from the Happy Corpus
As I am someone who will raise, I would say that those who behave like this are often the perspirating, shivering, thrilling and so on. But there are not only external signs of the body. Heartbeats' acceleration and inability to express oneself in an intelligent manner are the most easily noticeable ones (concerning those internal). A very good technique to calm down myself as well as those who are nervous besides me is to represent the obstacle to get over with as a common-day task which can be solved without any difficulty, for example to prepare the dinner. If it fails, we have other things to eat in the fridge so, no real challenge.
Those who worry too much are unable to manage the tasks they have been given or they succeed in doing them with much bigger effort than it would be necessary for others. A good illustration for that is to give a lecture in front of the classmates or to interpret a piece of music for the relatives. In both cases, the auditors are well-known by the protagonist, comprehensive with him/her.
However, worrying people cannot fight back their emotions when being 'on stage'.
My brother is such a kind of person. I remember as if it had happened yesterday, when he performed for the first time at a family gathering at my grandfather's house. The assembly was waiting for my brother's first song impatiently, as he was already above us with his guitar, settled on a chair when instead of beginning to sing, he started to cry, his pouring tears dropping on the instrument's strings. After having given him more confidence, reassuring him about his talent he finally managed to start singing, and the once his performance had ended, all of us applauded him, so that he got enough self-confidence, since then to not to lose it ever again.
People will surely be happy by avoiding stress and self pression because there is no real fearful situation about completing a task if the individual convinces and persuades itself of its ability to do it. Of course, they are circumstances when these factor, I mean self assurance, does not impede on what occurs and we are impotent facing them, however there is this eternal and very true statement which says in French "quand on vent, on pent" signifying "who really wants something, (he/she) manages to get it". And isn't the most enjoyable feeling when we reach the thing what we desired to have?