7.2 Methods
7.2.4 Instruments
155
final number of participants was reduced to 13 students due to the attrition in the two terms, including the two newcomers who joined the group in the second and the third term.
Table 30
Participant attrition in the main study in Phase 4 Number of
students
Rate of attrition (previous term)
Rate of attrition (first term)
1st term 18 - -
2nd term 16 (+1)a 11.1% 11.1%
3rd term 11 (+2) 31.2% 38.9%
Note: aThe numbers in brackets represent the newcomers. 100%=first term
156
towards the internet were first handed out to the participants at the beginning of the first term, then at the end of the project. The questionnaire about the students’ self-assessment of their language development, however, was first administered in May 2013, at the end of the second term, then at the end of the project, because it aimed at evaluating the perceived language development in the first two terms and the third term. As one of the objectives of the questionnaires evaluating the course including the tools and tasks was to provide feedback and help design the next term, students were asked to fill them in at the end of each term.
Table 31
The instruments in the main study in Phase 4
Instrument Measured variables Time of administration
1 Background questionnaire students’
background September 2012
2 Placement test language
proficiency September 2012 3 Language proficiency test language
proficiency December 2013 4 Self-assessment of language
proficiency questionnaire
language proficiency
September 2012 December 2013 5 Self-assessment of language
development questionnaire
language proficiency development, impact on the learning process
May 2013 December 2013 6 Questionnaire about the use and
disposition towards the internet
disposition towards internet and web 2.0 tools
September 2012 December 2013
7 Course evaluation questionnaire
disposition towards internet and web 2.0 tools, impact on the learning
process
December 2012 May 2013 December 2013
8 Semi-structured interview schedule
disposition towards internet and web 2.0 tools, impact on the learning
process
December 2013
9 Teacher’s diary all aspects September 2012 –
December 2013 10 Wiki statistics students’ use of the
wiki September 2012 -
157
The semi-structured interviews were intended to supply information about the whole project, thus, they were conducted in December 2013 at the end of the last term. The language proficiency test was administered at the same time to all students studying English at the college. The teacher’s diary was written continuously from the beginning until the end of the project. Similarly, the wiki statistics, which are provided monthly on the wiki, were accessible for the whole period of the study, and even after its ending. The instruments can be seen in Appendix A, B and L to S, including sample pages from the Teacher’s diary and the wiki statistics. The only exception is the Language proficiency test, which cannot be published due to the several items borrowed from the item bank of Euro Examinations, which are confidential (Lukácsi & Hegedüs, 2014).
Background questionnaire
In the first week of the course I asked students to answer six questions about their language learning history and their expectations of the course, including their short term and long term goals and the ways they improve their English. I also asked them to rate their skills on a 5-point scale and also indicate which skills they would like to improve most. Although the questions were in English, I asked them in Hungarian to avoid any misunderstanding. I intended to use the answers to guide the development of the course (See Appendix M).
Placement test
The placement test all first-year students have to fill in electronically on the college’s virtual learning environment consists of 60 items focusing on general English vocabulary and grammar. Students have 50 minutes to complete the test, which is then evaluated automatically (See Appendix L).
Language proficiency test
The language proficiency test was the instrument of a research project monitoring the progress of all students of English between December 2013 and December 2014. It was
158
developed by four lecturers of the college and tested listening, reading and use of English skills, including grammar and vocabulary. The measured construct of the test was defined as
“the foreign language ability to function in an English-speaking work environment” (Lukácsi
& Hegedüs, 2014, p. 6). The Listening and the Reading paper consisted of 20 items each, while the Use of English paper had 30 items. As the participants of my research completed their English studies when the project started, their proficiency was only measured once.
Self-assessment of language proficiency questionnaire
The self-assessment grid for the Common Reference Levels in the Common European Framework of Reference for Languages (2001) was used for the participants’ self-evaluation.
Although most students were familiar with the different levels, the grid was discussed before the students assessed themselves. As data collection was carried out in the classroom, the grid was handed out to students on paper and in Hungarian to ensure full understanding. In order to be able to compare the findings, students were asked to supply their names on the self-assessment grid.
Self-assessment of language development questionnaire
The seven questions about the participants’ language development were included in the course evaluation questionnaires at the end of the second and the third term. They included one question about general English knowledge, listening skills, reading skills, speaking skills, writing skills, grammar and professional vocabulary (business in the second and tourism in the third term). As students are advised to take the B2 professional language exam in tourism and catering at the end of their language studies, a further question was added at the end of the project: How well prepared are you for the language exam? Students had to indicate on a five-point scale ranging from not at all to a lot, how much their language proficiency had improved in the previous term (See Appendix O).
159
Questionnaire about the use and disposition towards the internet
Data about students’ use of computers and the internet and their dispositions towards their use in education were collected using the questionnaire developed and validated in Phase 1 (See Section 4.2.2). The questionnaire was in Hungarian to ensure full understanding and students were asked to supply their names on the questionnaires in order to be able to compare the findings (See Appendix A and B).
Course evaluation questionnaire
The course evaluation questionnaire was administered electronically at the end of each term, once in the pilot study and three times in the main study. The questions were in English (12 in Group 1 and 18 in Group 2), except for the Comments for each topic, which students could write in English or in Hungarian to provide a fair chance to students with lower language proficiency. The questionnaires at the end of the second and the third term were slightly different in several aspects. They included seven questions about the participants’
perceived language development; additionally, the end-project questionnaire also had some questions about the students’ language exams and their plans for exam preparation. Thus, they consisted of more questions (25 in the second and 33 in the third term). The questions for all terms can be seen in Appendix N and O.
Semi-structured interview schedule
To supplement information gained from the end-project questionnaires, a semi-structured interview guide was developed. The original plan was to conduct interviews with ten participants in December and January 2013 about their experience of the project, however, only two students volunteered for an interview. The main reason for this could be the inconvenient timing, since December and January are usually very busy with exams at the college. Most of the students took the B2 professional language exam in January, which required considerable preparation. When I approached the students for an interview, they gave
160
me a further explanation for their unwillingness beside time pressure. They felt that they had expressed their views about the project through the open-ended questions and comments in the questionnaire, which often yielded lengthy monologues. These answers were analysed qualitatively, similarly to data gathered from the two interviews I conducted. (See Appendix P and Q).
Teacher’s diary
During the whole project the events that occurred in the classroom and on the group wikis were recorded in the teacher’s diary, as suggested by several researchers (Elliott, 1991;
McDonough, 1994; Miles & Huberman, 1994). According to McDonough, the main merit of keeping a diary is its ability to document the everyday working experience including individual student’s behaviour, the teacher’s feelings, students’ attitudes and the atmosphere of the class. The purpose of the journal was to record all my observations, as well as students’
reactions and comments about the different tasks and tools of the project. It also served as a source of personal reflection about the events and challenges that emerged during the course. I typed my notes that were made during the classes dealing with the wiki after the class in a Word document along with further comments. I also added notes at other times during the course when the wiki was dealt with in the lessons. The data collected this way can also serve as a source of triangulation to complement data drawn from other sources. A sample page of the diary typed and translated into English can be seen in Appendix R.
Wiki statistics
The statistics available on the wiki have been used to supply data about students’
activities on the wiki pages. The different types of information provided include the number of viewers for a given day or month, as well as the number of edits and messages. Data can not only be yielded about the whole wiki but also about specific pages. Furthermore, it is possible to track each group member’s activities for a given period of time. Unfortunately, the
161
three types of data cannot be combined, thus it is not possible to establish a connection between members and views (See Appendix S).