3 RESEARCH DESIGN: the empirical investigation of learner needs
3.5 R ESEARCH INSTRUMENTS AND DATA COLLECTION PROCEDURES
3.4.6 The respondents of the needs analysis questionnaire
The needs analysis questionnaire targeting students’specialised linguistic needs was completed by 25 students studying different subject areas at MA level at UTE. The questionnaires were distributed in paper format at the end of two different semesters, in the final classes of two EU English groups. Since it was assumed that the students did not have sufficient experience in answering the survey questions, they were requested to participate in the research towards the conclusion of their classes. In each class, it was explained to the students that their participation in the survey was voluntary and the results of the questionnaire would not influence their final grades.
3.4.7 The respondents of the teaching material evaluation questionnaire
In an attempt to investigate UTE students’ views on the existing EU English teaching material, an evaluation questionnaire was administered in Hungarian to 14 students studying at MA level. The items of the questionnaire were designed to elicit answers on the students’
overall satisfaction with the existing teaching material and to evaluate the teaching material for further improvement of course and materials design.
the subject field in question (Dörnyei, 2007b). Consequently, broad questions about the topic could be developed in advance but the interview guide did not aim to lessen the richness of the respondents’ story. A set list of questions was drafted in advance, although the students were encouraged to elaborate freely on any related issues.
The interview questions (see Appendix A) were initially developed in line with the complex dynamic nature of the term motivation. The schedule incorporated ideas from Dörnyei and Ottó’s (1998) process model of motivation and were designed to cover a wide range of topics concerning motivation in an EU English class and the course material. In addition, the questions of the interview schedule were complemented by taking Hutchinson and Waters’
(1987) framework for analysing learning needs into account. The development and validation of the interview guides followed Maykut and Morehouse’s (1994) guidelines. First, the focus of the inquiry was identified and topics relating to the focus from the literature were collected.
Second, the selection of the categories of inquiry was followed by the elaboration of broad questions. The next step was to sequence and arrange the questions and prepare a draft of the interview guide. Finally, the interview guides were piloted and feedback was analysed to make the necessary revisions.
The first draft of the interview schedule was revised having been piloted to obtain clearly worded, relevant and unambiguous questions as well as to establish the structure of the schedule.
As a first step of piloting, an applied linguistics expert was asked to comment on all the interview items. With the help of her feedback, the scope of the original questions was extended and the original number the questions was increased from 20 to 33. Based on the linguistics expert’s views, the questions were rephrased and fine-tuned to better suit the purpose of the first research stage. The original schedule comprised two sections containing specific questions on the EU and on the teaching material, whereas the final list is divided into five sections:
personal background information; information on English language competence; students’
motivation, necessities, lacks and wants in the EU class; the teaching material, and learning methods. A pilot interview was conducted with a UTE student whose comments revealed that long and complex questions had to be refined to express clear and comprehensible questions for the students. (For the final version of the interview schedule, see Appendix A). The student interviews, which lasted about 15-20 minutes, were conducted in Hungarian either in the classroom after the lessons or in the office where the author used to work.
3.5.2 The UTE teacher interviews
A second semi-structured interview protocol was applied to conduct interviews with teachers who taught EU courses in Spanish and French. These interviews sought to capture the teachers’ views on the students’ immediate language needs and to elicit information about the course material (for the interview questions, see Appendix B). The first teacher interview schedule contained 14 questions on the teachers’ personal background and their views on teaching the specific language of the EU. To pilot the interview schedule, expert validation was pursued. A teacher colleague was invited to give her opinion on the relevance, the sequence and the wording of the questions. The original questionnaire items were not comprehensive enough to elicit the type of information sought: therefore, six additional questions were incorporated into the interview schedule. With the benefit of the consultation with the teacher, the final list of questions comprises 20 items relating to the teachers’ personal background, the students’ language competence, the specialist language of the EU, the students’ language competence, motivation, goals, interest, likes and dislikes and a final set of questions on the teaching material.
The teachers were interviewed in Hungarian at the Language Centre, UTE. The interviews lasted approximately 20 minutes and were recorded and transcribed by the author.
3.5.3 The teacher’s diary
In order to provide a picture from the teacher’s viewpoint, a diary was kept for the period of one semester in the spring of 2010 to record my immediate thoughts and comments after every lesson. These notes were collected in a word document. The diary entries included general impressions on the atmosphere of the lesson; what students liked and disliked and what difficulties were encountered in the lesson.
The diary also included general reflections on the following aspects suggested by McDonough (1994):
• the atmosphere of the lesson;
• what went well and what difficulties were encountered in the lesson;
• what I think the students liked and did not like;
• ideas for future classes.
A sample page of the teacher’s diary is provided in Appendix C.
3.5.4 The needs analysis questionnaire
Drawing on previous ESP research practices, a needs analysis type of questionnaire distributed among UTE students was found appropriate for the purpose of the next step of the inquiry about the students’ special language needs. As pointed out in the literature review, needs analyses have an essential role in course and materials design. The aim of the needs analysis questionnaire was to prioritise UTE EU English students’ specialised needs with focus on their perceptions of the course. The questionnaire was administered in two EU English classes in Hungarian. The English translation of the questionnaire is provided in Appendix D.
The design of the sections of the questionnaire was partly based on Dudley-Evans and St John’s (1998) concept of needs analysis to elicit responses on learners’ linguistic needs. One particular section of the questionnaire – questions on students’ difficulties – was adapted from Jablonkai’s (2008) needs analysis questionnaire developed for Hungarian EU professionals
with a focus on the use of genres used in the target situation.
The questionnaire administered to UTE students consisted of altogether seven sections containing questions on
(1) personal background;
(2) information concerning language background and proficiency including language examination level;
(3) interest in learning EU English and the importance of the course compared to other subjects;
(4) questions concerning the usefulness of the specialised language of the EU:
a) modules/units of the coursebook;
b) linguistic content of the course;
c) questions about the usefulness of language skills in the frame of the particular course investigated;
d) items on the usefulness of the various tasks of the coursebook;
(5) questions on the difficulties experienced by learners when using English language EU texts.
The sections ranging from one to five contained closed questions, while section six, seven and eight constituted three open-ended questions pertaining to students’ goals of learning EU English and the tasks with which they would furnish the coursebook. In constructing the questionnaire, a 5-point Likert-scale was used.
Testing construct validity should involve triangulation (Merriam, 2009), that is, the constructs explored should be tested from several perspectives. This view to check construct validity of the questionnaire was based on triangulation of data sources. The questionnaire was constructed on the basis of previous UTE student and teacher interview results. Moreover, content validity was established by asking a linguistics expert specialised in ESP to give her opinion on the first draft of the questionnaire. She was asked to comment on how relevant the questions were to the purpose of the questionnaire. As a result, wording and interpretation problems were resolved and the original number of items in the questionnaire was increased.
The questionnaire was piloted with a UTE EU English student in class who filled out the questionnaire and gave me feedback on its applicability.
3.5.5 The course material evaluation questionnaire
Survey research is considered to be essential for effective programme evaluation (Brown, 2001). Programme evaluation in Brown’s view means “the systematic collection and analysis of all relevant information necessary to promote the improvement of the curriculum and assess effectiveness within the context of the particular institutions involved” (p. 15). For the present study, the available information to be evaluated was the teaching material itself, therefore an evaluation questionnaire was designed that explored the components of the coursebook.
The questionnaire was constructed by the author to collect information about UTE students’ opinions on the course material used in the EU English class. The items of the questionnaire were based on previous needs analysis conducted in 2010 during the spring semester. The questionnaire contained 24 closed questions and one open-ended question. In the first section of the questionnaire seven items were included to collect data on students’ general opinion of the coursebook, the second section included questions asking about students’ pre-course EU and EU English knowledge, in the third section students’ opinion was requested on the importance of theoretical and practical knowledge, in the fourth section questions focused on whether the coursebook contained appropriate number of tasks and in the fifth section of the closed items the students evaluated the effectiveness of the course material in view of the improvement of their skills. The open-ended question of the questionnaire asked them how they would complement the teaching material, to write about what would be interesting and important for them. Students could circle or underline their answers. A 5-point Likert-scale was used to allow the participants to express how much they agreed or disagreed with the statements of the questionnaire.
The initial version of the questionnaire was given to two experts for review to identify potential problems related to wording and the order of the questions. Three UTE students, who were not attending the course, were requested to participate in a focus group discussion to see their understanding of the questions, which were then refined on the basis of their feedback.
The final version was piloted by a UTE student studying EU English. The language of the questionnaire was Hungarian. The questionnaire is included in Appendix E.
3.5.6 The teacher-researchers’ interviews
In an attempt to investigate the specific present and target situation needs of EU English learners at tertiary level and to generate pedagogical implications for materials design and teaching EU English in the higher education context, an interview study was planned with teacher-researchers. Based on the results of previous UTE student and teacher interviews (Koltai, 2012a) and on the results of the needs analysis questionnaire (Koltai, 2012b), a semi-structured interview schedule (see Appendix F) was developed to explore teacher-researcher participants’ perceptions and interpretations of EU English course content and materials design.
To enrich the arguments on the language use of the EU and to gain pedagogical insights, interviews were conducted with teachers of three different higher education institutions. As my interviewees were all researchers of the subject they taught, they were considered key informants for the issue under scrutiny. Their experience as teachers of EU English classes was assumed to contribute to a more holistic exploration of EU English course content to be taught in higher education.
After the interview schedule had been first drafted, it was modified to leave out some questions subsequentto the consultation with an EU expert in the field. Initially, the questions were targeted to elicit data on the definition of EU English for the Hungarian context. However, the expert validation revealed that this focus had to be narrowed down to better underscore learner needs. The validation was proved to be helpful for two reasons.
Firstly, it was pointed out that the original number of questions (20 questions) could be reduced to 13 significant questions to be asked from each participant. Secondly, it showed that paraphrasing the questions was necessary in order address the issues that are to be explored.
Consequently, the original questions concerning the definition of EU English were ignored and questions relative to the immediate teaching context were prioritised. Hence, the focus of the inquiry shifted towards gathering data primarily on the following issues:
(1) the differences between general and EU English;
(2) the rationale behind teaching EU English in higher education;
(3) EU English teachers’ instructional methods;
(4) EU English syllabus and materials design in the higher education context.
The interviews lasted approximately 40 minutes and were conducted in Hungarian.
They were audio recorded at the participants’ workplace. The interviews were transcribed by the author.
3.5.7 The EU professional interviews
To explore EU English learners’ target needs, EU professionals were selected to elicit data in the EU context to be able to compare perceived and real-life learner needs. The professionals were interviewed in 2012. Based on preliminary research results (Koltai, 2013) a semi-structured interview schedule (see Appendix G) was found appropriate to request the professionals’ views. Three of the interviews were conducted at the study participants’
workplace and one was recorded at the author’s workplace.
The interview schedule was piloted with the help of a Hungarian professional who spent several years working for the European Commission. The professional commented on the relevance of the questions and their sequence. The first interview schedule was deemed to betoo extensive and irrelevant questions were omitted out from the final set of questions. The list of the 33 questions altogether focused primarily on the following issues:
(1) EU English employees’ target language needs (2) Teaching EU English in higher education
(3) Communicative competencies in the target situation
(4) Language expectations from employees in the target situation (5) EU English course content
The interviews lasted about 40 minutes, were conducted in Hungarian and were audio recorded and transcribed by the author.
3.5.8 Document analysis
Based on my previous research results (Koltai, 2013), the existing course material used in my class was subjected to thorough analysis. The purpose of the investigation was to elicit information about the appropriateness of the teaching material for UTE students. Furthermore, the strengths and weaknesses of the coursebook in the light of the results obtained through the analysis of learner, teacher, teacher-researcher and EU professional interviews, the needs analysis and the evaluation questionnaire will also be analysed. Littlejohn (1998, p. 195) suggests three levels of textbook analysis:
(1)‘What is there’
Description of the material, analysis of statements of aims and targeted audience.
Analysis of the importance given to the use of sources in relation to other areas (in terms of page numbers), etc.
(2)‘What is required of users’
Analysis of tasks, what techniques are introduced and practiced, what students are expected to do
(3)‘What is implied’
Deducing the underlying approach and principles.
Following Littlejohn’s guidelines, I intend to analyse, describe and evaluate the overall characteristics of the available course material and draw conclusions for further use and improvement in the EU English teaching context.