Limitations of the materials analysis and evaluation

As with all research, there were potential drawbacks of the materials analysis and evaluation carried out for this study as well, which were seen as follows:

(1) The frameworks developed for this study ‘are a reflection of the time at which they were conceived and of the beliefs of the designer’ (McGrath 2016: 46).

Therefore, the suggested frameworks should be revisited and refined periodically in the future.

(2) ‘The problem of the reliability of the evaluator judgements’ (McGrath 2016: 73) can place limitations on the findings of this study because, as McGrath (2013) states, the evaluation could be shaped by the evaluator’s experience. In addition, the semantic differential-based rating scale designed for the evaluation of the materials might generate increased cognitive demand which can result in errors (Friborg et al.

2006) in the judgements of the researcher (as might have been the case in the survey questionnaire regarding the teacher participants, see Section 4.2.8). According to Tomlinson (2018: 54), ‘no two evaluations can be the same’ regardless how well-structured they are, since they are fundamentally subjective. Although the researcher’s supervisors overviewed the development of the research methodology and instruments, a collaboration of more analysts and evaluators of different ELT backgrounds, as was deployed, for instance, in the survey of adult EFL courses undertaken by Masuhara and her colleagues (2008), would have further increased the reliability of the results in this study.

(3) As the researcher’s own analysis and evaluation of the most often used materials (based on the results of the teacher questionnaire) was carried out after the process of data analysis from the teacher questionnaire, which explored teachers’

444

perspectives on the cultural content in the most frequently used materials of their choice, the researcher’s own analysis and evaluation of the materials could have been influenced by the results of the teacher questionnaire. To settle this issue, the researcher endeavoured to distance himself from the results of the questionnaire both in time and space (i.e. not looking at the findings of the survey questionnaire).

The researcher’s exploration of the literature of materials analysis and evaluation was intended to help to combat these limitations and to strengthen this study. However, throughout the analysis and evaluation of the materials, the researcher was aware of the potential drawbacks detailed above, and together with his supervisors, endeavoured to find the best solutions as presented in this section (4.3) to mitigate them.

4.4 CONCLUSION

This chapter provided a detailed description of the design and methodology of this doctoral study. First, this chapter presented the research paradigm (4.1) drawing on the ‘research onion’ designed by Saunders et al. (2019). Then, the methodology of the survey questionnaire (4.2) was presented, followed by the discussion of the methodology of the researcher’s materials analysis and evaluation (4.3).

The section on the survey questionnaire (4.2) provided insights into the instrumentation of the survey (4.2.1). The section described the questionnaire design (4.2.2), and presented an account of the study participants (4.2.3): 33 in-service teachers delivering ESOL courses in ETB centres across Ireland. The section provided an account of the procedure of the study (4.2.4) highlighting the revisions needed to be made in the questionnaire after the pilot study to enhance the reliability of the questionnaire, and including the rationale for conducting the questionnaire via SurveyMonkey. The section also offered a description of the methods used for the data analysis of the main study questionnaire (4.2.5) focusing on analysing the quantitative and qualitative data from the survey and triangulation of data. It was noted that the use of SPSS and NVivo as analytical tools further added to the validity of the research findings (4.2.6). As this empirical study included human subjects,

445

ethical considerations were noted (4.2.7), and finally any methodological limitations of this survey study were examined (4.2.8).

The section on the researcher’s materials analysis and evaluation (4.3) presented the basic terminology used, then provided an account of the methods used for the selection of materials to examine the most often used Irish produced textbook The Big Picture and the most frequently used non-Irish published (UK) coursebook New Headway Pre-Intermediate (4.3.1). The section provided theoretical insights into the analysis and evaluation criteria by briefly re-presenting the checkbox-type frameworks for cultural content knowledge, cognition, and affect proposed in Chapter 3 and implemented in this study (4.3.2). The section discussed the theoretical foundation of analysing and evaluating the materials by providing a detailed description of the methods used for the data analysis at three levels (objective description, subjective analysis, and subjective inference), and the evaluation of the data sourced from the analysis (4.3.3). Then, the section outlined the data analysis methods (4.3.4), and dealt with the trustworthiness of the researcher’s materials analysis and evaluation by highlighting the importance of the use of NVivo and the theory behind the proposed frameworks to ensure validity and reliability (4.3.5). As the materials analysis and evaluation included published materials, ethical considerations were noted as well (4.3.6). Finally, any methodological limitations that might influence the findings of the researcher’s materials analysis and evaluation were discussed (4.3.7).

This chapter contained the design and methodology of the empirical study (teacher questionnaire), and the researcher’s own materials analysis and evaluation. Chapter 5 comprises the actual findings of the research, while Chapter 6 contains the integration of the findings of the questionnaire and the materials evaluation, and the triangulation of the data collected from the two data sets, including a comparison of the teachers’ perspectives on the cultural content in the materials in use with those of the researcher. Finally, Chapter 7 draws conclusions regarding the suitability of the materials used in Irish ESOL classrooms for fostering learners’ cultural awareness as part of their intercultural competence.

446