Analysing and evaluating the materials

It seems to be difficult to draw a line between analysis and evaluation because ‘many experts […] mix analysis and evaluation’ (Tomlinson and Masuhara, 2018: 55) as was noted in Chapter 2. However, in simple terms, analysis refers to what is in the material, and evaluation denotes judgements made about ‘the effects of the material in a given context’ (Mishan and Timmis 2015: 57 based on Tomlinson 2003) as discussed earlier. This approach to materials analysis and evaluation was endeavoured to be followed in this study. Table 4.20 below illustrates the

436

methodological route from the analysis of the cultural content of the textbooks to the evaluation of the textbooks by content analysis (see 4.3) regarding their potential to foster cultural awareness.

Table 4.20. From materials to materials evaluation in this study

materials evaluation

As Table 4.20 above shows, after the identification of sections to examine within a sample unit of the most often used Irish and non-Irish published textbooks (as discussed in Section 4.3.1), texts and illustrations, as well as activities (as data sources) were examined using the frameworks developed for this study by means of content analysis. Then, the findings were synthesised in order to assess the materials regarding their potential for the development of the components of intercultural competence (cultural content knowledge, cognition, and affect) using the assessment grid so as to estimate the degree to which the textbooks could foster learners’ cultural awareness in an Irish context. This section provides further theoretical background to the methodology of the researcher’s materials analysis and evaluation carried out in this study.

437 4.3.3.1 Analysing the materials

The materials analysis process moved through three stages (see Table 4.21 below) that became increasingly deeper and more subjective: (1) objective description, (2) subjective analysis, and (3) subjective inference. The stages would often overlap and recur, especially in the case of subjective analysis and subjective inference. Although the terms were adopted from Littlejohn’s (2011) innovative approach to materials analysis, their usage was modified for the context of this study; however, overlaps as to what the terms refer to in Littlejohn’s work and what is meant by them in this research can be clearly seen as shown in Table 4.21.

Table 4.21. Stages of materials analysis (based on Littlejohn 2011)

Levels of

with texts and illustrations with activities 1 objective learners are required to do

3 subjective

As Table 4.21 above presents, Littlejohn’s (2011) terms are largely concerned with how materials function, but in this research, they are primarily used in the context of content analysis (see introductory part of Section 4.3) and the effects of the content on the learners. According to this, at the level objective description, an account of the division of texts, illustrations, and activities was provided, together with statements found in the textbooks; for instance, in the acknowledgements section, regarding the provenance of texts and illustrations. The subjective analysis identified (a) the basic

438

features of texts and illustrations and (b) the cultural elements in the texts and illustrations; and identified the pieces of work required of the learners to carry out through the activities. At the level subjective inference, deductions were made so that the recognised and identified elements could be categorised into the most relevant area of the proposed frameworks. Cultural elements from texts and illustrations were grouped using the framework for analysing cultural content in texts and illustrations (Table 4.13 in 4.3.2.1) following the definition of the components of the frameworks (Section 3.2.2). Activities from the aspect of cognitive processing of cultural content at different levels were categorised by the use of the framework for analysing activities for the cognitive processing of cultural content (Table 4.14 in 4.3.2.1) and a determined key verb (Table 4.15 in 4.3.2.1). The same activities from the aspect of affective processing of cultural content at different levels were grouped using the framework for analysing activities for affective processing of cultural content (Table 4.16 in 4.3.2.1) and a determined key verb (Table 4.17 in 4.3.2.1).

The overlap between analysis and evaluation could be particularly noticeable in the case of the deepest level of analysis, subjective inference, since it refers to ‘deducing demands on learner’s process competence’ to ‘come to general conclusions’

(Littlejohn 2011: 185). In the context of this study, for example, it implies decision-making as to which category of the framework for content knowledge a cultural element could be grouped into, or which of the levels of the cognitive and affective domains of learning an activity could be categorised into.

4.3.3.2 Evaluating the materials

The evaluation of the materials was based on the discovery of whether what had been looked for in the materials was found; in other words, cultural content (as defined in 2.3.4 in Chapter 2) guided by the proposed frameworks (see Tables 4.13, 4.14, and 4.16 in 4.3.2.1); and concerned the placement of a ‘value’ on the findings by subjective judgement-making (McGrath 2001). For example, after having collected data on the representation of Irish cultural content in texts and illustrations in The Big Picture, a decision was made as to the extent the findings are likely to improve cultural content knowledge in relation to Ireland as a component of cultural

439

awareness (for this specific example, see Table 5.16 in Chapter 5). As mentioned in Section 4.3.2.2, in this study, this ‘value’ denoted the extent to which materials foster cultural content knowledge, and cognitive and affective processing of cultural content in relation to different country-specific contexts (see Figure 4.4). The evaluation was expressed semantically: not at all, to a small extent, to a sufficient extent, and to a great extent (based on Tomlinson and Masuhara 2013) (Table 4.18).

According to McGrath, there are three principal methods that can be ‘broadly applicable to the evaluation of any materials for teaching-learning’ (2016: 45): the impressionistic, checklist, and in-depth methods. The impressionistic level implies the visual examination of textbooks that leads to general impressions about the material; the check-list method is applied to ascertain the extent to which pre-determined criteria are present in the material; and in the in-depth approach, particular features of the materials are examined (McGrath 2016). In this study, the combination of the check-list and in-depth methods dominated the evaluation of the materials by means of the assessment grid (Table 4.18 above) – with the Irish ESOL context in mind, based on the use of the tick-box-type frameworks (Tables 4.13, 4.14 and 4.16) and the consideration of the country-specificity of the cultural content.