Chapter 2: Theoretical Background
2.1 Tasks in language learning and teaching
2.1.5 Measures of task performance
production, and therefore, contradicts Skehan’s claim (Skehan & Foster, 2001; Skehan, 2007) that accuracy and complexity particularly compete for attention.
Robinson’s hypothesis reflects a different conception of how attentional resources are allocated as performance dimensions compete for attention. The psycholinguistic rationale for the effects of resource-directing task variables is based on Levelt et al. (1999) who suggest that increased conceptual preparation should promote lexical selection procedures and grammatical encoding. Drawing on the acquisitional arguments of Cromer (1974), Givon (1985), Perdue (1993), Slobin (1993), and Robinson (2003a, 2005b) claims that the gradual approximation of target tasks from simple to complex by increasing the communicative and cognitive demands of tasks should elicit more grammaticized and syntacticized language and push learners to revise their interlanguage system, which in turn promotes language learning.
The two models have triggered an array of empirical studies (for example Kuiken & Vedder, 2007a, 2007b; Robinson, 2007b; Gilabert, 2005, 2007; Skehan & Foster, 1999) which will be reviewed extensively in Chapter 3 with a specific focus on studies of task features from an information-processing perspective.
that takes place during the execution of a task (Gass, 1997; Pica & Doughty, 1985; Pica, 1994; Duff, 1986; Révész, 2007, 2009).
The general measures of complexity, accuracy and fluency (CAF) have been widely used in TBLT studies both following interactional and information-processing research programs. TBLT being concerned essentially with the way a focus onmeaning can be reconciled with a focus on form, these measures can be particularly “revealing as to the ways task characteristics and task conditions impact upon performance” (Skehan, 2009, p. 528). Complexity and accuracy both show how different aspects of form are attended to while fluency reflects the way meaning is conveyed during language production.
Before discussing the current developments in CAF research and a few challenges for its study, I will briefly introduce these constructs as they are used in TBLT studies.
1 Defining complexity, accuracy and fluency
Defining the construct of complexity is especially important in this study as this term is used both in relation to tasks, i.e., task complexity, cognitive complexity, and as a performance descriptor such as structural complexity and lexical complexity. As we have seen in section 2.1.4 above, Robinson (2001b) distinguishes between task complexity and task difficulty, the former referring to the cognitive load a task imposes on the learner, and the latter to how the participants’ own cognitive abilities and affective qualities influence their subjective judgement of the difficulty level of a task. Skehan (1998, 2009) uses the term task difficulty that includes both the objective cognitive and code complexity factors and the interactional and subjective learner factors in his model. Thus the terms difficulty and complexity are often used synonymously in the literature.
As a performance measure complexity contains a structural/syntactic reference, often oprationalized as the number of clauses per c-units or the proportion of subordinate clauses (for more examples and
illustrative references see Table 4 below). Complexity is also used in the literature with reference to lexical variety and sophistication which can be operationalized in various ways by text-internal measures (e.g., type-token ratio (TTR), D-value) and text-external measures requiring reference material to which the text is compared (e.g., lambda, Lexical Frequency Profile) (Daller et al., 2003).
Accuracy in L2 production reflects the way learners are able to avoid errors and conform to the grammatical norms of the L2. More accurate performance may indicate a higher level command of the L2, although it may also be the result of avoidance (i.e., the learner deliberately using simpler structures to avoid errors). However, even in the case of avoidance, higher accuracy scores indicate a better control of existing knowledge. Accuracy is most often operationalized as the proportion or number of error-free clauses and/or the target-like use (TLU) of various grammatical forms and structures.
Fluency can be defined as “the capacity to use language in real time to emphasize meanings...”
(Skehan & Foster, 1999, p. 96). Similarly to complexity, the construct of fluency is also multidimensional: Tavakoli and Skehan (2005) differentiate between three dimensions: breakdown fluency (measured by the number of pauses or silences), repair fluency (repetitions, false starts, reformulations) and speed fluency (e.g., the number of words per minute). While these dimensions are rather transparent, several different measurements have been used to evaluate L2 production in this respect (for a detailed summary of the measures used in recent studies see Table 4).
CAF dimension Focus Measures Illustrative references Structural complexity general complexity · mean length of
T-unit
· mean length of c-unit
· mean length of turn
· mean length of AS unit
Ishikawa (2007) Robinson (2007b) Robinson (2007b)
Tavakoli & Foster (2008)
complexity by subordination
· clauses per T-unit
· clauses per c-unit
· clauses per AS unit
· S-nodes per T-unit
· S-nodes per AS unit
· S-nodes per clause
· dependent clauses per total clauses
· subordinate clauses per total clauses
Albert & Kormos (2004) Elder & Iwashita (2005) Ellis & Yuan (2004) Gilabert (2007)
Lambert & Engler (2007) Ishikawa (2007)
Ishikawa (2007) Trebits (2010)
complexity by
coordination · coordination index Bardovi-Harlig (1992) phrasal complexity · mean length of
clause
Ishikawa (2007)
variety of certain
grammatical forms · frequency of certain classes of verbs
· frequency of certain syntactic structures
Robinson (2007b) Trebits (2010)
Lexical complexity lexical variation
· type-token ration
· D-value
· Guiraud index
· percentage of lexical words
Trebits (2010) Gilabert (2005) Rahimpour (1997)
lexical sophistication · lexical frequency profile (RANGE)
· lambda
Skehan (2009) Skehan (2009)
CAF dimension Focus Measures Illustrative references Accuracy
overall accuracy · percentage of errors
· number of error-free clauses per total clauses
Bygate (1999) Trebits (2010)
accuracy of
subordinate clauses · number of error-free subordinate clauses per total number of clauses
· target-like use of subordinate clauses
Trebits (2010)
Mochizuki & Ortega (2008)
accuracy of certain grammatical forms &
structures
· target-like use of various grammatical forms and structures
Trebits (2010)
Fluency
breakdown fluency · number of pauses per T-unit
· percentage of total pausing time
Bygate (2001) Sangarun (2005) repair fluency · number of
repetitions
· number of false starts
Bygate (1999)
Tavakoli & Skehan (2005)
speed fluency · number of words per minute
· number of syllables per minute
Mochizuki & Ortega (2008) Gilabert (2007)
2 General and specific measures of CAF
Many studies exploring the effects of task demands on L2 production have used general measures of complexity, accuracy and fluency (e.g., Skehan & Foster, 1999; Tavakoli & Skehan, 2005; Robinson et al., 2009). The obvious advantage of using general measures is that the results obtained for task effects across a wide range of task manipulations may be compared.
In support of Robinson’s Cognition Hypothesis it has been argued that specific measures of learner language would in some cases be better-suited to capture the specific resource-directing effects of tasks on CAF. For example, having to narrate a story in the past (There-and-Then condition) is cognitively more
complex then telling the same story in the present, and it is logical to assume that while it would stimulate the production of past tense verbs and time adverbials, this condition would probably not lead to greater lexical sophistication. Therefore the proposed effects of a task complexity factor on CAF could be shown using a task-specific measure which is more sensitive to the feature that the task is designed to direct learners’ attention to. Following this reasoning, a range of task-specific measurements have been devised and applied as demostrated by a growing number of studies (for example: Gilabert, 2005; Robinson et al., 2009; Trebits, 2010). The pedagogical relevance of using task-specific measures is clear: they provide empirical evidence for the way task design and sequencing can push learners to use certain linguistic structures and vocabulary items.
2.1.5.3 Beyond CAF measures
While L2 linguistic performance is almost always described in terms of CAF measures in TBLT and SLA studies, as some authors point out there are also a few challenges in connection with its underlying constructs.
Norris and Ortega (2009), for example, underline the importance of recognizing the multidimensional quality of each component of CAF. By way of a factor analysis of CAF measures used in recent TBLT studies, they show that syntactic complexity contains three dimensions which the different measures tap into: global/general complexity, subordination complexity and subclausal complexity.
Therefore, when interpreting and comparing the findings of different studies, we must be certain about what exactly was being measured by each. This is especially important because singling out one dimension of the CAF sub-constructs (for example, the dimension of subordination complexity) and using particular measures tapping into that one dimension (e.g., the proportion of subordinate clauses per c-units) is a general practice in TBLT research as shown by Norris & Ortega (2009).
Also, as several authors point out (e.g., Larsen-Freeman, 2009; Skehan, 2009), the different constructs within CAF are interrelated in a dynamic, multi-faceted manner. L2 learners do not progress towards more a complex, more accurate and more fluent performance in a linear fashion, but rather, the different components of CAF develop dynamically at different rates. For example, fluency measures can differentiate reliably between lower proficiency levels, while lexical complexity indicates differences between higher level L2 users. Under this light, and of much interest to SLA researchers, it is interesting to consider that these different dimensions of L2 output do not only compete for attentional resources but also support each other in the sense that learners’ development, and therefore, better performance in one area may depend on their development in another. The dynamics of the interrelationships between the various dimensions of CAF, however, can only be studied longitudinally (Larsen-Freeman, 2009). One exciting area of future research would be to match L2 grammar and vocabulary acquisitional sequences to CAF performance indicators (an example of this can be found in Révész, 2009). This then, would also allow for a more reliable comparison between the findings of different TBLT studies concerning the effects of task complexity on L2 output.
Relative to the challenges researchers face when interpreting CAF results, Skehan (2009) calls our attention to the fact that the way learners interpret a task or the task’s main goal may also influence which aspects of performance they prioritize. A similar argument is advanced by Larsen-Freeman (2009) and Norris & Ortega (2009) when they note the influence of context on language performance.
A very important aspect of L2 performance, though neglected thus far in TBLT studies, is communicative adequacy. As Pallotti (2009) points out it would be very important to take this dimension of language performance into consideration when interpreting CAF results. He argues that more complexity may not necessarily equal more advanced L2 production based on research comparing native and non-native speakers’ performance on the same tasks. Using more complex language when telling a
story can be interpreted as the sign of a more advanced control of the L2, while saying long and complex sentences when making an appointment on the phone may cause confusion in addition to sounding unnatural and being pragmatically incorrect. Communicative adequacy as a performance measure could be all the more relevant to TBLT studies that they have a very explicit pedagogical aim, namely, to contribute to our understanding of how tasks should be optimally sequenced in an L2 syllabus in order to gradually approach real-life language use. Given that the effectiveness of real-life language use often hinges on communicative adequacy, it is clear that discourse features measuring pragmatic efficiency would need to be added to the “classic” measurements of L2 output in TBLT studies if we wish to achieve a better understanding of how gradually approximating real-life target tasks may enhance learners’
interlanguage development by directing their attention to certain aspects of performance in a pedagogical setting.
As this section has shown, L2 performance measures of complexity, accuracy and fluency may be as controversial as they are vital in task-based research. Moving beyond the assumption that progress is a linear process, it is important to take into account the dynamic, multi-faceted nature of the interplay of the different dimensions of CAF themselves and the influence of task-external factors such as context and learner IDs on L2 production in order to more meaningfully interpret and reliably compare the findings of task-based L2 research.