First Validation

School in Budapest, which is a franchise programme of Oxford Brooks University (OBU).

These students gave 20-minute-long business presentations as part of their exam, upon completing a one semester course on Presentation Skills, according to the OBU syllabus.

The presentations were video-taped for benchmarking and moderation purposes. The video recordings of the Q&A sessions were transcribed and used for the pilot study. The transcription conventions described by Schiffrin (1994) and Grundy (2000) were applied.

The pilot study brought very promising results, specifically relating to the instrument. Therefore, the coding system was further refined and expanded to facilitate analyses of whole presentations. This marked the beginning of the development and validation process.

2) Extraction of unit categories: The second step consisted in extracting the categories used in the 17 analyses of the Q&A pilot study and expanding these to cover the whole of the presentation. In the extraction, two different types of units were identified: structure units (reflecting the linear organisation of the talk) and strategy units (speech acts and rhetorical techniques of presentations). In developing the list of units the constant comparative method of coding described by Maykut and Morehouse (1994) was used with new categories continuously emerging and later perhaps being collapsed into one through sustained simultaneous comparison of all the coding labels. It was at this stage that the number of categories was significantly expanded to encompass the whole of the presentation.

3) Expert interviews: In the third phase of the validation procedure, three experts were consulted and interviewed in April 2004. They were asked to give both oral and written suggestions on the improvement of the instrument and the interviews were

audio-taped. This first interview was with an Academic Writing teacher at IBS, who was asked to recommend ways of expanding the list of structural units to cover the body of the presentations. He comments contributed to the development of a system for analysing the linear organisation of the presentation into introduction, body and ending (each consisting of several sub-parts) illustrated in Appendix C. The other two interviews were with Presentation Skills teachers, who were asked to comment on the strategy units. As a result of their contribution, the list was expanded and definitions and examples were provided to clarify the more oblique labels of speech acts ( see point II. of Appendix A).

4) Trial application: Since the contributions of the interviewees significantly altered the list of speech categories and structure units, it was necessary to see whether it was possible to operate with the newly enlarged version of the analytical system. This enlarged system contained over 80 speech acts and structural units and applying the system was cumbersome, laborious and time consuming. Therefore, an attempt was made to merge and fuse some of the categories of the list. This made the application of the system easier, but also defeated the aim of the analysis because the results it yielded were too general. Ultimately, the very long list of speech act and structure categories was kept and extra effort was invested in the coding to obtain richer data.

5) Think-aloud session: After the instrument had been developed and expanded it was necessary to see if it is clear and flexible enough to be used by other researchers. A fellow researcher was trained to use the instrument and asked to think aloud while coding a presentation transcription. The first problem the researcher encountered was how to identify the basic unit of analysis. This had been easier previously in the Q&A analysis where the units were more visibly divided through turn-taking. This meant that clearer

instructions had to be provided for the unitising and separation of the speech acts. The second problem was, once again, the sheer number of speech act and structure categories and reading through the long list in an attempt to find the closest match. The difficulty was exacerbated by the fellow researcher’s unfamiliarity with the transcription. It became clear that before the coding process commences it is of vital importance to view the video tape of the presentation and to read the transcription several times to become very familiar with the details of the text.

6) Inter-rater comparison: The coding by the two researchers was then compared to assess the level of agreement. Unfortunately, due to the nature of the instrument, statistical analysis could not be carried out in the first validation phase. This was mainly due to the difficulties involving the separation of the speech acts in a monologue. Since the basic unit of analysis had not been defined at this stage, the number of units identified varied greatly between the raters. As a consequence, it was impossible to statistically measure the agreement in coding. Instead a qualitative comparative analysis was conducted. This process revealed that there were serious discrepancies between the coding of the two raters and that there was a need to make further changes to the instrument to achieve reliability and validity.

7) Finalisation of categories: To overcome the difficulties of applying the analytical system, which were identified through the trail application, think-aloud session and inter-rater comparison, it seemed reasonable to divide the long list of speech acts into categories. Language functions were chosen to act as the basis of this categorisation because they could easily be related to speech acts through the shared notion of intentionality. Consequently, the speech acts were grouped according to the general type of communicative function they perform based on Halliday’s (Halliday & Hasan, 1989)

overview of language functions. The groups included organisational, informative and interpersonal acts. The interpersonal acts are further sub-divided into territorial and cooperative acts derived from Widdowson’s (1983) territorial and cooperative principle.

This resulted in the current taxonomy of speech acts used in the Intentionality Model and illustrated in Table 1.

8) Intra-rater comparison: When the changes were completed and the instrument finalised, the cyclical methodology made it necessary to reapply the system for two reasons: to see how the changes had influenced the application and to have a second analysis by the same researcher to enable an intra-rater comparison. Similarly to the inter-rater comparison, statistical analysis was not possible here either, but the two analyses were compared to identify areas of discrepancy. The grouping of speech acts provided a more detailed picture of the discourse in the analysis. Also, in terms of splitting the text into speech acts, there was a tendency in the final analysis to include a larger number of smaller units. This trend of more detailed labelling and smaller unitising was present throughout the whole of the final analysis. Later on this lead to the definition of the speech act unit as the minimal possible unit of intentionality.

Another significant change of this initial or preliminary system, which was developed and validated at this stage, was that the structural units (see Appendix C) were later dropped in favour of Bhatia’s (1993) seven moves of promotional genres. The structural units were used in some of the empirical studies which will be discussed later on, but since they were based on academic writing, they seemed out of line with the rest of the system which focused on speech acts, language functions and communicative purpose.

The promotional genre moves, on the other hand, were much more in line with the overall model, and their use was supported by the idea of a hierarchy of intentionality.