Reliability indices of the scales - Results of the questionnaire study

7.1 Results of the questionnaire study

7.1.1 Reliability indices of the scales

In spite of the fact that previously unreliable items and scales were reconsidered, calculating the internal reliability coefficients and submitting the data to principal component analysis revealed that there was still room for improvement in terms of the internal consistency and stability of the scales. The scale Grammar preserved its reliability, getting a .782 Cronbach α value; however, even those scales that had previously turned out to be reliable, such as Actional competence, got comparatively low reliability indices. Additionally, principal component analysis identified two

distinct factors in the scales Spoken discourse competence, Written discourse competence and Strategic competence.

Table 9 summarizes the results of the reliability and principal component analyses (with no rotation).

Table 9 Internal reliability coefficients and the number of components extracted before the application of correction methods in the main study

SCALES NO. OF

COMPONENTS

CRONBACH ALPHA

NO. OF ITEMS COMMUNICATIVE COMPETENCE

Linguistic competence / Vocabulary 1 .545 4

Linguistic competence / Grammar 1 .782 4

Spoken discourse competence 2 .490 4

Written discourse competence 2 .483 4

Actional competence 1 .602 4

Sociolinguistic competence 1 .485 5

Strategic competence 2 .158 5

CONTACT VARIABLES

Direct spoken contact 1 .804 5

Direct written contact 1 .702 3

Indirect contact 1 .610 4

Cultural contact 1 .733 5

Due to the low Cronbach α values and the presence of two components in certain scales, it was necessary to perform a closer examination of the problematic scales one by one.

Concerning the scale Vocabulary, even though only one component was identifiable, indicating that the items load onto a single latent dimension, the internal reliability index was slightly low (.545). When items’ impact on the Cronbach α values

and expressions’) covered a different facet of the component. Taking a closer look at the items belonging to the scale revealed that whereas Q28 concentrated on the liking of building vocabulary, the rest of the items explored the perceived importance of having an extended vocabulary. Therefore, Q28 was excluded from further analysis, which, in turn, resulted in a higher internal reliability coefficient (.618).

Based on the results of principal component analysis, two scales, Spoken discourse competence and Written discourse competence, were found to include two components, and, at the same time, had very low Cronbach α values, .490 and .483, respectively. The examination of the relevant questionnaire items demonstrated that two latent dimensions, an active and a passive, were observable in both scales. This apparent duality reflected the way ETL professionals made similar distinctions between the four language skills, treating speaking and writing as active, or productive skills, whereas reading and listening were described as passive, or receptive skills (Kamiya, 2006; Llurda, 2000; Savignon, 1991; Thompson, 1996). The same latent dimensions, i.e., speaking and listening in the Spoken discourse competence scale, and writing and reading in the Written discourse competence scale, were identifiable in the data.

In the Written discourse competence scale, excluding the passive dimension of the component (Q8) from the analysis induced a comparatively higher internal reliability coefficient index (.662). However, in the case of Spoken discourse competence, in addition to the problem with the passive dimension (Q17), the content of Q26, upon closer investigation, seemed to be covering a facet of Strategic competence. Omitting Q26, the less related item of the two problematic ones, only induced a slight increase in the Cronbach α value (.548).

Nevertheless, both components definitely need more attention and further elaboration in future research owing to the active-passive polarity of the scale. All the

more so, since the teacher participants also indicated the distinct facets of discourse competence, being hesitant to find the appropriate place for the component in their orders of preference. Therefore, it would be advisable to separate the two latent dimensions in both discourse competence scales, i.e., to split Spoken discourse competence into Speaking and Listening, and Written discourse competence into Writing and Reading, to obtain a more comprehensive picture of students’ perceived importance of discourse competence.

Even though the principal component analysis only indicated the presence of one factor in the Actional competence scale, the Cronbach α value was relatively low (.602).

Upon examination of the relevant questionnaire items, it was revealed that whereas items Q7, Q19 and Q22 concentrated on language production, Q27 was focusing on comprehension. This duality might have been the reason behind the low reliability coefficient. Indeed, excluding Q27 from the analysis significantly increased the Cronbach α value of the scale (.691).

Finally, in the Strategic competence scale two components were identified, and the reliability coefficient was also extremely low (.158). Similarly, in the Sociocultural competence scale, despite its apparent homogeneity, the Cronbach α value was again relatively low (.485). However, examining the content of the individual items of either of the scales did not shed any more light on the root of the problem. Seemingly, the strategic component of communicative competence was a multi-layered construct, which was difficult to measure with one scale. That being said, excluding items Q10 and Q18 from the analysis did result in a higher Cronbach α value (.502).

However, owing to the multifaceted nature of both strategic competence and sociocultural competence, these scales also need further elaboration in future research

views of these two components. Additionally, interview data also confirmed these intricacies. Some of the teacher participants considered strategic competence as the mere acquisition of a limited number of communication strategies, whereas others claimed that it was a dynamic skill, vital in oral communication, which needed constant attention during the lessons. Likewise, sociocultural competence also had a mixed reception from the teachers, some regarding it as highly essential, and others ranking it as the least important component. These uncertainties in the treatment of strategic competence and sociocultural competence call for more explanation.

Finally, it must be highlighted that the contact variables scales did show an increase in the internal reliability coefficients compared to the results of the pilot phase.

However, the fact that several questionnaire items had to be deleted from the communicative competence scales resulted in three-item scales in five cases. As a result, these scales demonstrated slightly low internal reliability coefficients, which indicated that the questionnaire is in need of further elaboration. See Table 8 for reliability indices after correction methods were applied.

Table 10 Internal reliability coefficients and the number of components extracted after the application of correction methods in the main study

SCALES NO. OF

COMPONENTS

CRONBACH ALPHA

NO. OF ITEMS COMMUNICATIVE COMPETENCE

Linguistic competence / Vocabulary 1 .618 3

Linguistic competence / Grammar 1 .782 4

Spoken discourse competence 1 .548 3

Written discourse competence 1 .662 3

Actional competence 1 .691 3