Chapter I. Introduction, Rationale and Overview of the Book
Chapter 5 Research findings
5.5 Comprehension Measures
5.5.2 Comprehension Questions
E13 4.78 61.9 25 30.56
F1 22.33 69.04 50 47.12
F2 30.61 77.07 100 69.22
F3 34.92 100 75 69.97
F4 10.14 52.37 25 29.17
F7 24.04 54.75 75 51.26
F8 2.43 28.57 25 18.66
F10 52.3 100 75 75.76
F14 12.67 61.9 100 58.19
F15 13.32 61.9 100 58.4
F17 10.32 59.52 50 39.94
F18 12.67 40.47 50 34.38
F19 11.92 61.9 50 41.27
F22 8.35 61.9 0 23.41
G2 19.16 76.18 25 40.11
G5 20.46 69.04 25 38.16
G9 41.03 100 50 63.67
G10 26.21 50 75 50.4
G13 31.83 92.85 50 58.22
G14 16.24 76.18 50 47.47
G15 8.35 64.28 25 32.54
G19 35.01 100 50 61.67
Having another comprehension measure – questions to check understanding – proved essential to the present research. So comprehension was double-checked, although comprehen-sion questions proved to be more useful in establishing the learners’ comprehencomprehen-sion rates. The results of testing learners’ comprehension through comprehension questions can be found in the following section.
The means and standard deviations for the subtests and the whole test are shown in Table 34.
TABLE 34. Descriptive statistics (n – number of comprehension questions)
MEANS STANDARD DEVIATION
SUBTEST 1 (TEXT 1 – n=8) 4 (50%) 2
SUBTEST 2 (TEXT 2 – n=8) 4.4 (55%) 1.68
WHOLE TEST (n=16) 8.09 (52.5%) 3.24
TABLE 35. Comprehension scores (n=44, mean=47.25%, standard deviation=21.86)
Learners Total raw
scores Percentage (%) Learners Total raw
scores Percentage (%)
G19 16 100 D14 7 43.75
C4 15 93.75 F19 7 43.75
F3 15 93.75 G15 7 43.75
F10 15 93.75 D9 6 37.5
G13 14 87.5 E4 6 37.5
G14 14 87.5 F4 6 37.5
F2 13 81.25 F8 6 37.5
C10 12 75 D7 5 31.25
F7 12 75 E1 5 31.25
G9 12 75 E2 5 31.25
C6 11 68.75 E6 5 31.25
F1 11 68.75 E13 5 31.25
A16 10 62.5 B14 4 25
B9 10 62.5 C3 4 25
G5 10 62.5 E11 4 25
C5 9 56.25 F22 4 25
F14 9 56.25 C8 2 12.5
G10 9 56.25 C12 2 12.5
F15 8 50 E5 2 12.5
F17 8 50 C7 1 6.25
F18 8 50
G2 8 50
B1 7 43.75
D4 7 43.75
The means in Table 34 show that the learners did well at the comprehension questions test.
From this derives that the retelling gives a distorted view of the learners’ understanding because of the expectation that they should remember every event.
Table 35 shows the raw and percentage scores of individual learners.
The variability of learners’ percentage scores is very wide as there is 93.75% difference between the pupil with the best score (100%) and the worst one (6.25%). A standard item analysis was needed to see what might have caused problems for the pupils in the test items.
This analysis was conducted to determine the facility values (F. V.) of the questions and their discrimination indices (D. I.). The former indicate the difficulty level of an item, the latter show how well an item distinguishes among students at different levels of ability. The results of this analysis are shown in Table 36.
TABLE 36. Item analysis results (F. V. = facility value; D. I. = item discrimination index;
Q 1-16 = number of comprehension questions)
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8
F. V. .68 .75 .34 .34 .25 .06 .47 .77
D. I. .6 .66 .73 .66 .6 .2 .93 .66
Q9 Q10 Q11 Q12 Q13 Q14 Q15 Q16
F.V. .59 .52 .9 .56 .47 .11 .29 .93
D.I. .93 .73 .26 .66 .86 .26 .6 .2
It is clear from Table 36 that roughly half of the questions – Questions 1, 2, 8, 9, 10, 12, and 13 – were the ones that most learners could answer (facility values ranging from 52% to 77%).
Questions 11 and 16 turned out to be extremely easy as more than 90% of the learners gave correct answers to them; therefore, they discriminated least between the learners (discrimination indices of .26 and .20). Questions 3, 4, 5, 6, 14, and 15 proved to be difficult, as the percentage of learners giving correct answers to these questions ranged between 6% and 34%.
What comes next is a brief analysis and discussion of the content of the comprehension questions to show what made them easy or difficult. The final English versions of the comprehen-sion questions are provided in Appendices 14 and 15.
First of all, Questions 11 and 16 were very easy for all the learners as more than 90% of them were able to give correct answers to them. They asked about the characters of the second story about Harry Hippo – ‘Who did Harry live with?’ and ‘Who was happy at the end of the story?’. Other questions about the characters of the stories also proved to be easy – Questions 1, 2, 9, 10, 12. There was one exception in this tendency for the questions on characters to be easy:
Question 5. The difficulty might have been caused by the word hunter because many learners simply did not understand its meaning and tried to deduce it from the context with more or less success, for example:
The pigeon lived in a tree. He used to play with the water. He was very happy. But one day the woodcutter came and he wanted to cut the tree. The pigeon didn’t let him do it. He didn’t want his tree, his home to be cut out. Finally, the pigeon and the woodcutter became good friends.
(Learner B1)
On the other hand, all the questions – 3, 4, 6, 7, 14, and 15 – asking about the events of the stories turned out to be difficult for the learners, except for one Question 13. This latter one inquir-ing about how his friends wanted to help Harry with his problem happened to be easy in terms of content as those who understood that Harry had a toothache, i.e. his tooth hurt),could easily give a response to this question – partly based on their schemata of actions in case somebody has a toothache.
Both questions asking about the themes of the stories – friendship and helping each other – proved easy for the learners because most of them could give positive answers to these questions.
In sum, the analysis of the content of the comprehension questions showed that understand-ing and rememberunderstand-ing events of the stories caused the greatest difficulties in learners’ comprehen-sion. It was less difficult for them to respond to questions on the characters of the stories, and the least difficult questions were those that asked about the main topics of the two texts.
It is possible to compare the schools where the participants of the miscue study came from to see if any of them is better than the others. This can be done through the examination of the mean scores on comprehension questions achieved by learners in different schools. Table 37 presents the means and standard deviation by schools. Only one learner was coded in School A therefore the mean is the actual score of that pupil and the standard deviation is 0. Because there was only one data registered in School A, it cannot characterize the group and therefore is not considered in the comparison below.
TABLE 37. Mean results (max. score = 16) of comprehension questions by schools
Schools Means Standard
Deviation
A 10 0
B 7 2
C 7 4.75
D 6.25 0.75
E 4.57 0.89
F 9.38 2.93
G 11.25 2.75
Table 37 shows that learners in Schools F and G – two secondary schools in two towns of Transcarpathia – in general managed to understand the comprehension questions and answer them correctly (means 9.38 and 11.25). Pupils in Schools D – a town primary schools – and School E – a village primary school – scored lowest on the comprehension questions (means 6.25 and 4.57).
The standard deviations in both schools indicate that the learners demonstrated a very equal and balanced performance on this comprehension measure. Although learners in School B – a town secondary school – and School C – a village primary school – had the same mean score (7), the de-viation from this mean was larger in School C. In conclusion, learners in secondary schools scored better at the comprehension question test than those in primary schools.