Scottish Government

Pupils' test-based level attainments

The booklet pairs administered to the Gaelic pupils were booklets 17 and 28 at P5, and booklets 33 and 44 at P7. Every booklet contained assessment tasks from all three 5-14 curriculum outcomes - Earth and Space, Energy and Forces, and Living things and the processes of life. And each booklet contained tasks at three different 5-14 levels: Levels B, C and D at P5, Levels C, D and E at P7. At Levels C and D the tasks were common to both stages. Within a booklet tasks were grouped by topic. In booklet 17, for example, the topics were 'Magnets' (four different tasks), 'Electricity' (also four different tasks), 'Heat transfer' (two tasks), 'Making materials change' (five tasks), 'Water' (three tasks), 'Gases' (two tasks), 'Minibeasts' (three tasks), 'Animals' (two tasks) and 'The human body' (two tasks). Finally, to avoid the possibility of carry-over effects or test fatigue affecting some tasks more than others, each booklet was printed in three versions, differing only in the order of presentation of the topics. The topic order just given is that relating to booklet 17A.

A booklet was simply a vehicle for delivering assessment tasks, two booklets being required for the delivery of the single-level tests. Decisions about pupils' level attainments were based on their scores on the level tests, i.e. on their total scores on that set of tasks at a particular level in their booklet pair (the number of tasks involved varied from one level test to another and maximum task marks varied between 1 and 3, so that maximum possible test scores also varied from one level to another). In the main survey three different cut-off scores were applied to pupils' test scores: pupils scoring 80% or more of the marks were deemed to have shown 'very good' knowledge and understanding in science; pupils scoring between 65% and 79% were considered to have demonstrated 'well-established' knowledge and understanding, and those scoring 50% to 64% were classified as having made a 'good start' in knowledge and understanding. For the purpose of this study we focus on the criterion for 'well-established' science knowledge and understanding at a level, i.e. the 65% cut-off score. Pupils achieving a score of 65% or higher on a level test were considered to have shown 'well-established' knowledge and understanding in science at that level. Table 5 presents the results.

Table 5
Test outcomes for science knowledge and understanding
(% pupils achieving 65% or more on their level tests)

These are the proportions of tested pupils considered to have demonstrated evidence of 'well-established' knowledge and understanding in science

A comparison of Tables 4 and 5 illustrates the disparity between these test-based outcomes and those based on teachers' judgements of their pupils' science levels, the level distributions produced by the teachers being markedly more positive than those produced by the SSA tests. To explore this phenomenon further, Table 6 and Figure 2 show the level distributions for the pupils in the Gaelic samples for which both a test result and a teacher's judgement were available.

Table 6
Test outcomes versus teachers' judgements
(% pupils judged at the different levels by tests and by teachers)

To facilitate the comparison, each pupil in the Gaelic sample was allocated a level in the following way. If, for example, a P5 pupil had reached the 65% criterion cut-off score on the Level D test then that pupil would be classified as Level D. If the pupil had achieved the criterion score on the Level C test but not the Level D test then the level would be C. If the pupil had reached the criterion score on the Level B test but not the C or D tests then the level would be B. If the pupil had not reached the criterion score on any of the three level tests then the pupil would be considered to be at Level A (and could have been below A). Should a pupil have failed to reach the criterion score on one of the level tests but reached it at the level above, then the highest level was allocated (this happened in just 0.5% of cases at P5 and 2.4% of cases at P7).

As we see, while the teachers judged the majority of P5 pupils to be at Level B or Level C for science, with the majority at Level C, the tests judged almost half of the pupils to be at Level A (with some possibly below Level A). And, similarly, while the teachers judged virtually all their P7 pupils to be at Level C or Level D, with the majority at Level D, the tests put more than half of them at Level B (some possibly at Level A).

Figure 2
Test outcomes versus teachers' judgements
(level distributions of tests and teachers for the same pupils)

This raises a number of questions. Are teachers overly positive when they judge their pupils' attainments in science? Or are the levels, as operationalised in the tests, inappropriate? Or were the teachers and the tests judging different aspects of 'science'? Were teachers perhaps judging pupils' 'scientific skills' rather than their factual knowledge and conceptual understanding? Or were they judging their pupils on a subset of the knowledge and understanding curriculum - that subset of subject knowledge that they themselves had taught? These are difficult questions, for which there are no obvious answers right now. All we can say at this point is that the picture of difference for Gaelic-medium pupils reflects the situation in the country at large ³.