Scottish Government

Section H: Practically-based assessment results

H.1 The practical assessments

A variety of different types of skill were assessed in the practical component of the survey: listening skills, talking skills, ICT skills, skills in working with others and problem solving skills. It should be noted that the practical assessments, which were carried out by field officers, involved relatively small numbers of pupils in a convenient subsample of the survey schools, i.e. schools with at least 20 sample pupils at a single stage and within easy travelling distance of field officers' home bases. Typically, three or four pupils in any 'practical' survey school will have undertaken any particular type of assessment. See Sections B and C for further details of tasks and samples.

Throughout the following sections, findings are presented as sample statistics only.

H.2 Listening skills

Listening tasks were based on CD or video-based source material. Pupils listened to the relevant CD, or listened to and watched the video, before answering the test questions in their listening booklets. Since the listening tests contained only six or seven test questions at any particular level, it would not be useful to apply the usual cut-off scores to test scores to determine attainment levels. Therefore, for listening, attainment results are given as average item facilities (see Table H1), for all the items at a level across all relevant tests, and not as percentages of pupils 'working at' a given level.

Table H1
Average item facilities for listening, by stage*
(% pupils answering test items correctly, averaged over all items at the same level in all relevant listening tests)

* In total, around 300 pupils at each stage attempted listening tests, typically 3-4 pupils per school; in any school the pupils attempted the same listening test.

H.3 Talking skills

To assess pupils' talking skills, the field officers interacted individually with randomly selected pupils, engaging the pupil in a dialogue and eventually allocating a 5-14 level to the pupil's performance. Each pupil was given a choice of dialogue topic: their interests (hobbies, clubs, etc), TV/Films, school, or a piece of their own writing. The highest proportion of pupils, at just under 40% at every stage, chose to talk about their interests, followed by school and TV/Films, each chosen by around a quarter of the pupils, with own writing the least popular subject, at just over 10% of pupils. Boys tended in greater numbers to choose to talk about their interests and, at P7/S2, TV/Films, with girls tending in larger numbers to choose to talk about school. There were no differences in the attainment results related to topic choices.

It should be noted that no inter-rater agreement trials were conducted before the survey that might provide an indication of how dependable the field officers' necessarily subjective judgments about level-based performance in talk might be. This should be borne in mind as the results in Table H2 are reviewed.

Table H2
Level attainment results for talking, by stage*
(% pupils deemed by field officers to be at indicated levels)

* Between 250 and 325 pupils at each stage were individually assessed, typically 3-4 per school.

At all stages there were statistically significant gender differences in level profiles: girls' talking skills were generally more highly rated by the field officers than were the boys'.

H.4 Writer's craft

The attainment results for Writer's craft are shown in Table H3. Pupils were classified at a level by the field officers, who evaluated the pupils' writing productions using the national criteria for writing.

Table H3
Level attainment results for writer's craft, by stage*
(% pupils deemed by field officers to be at indicated levels)

* Between 250 and 300 pupils at each stage were assessed, typically 3-4 per school.

At P5 and P7 there were statistically significant the gender differences in level profiles: girls' writing skills were generally more highly rated by the field officers than were the boys'.

H.5 Knowledge about language

Tests of knowledge about language contained items at more than one level. Since the numbers of items at a level within any one KAL test were too small for the application of cut-off scores to determine a pupil's level be a sensible procedure, the results of the assessment are here presented in terms of average item facilities (the percentage of pupils answering an item correctly, averaged over all the items at the same level in all the tests administered at the stage concerned).

Table H4
Average item facilities for KAL, by stage*
(% pupils answering test items correctly, averaged over items at the same level in all relevant KAL tests: typically 15 items per level, but 27 at Level B and 9 at Level C at P3)

* Between 250 and 320 pupils at each stage were assessed, typically 3-4 per school.

H.6 Using ICT

Pupils were individually assessed for ICT skills, with the field officer using a checklist to record observations before coming to a conclusion about the pupil's attainment level in this area. Since no inter-rater agreement trials were conducted before the survey that might provide an indication of how dependable the field officers' necessarily subjective judgments about level-based performance in ICT might be, the results in Table H5 should be considered indicative only. That said, there is a clear pattern of stage development evident in the table, as one would expect to see.

Table H5
Level attainment results for ICT skills, by stage*
(% pupils deemed by field officers to be at indicated levels)

* Between 200 and 250 pupils at each stage were individually assessed, with typically 3-4 pupils assessed in each school.

H.7 Working with others

Table H6 presents the results of the assessment of pupils' skills when working with others. Again it should be noted that no inter-rater agreement trials were conducted before the survey that might provide an indication of how dependable the field officers' necessarily subjective judgments about pupils' abilities to work together might be. This notwithstanding, the assessment results are illuminating.

Table H6
Results of the assessment of the skills of working with others, by stage
(% pupils rated as indicated for various aspects by field officers)

The general picture is that 70-85% of pupils at all stages contributed new ideas, 80-95% showed at least some evidence of listening to others' ideas and building on them, 90% or more showed at least some interest in the task set, and 85-95% contributed to the discussion without dominating it or being disruptive. P3 pupils were less skilled than others in contributing own ideas and building on the ideas of others.

H.8 Problem solving

Field officers rated entire pupil groups for various aspects of problem solving as they worked together to solve their given problem. The results are shown in Table H7. Again it should be noted that no inter-rater agreement trials were conducted before the survey that might provide an indication of how dependable the field officers' necessarily subjective judgments about group problem solving behaviours might be. This should be borne in mind as the results in Table H7 are reviewed.

The general picture is that 90% or more of the groups at each stage showed some evidence or strong evidence of completing their given task successfully, with significantly fewer of the P3 groups compared with those at other stages showing strong evidence for this. For other aspects, it was the S2 groups as well as the P3 groups that attracted the most negative evaluations from the field officers, with 25-50% of the groups at these two stages demonstrating no evidence of the behaviour concerned.

Table H7
Problem solving attainment, by stage
(% groups judged as shown by field officers: 53 S2 groups, 60 P7, 59 P5, 58 P3)