Scottish Government

4. Core skills

4.1 Introduction

The core skills reported in this chapter are reading, writing and use of ICT.

Fifteen reading tasks were newly developed for use in the survey, three each at Levels A to E. At all levels, writing tasks were also devised that used as stimulus material the same texts as the reading tasks. There were thus also 15 different writing tasks in total. Keeping within the subject theme of the survey, the source texts for the reading and writing tasks focused on some issue or personage in mathematics, though without any requirement for prior knowledge and understanding of mathematics per se.

Six different ICT tasks were also newly developed for use, each task carefully set in a 'mathematical' context. These were administered in schools by trained field officers (see Chapter 1 for general information about the practical assessment).

In principle, all the survey schools participated in the assessment of reading and writing. Mathematics assessment, though, was given priority, so that where the number of pupils in a school's sample was not large enough to accommodate both mathematics and reading/writing assessment, then that school took part in the mathematics assessment only. Among the schools that agreed to take part in the survey, with enough pupils to take part in reading and writing assessment as well as mathematics assessment, 90-95% did so at each stage: 200-250 schools at each primary stage and just over 150 schools at S2. Over 40% of the schools at each stage also participated in the assessment of pupils' practical skills, including ICT skills (see Appendix B for further details of samples and task administration).

4.2 Reading

4.2.1 The reading tasks

Each of the 15 reading tasks administered in the survey took the same general form as those typically used in AAP English surveys, i.e. a source text, or series of related texts, plus a set of comprehension questions. In this case, however, the texts focused on mathematical topics or issues, as the titles in Table 4.1 suggest. Table 4.2 describes one task at each of Levels A, C and E.

The number of test items varied across tasks, from 21 per task at Level A rising to 30 per task at Level E, and featured a variety of different formats, including multiple choice, summary completion, sentence completion, and short response. Items sharing common formats were grouped into sections, as illustrated in the task examples given in Table 4.2.

Reading tasks were classified by 5-14 level on the basis of the perceived level of their source texts. No prior mathematical knowledge was needed in order to respond successfully to the tasks, although inevitably prior knowledge might occasionally have helped.

Tasks were presented to pupils in the form of test booklets, a single reading task comprising a test booklet. Individual pupils attempted two test booklets, one at each of two different levels: P3 pupils attempted reading tasks at Levels A and B, P5 pupils attempted tasks at Levels B and C, P7 and S2 pupils attempted tasks at Levels C and D, Levels C and E or Levels D and E. The reading passages for the two tasks were presented within a separate source booklet.

Completed booklets were processed centrally, and marked by students, who had received training and induction into the necessary procedures. Each booklet was attempted by between 250 and 400 or so pupils, the number varying by booklet and stage. In total, just under 10750 booklets were analysed, emanating from around 5500 pupils in just under 800 schools.

4.2.2 Pupils' reading attainment

The attainment results for reading were produced by applying the usual 65% cut-off score to the pupils' total task scores, as described in Chapter 1, and then averaging the proportions of 'secure' attainers over the three tasks at each level. The resulting proportions of pupils classified as 'secure' at the level concerned (the level of the task) are given in Table 4.3, and illustrated in Figure 4.1.

* 1000-2000 pupils per stage in total; margins of error for the estimated proportions vary between 1.5 and 3.5 percentage points

As Table 4.3 shows, around two-thirds of the P3 pupils demonstrated secure attainment at Level A, and over half also at Level B. At P5, just over 80% of the P5 pupils demonstrated secure attainment at Level B and over half also at Level C. At P7, the picture is similar for Levels C and D (80% classified as secure at Level C and 60% at Level D also). At S2, 85% of the S2 pupils were classified as secure at Level C, just under three-quarters also at Level D and just under 30% at Level E. Thus, at all levels there is a clear picture of stage progression, this progression being weakest between P7 and S2 at Level C, where attainment proportions were already high at P7.

Interestingly, while the reading attainment results in the 2003 AAP Science survey ⁶, where reading passages were set in 'scientific' contexts, resembled those in the 2001 AAP English Language survey ⁷, the attainment results in this 2004 survey differ. Table 4.4 presents the relevant data for P7 and S2 (the two stages that featured in all three surveys) at those levels assessed on all three occasions.

Figure 4.1
Reading in a mathematical context: attainment across the stages
(% pupils achieving 65% or more marks per task, averaged over three tasks at each level

As Table 4.4 shows, markedly higher proportions of the P7 pupils in the 2004 survey were deemed 'secure' at both Levels C and D compared with 2003 and 2001, with a similar picture emerging for the S2 pupils at Level D also. At Level E, however, the pattern is less clear. The differences could be interpreted as indicating real differences in reading attainment between 2003 and 2004 at the different stages and levels or, alternatively, as reflecting the influence of different reading contexts. But they could equally, or perhaps more likely, be the result of topic effects, or even more simply of differences in task difficulty, given the very small numbers of tasks used at each level in the surveys of 2003 and 2004 (the fewer tasks that are used the more influence that task topic and task difficulty will have on average attainment).

* Three tasks per level in 2003 and 2004, four to six per level in 2001

As far as gender differences are concerned, Table 4.5 shows a tendency for the girls to have produced better performances in general than the boys (differences of five points or more reach statistical significance). This is a common finding in surveys of reading attainment, but one which did not emerge in the 2003 AAP Science survey, where the stimulus passages used were set in scientific contexts (although still requiring no prior science knowledge for successful completion of the reading tasks themselves). However, only three tasks featured at each level here, and gender gaps varied in size from one task to another within any level. Caution should therefore be exercised when attempting to interpret these overall findings.

* 500-1000 pupils in total per gender per stage

4.3 Writing

4.3.1 The writing tasks and writing evaluation

Each of the 15 reading tasks had an associated writing task, stimulated by the same reading passage (see Tables 4.6 and 4.7).

Each writing task was loosely related to the relevant reading text, but could be attempted without reading the passage: in other words, the writing tasks were associated with, but not dependent on, the reading tasks. The tasks were all within a mathematical context but generally related to how we might use numbers in our everyday lives. The general format of the writing task was constant throughout the levels: a short statement, followed by the actual task with a number of bullet points to provide a degree of support.

Each pupil was given a choice between two different writing tasks, relating to the reading tasks the pupil had been assigned and presented within the reading source booklet. Teachers were asked to organise a third assessment session dedicated to the writing. The writing was evaluated using the holistic scheme shown in Table 4.8, a scheme that had been used for the same purpose in the 2003 Science survey.

In total, 100 of the field officers who had undertaken the practical assessments in the survey volunteered to participate in the evaluation exercise. While most of the evaluators (85) were practising primary class teachers, some secondary mathematics teachers were also involved. In June 2004, the writing evaluators were given some orientation for the writing evaluation activity, after their post-survey practical assessment debriefing. During the orientation meeting they first reviewed a small sample of scripts as a group, applying the holistic evaluation scheme. For various reasons, it was later in the year before scripts were actually sent out to the evaluators for judgment. At this time, and with access to exemplification materials and written guidance (see Table 4.9), they each independently evaluated their assigned scripts.

The evaluation of writing is inevitably a subjective process, leading to natural concerns about the comparability of rating standards between different raters. It cannot be assumed that different evaluators judging the same script will come to the same decision about writing relevance or quality, whatever the nature of the evaluation scheme that is used and however much guidance and exemplification material they are given. This issue was explored within the 2003 science survey, and was explored again here.

In total, across the four stages, 3192 pieces of writing were evaluated, using the best fit scheme. Among these, 3110 were each evaluated by three different raters, working independently in rater groups, with each group rating up to 200 scripts from one booklet at one or two stages. Of the 3110 writing booklets randomly selected for evaluation, 82% were classified in the same way by at least two of the three raters, the proportion dropping to around 25% for unanimous agreement (Table 4.10 provides the rater agreement findings for the various different tasks).

* % scripts independently judged at the same level by at least two of three different raters, with (in brackets) the unanimous agreement rate

Of the 3110 booklets, 10 in fact contained no writing. Among the 3100 pieces of writing that were able to be judged, 1-3%, varying by stage, were considered not to be of the correct genre and a further 1-5% were considered to have irrelevant content. The number of remaining 'valid' scripts for which there was majority or unanimous agreement about appropriate levels was 2372.

4.3.2 Pupils' writing attainment

The attainment findings presented in this section are based on those scripts of the appropriate genre, with relevant content, that were allocated the same level by at least two of the three raters that evaluated them, i.e. 2372 scripts. Table 4.11 presents the general findings in terms of stage profiles, with Figure 4.2 illustrating the profile differences.

The majority of the P3 pupils produced writing at Levels A and B, the majority of the P5 and P7 pupils produced writing at Levels B and C, while the majority of the S2 pupils produced writing at Levels B, C and D. The stage progression is particularly striking between P3 and P5: whereas more than 60% of the P3 pupils were deemed to be working at Level A or below, the corresponding proportion at P5 was under 25%, at P7 under 15% and at S2 under 10%. The proportions of pupils deemed to be working at Levels D or E at P5, P7 and S2 were 5%, 15% and 32%, respectively.

Table 4.12 presents the results of the writing evaluation for boys and girls separately, and reveals a very clear gender gap in favour of the girls at every stage (all the profile differences reach statistical significance at the 0.1% level). At S2, just over 20% of the boys produced work considered to be at Level D or Level E compared with just over 40% of the girls. At P7, 40% of the boys produced work at Levels C, D or E compared with 60% of the girls. At P5, two-thirds of the boys produced writing deemed to be at Level B or above compared with over 80% of the girls. At P3, around 80% of the boys produced work at Level A, B or C compared with around 90% of the girls; around twice the proportion of boys as girls produced writing considered to be of a quality below Level A (19% versus 11%, respectively).

* Figures based on valid scripts (correct genre and relevant content) with at least majority rater agreement about appropriate level allocations

Figure 4.2
Writing in a mathematical context: attainment across the stages
(% pupils deemed to have demonstrated attainment at the given levels, with at least majority rater agreement)

The results shown here for writing in a mathematical context reveal a more positive picture of attainment than that reported for writing in a scientific context, as assessed in the 2003 Science survey ⁸. For example, in the science survey just under half the S2 pupils, two-thirds of the P7 pupils and over 80% of the P5 pupils were classified as having produced writing at Levels A or B, whereas in the 2004 survey these proportions have dropped to under a third, one half and under two-thirds, respectively (the writing attainment of P3 pupils was not assessed in 2003). The apparent improvements are less likely to be attributable to the change in context than they are to other factors, including differences in the tasks and their administration, differences in the ways writing evaluators were trained, and differences in data validity and reliability.

For example, in the 2004 survey the 'maths context' tasks were only loosely related to the context of the reading passages, and good writing production did not depend on pupils having first read and understood the passages concerned. In the 2003 survey pupils were obliged to read the relevant passage and reproduce some of the material contained in the passage in their writing. In addition, the 2004 tasks were formulated in a different style, more familiar to pupils. In the 2004 survey pupils had a choice of topic, since they were required to produce a single piece of writing, whereas in 2003 pupils were obliged to produce two pieces of writing, related, respectively, to their two reading tasks.

* Figures based on valid scripts (correct genre and relevant content) with at least majority rater agreement about appropriate level allocations

In the 2004 survey teachers were advised that sufficient time should be provided for pupils to undertake their single writing task comfortably, and the writing was carried out in a dedicated third test session. In the 2003 science survey the writing tasks were undertaken within the reading test sessions, after the stimulus reading tasks had been completed.

In the 2004 survey the writing evaluators were given more training support than had been offered to their colleagues in 2003, and in particular were given substantially more exemplification material. The guidelines for assigning levels were also simplified. The fact that only 15% of the evaluators in 2004 were secondary subject teachers (of mathematics) compared with over 30% in 2003 (secondary science teachers) will also have had an effect, since subject teachers, other than those of English, are not generally involved in assessing writing quality, and this lack of familiarity is rendered even more difficult for them when they are faced with factual inaccuracies in the writing.

Finally, the attainment results for 2004 will be more valid and reliable than those for 2003, since they are based on level classifications independently agreed by at least two evaluators working independently, whereas in 2003 they were based on the judgment of single evaluators.

4.4 Using information technology

4.4.1 The tasks and their administration

Six different CDROM-based tasks were developed for use in the survey, all designed to assess the same range of ICT skills, focusing on the strands 'Using the technology', 'Creating and presenting', 'Collecting and analysing' as described in the 5-14 National Guidelines for ICT⁹. Each task comprised a series of activities, and was planned to take about 30 minutes of testing time. The activities included modifying graphics and editing text (using Word), and entering data into a spreadsheet and manipulating that data (using Excel). Table 4.13 overviews the general format of the ICT tasks.

The tasks were administered by the itinerant field officers, who also administered the mathematics practical tasks described in Chapter 2 (see Chapter 1 for details of field officer numbers and training). Four of the tasks were administered at P3/P5, and four at P7/S2, with two tasks common to all four stages. Typically, in any one school four pupils undertook ICT tasks, each pupil attempting a different one of the four tasks assigned to that stage. In total, the field officers assessed 1320 pupils in 335 schools (40% of the survey schools).

Pupils worked on laptop computers provided by the field officers. They were given a few minutes to familiarise themselves with these before the assessments began, using the mouse to control the cursor. As they did so, they were questioned by the field officers to explore their familiarity with using a keyboard and mouse, with using a PC with Windows, and with using Word and Excel. The results of this enquiry are given in Table 4.14.

As far as the technology is concerned, and on the basis of pupils' self-reports, Table 4.14 suggests a high degree of familiarity among pupils at all four stages in the use of a keyboard and mouse, with much lower levels of familiarity among the P5 and P3 pupils with the use of a PC with Windows. As to the software that featured in the ICT tasks, the majority of P7 and S2 pupils claimed to be familiar with Word (80-90%), but familiarity rates were lower for Excel, at under 60% of the pupils at S2 and around one-third at P7. Among the younger pupils familiarity with the software was even less widespread: half the P5 pupils and over two-thirds of the P3 pupils claimed not to be familiar with Word, while almost 90% of the P5 pupils and virtually all the P3 pupils claimed to be unfamiliar with Excel. Clearly, these findings should be borne in mind when the attainment results are reviewed.

After the brief laptop familiarisation and questioning, the pupil under assessment was given the relevant task sheet and asked to follow the instructions. If the pupil was struggling, the field officer directed him/her to the next question or brought the session to a close. Individual activities were rated as 'completed successfully and independently', 'completed successfully with some support' or 'no real attempt'. At the end of the task the pupil's task sheet was collected in and filed with the checklist for later analysis.

Before the attainment results are presented, some of the problems that arose during this exercise merit comment. Firstly, as already noted, pupils were constrained to work with the laptops provided by the field officers, despite the fact that such a laptop environment was unfamiliar to most of them. Secondly, the tasks were only available for use on PCs, so that those pupils who were not familiar with the PC environment would have been disadvantaged (see Table 4.14). Thirdly, the text handling and data manipulation tasks required use of Word and Excel, software which was unfamiliar to most of the younger pupils (again see Table 4.14). Fourthly, there were occasional technical difficulties, which meant that some pupils could not actually attempt one or more of the activities.

4.4.2 Pupils' ICT performance

In addition to actively using the technology to access and save files, pupils were questioned to explore their passive file handling knowledge. The results of both enquiries are given in Table 4.15, which shows clear evidence of stage progression, particularly between P3 and P5, and between P5 and P7/S2.

With rare exceptions, all the P7 and S2 pupils were able to open a folder and file without help, and to scroll the file. High proportions were also able to save the text under a new file name and understood why this is good practice. At P5, around 90% of the pupils could open a folder and file and scroll the file; two-thirds could save the text under a new name and almost 60% knew why they should do this. At P3, two-thirds or more of the pupils could open a folder and file, and scroll the file. Just 30% knew how to save the text under a new file name and why this is a useful thing to do.

The term ' CDROM' and the format of a typical web address were familiar to 90% or more of the P7 and S2 pupils, to three-quarters of the P5 pupils and to around 45% of the P3 pupils. However, when asked how they might take a Word file home, or to another class, to work on, the majority of pupils at all stages suggested printing the document: two-thirds of the P3 pupils rising to over 90% of the pupils at S2. High proportions of the P7/S2 pupils (80-90%) also suggested copying the file onto a floppy disk or CDROM, or sending it as an email attachment (66% and 76%, respectively). Just 40% of the P5 pupils and 20% or fewer of the P3 pupils suggested one or both of these methods. Around a third of the S2 pupils and a quarter of the P7 pupils suggested accessing the file over a network; at P5 the proportion was just under 15% and at P3 under 10%. Naturally, however good their knowledge about different ways of transferring and accessing files might be, the pupils' responses to this question must surely have been tempered by what they were actually able to do, or typically did, in their schools and homes on a day-to-day basis.

* 250-400 pupils per stage; in each case the percentage "correct" is based on all pupils embarking on the task, whether or not they actually reached the question/activity concerned.
** This is not the wording actually used with pupils

Table 4.16 presents the results of the assessment of pupils' abilities to manipulate text and graphics in Word. The pupils were asked to do whatever they thought appropriate to improve the appearance of a given text for a given purpose (the purpose varied by task). The text concerned was held in a Word file - the file they had earlier opened, scrolled and saved under a new name - and contained one graphic (a picture).

As Table 4.16 shows, high proportions of the S2 pupils (85% or more) successfully added some text of their own, changed the appearance of fonts, added emphasis with italics or bold face, underlined some text and/or centred the heading/title. Almost three-quarters of the S2 pupils also used the spell checker. Under half justified the text and/or added bullet points or indentation. Just under 30% introduced text columns, and about a quarter made other types of improvement, most using Word art. One S2 pupil introduced headers and footers. Most P7 pupils (70-90%) and P5 pupils (50-85%) also typed text, changed font appearance and/or added emphasis. Just over 60% of the P7 pupils and around a third of the P5 pupils centred the text heading and/or applied the spell checker. A quarter of the P7 pupils justified the text and/or made other improvements, mainly using Word art (one P7 pupil previewed the document); at P5 proportions were lower. At P3, almost two-thirds of the pupils added text, and just over a quarter demonstrated the ability to alter the appearance of text by changing font, font sizes and/or font colour. Over 15% of the P3 pupils showed how to emphasise text through use of italics, boldface and underlining.

* 250-400 pupils per stage; in each case the percentage successfully demonstrating the activity is based on all pupils embarking on the task, whether or not they actually voluntarily engaged in the activity concerned.
** This is not the wording actually used with pupils.

The proportions of pupils who were able to move an existing graphic rose from 55% at P3 to over 95% at S2, whereas the corresponding proportions able to resize a graphic ranged from a lower 30% at P3 to over 90% at S2. The stage difference was marked also for the skill of inserting a new graphic (a picture) into the text: just over 10% of the P3 pupils managed to do this, rising to over 80% at S2. Over a quarter of the S2 pupils successfully wrapped text around a graphic: this compares with 15% at P7 and under 5% at P5 and P3. Just under one in ten pupils at P7 and S2 demonstrated other skills here, including shading, adding titles, and so on.

The final set of ICT skills that featured in the assessed comprised spreadsheet skills. Table 4.17 presents the results.

* 250-400 pupils per stage; in each case the percentage successfully demonstrating the activity is based on all pupils embarking on the task, whether or not they actually engaged in the activity concerned.
** This is not the wording actually used with pupils.

The evidence is that spreadsheet skills are in general less well-developed than are basic word processing skills, but once again stage progression is clear. Two-thirds of the S2 pupils were able to enter data into a spreadsheet table, dropping to around one-third at P5 and P7, and to less than a fifth at P3. One-third of the S2 pupils were successfully able to create a required chart using the tabulated data: higher proportions were able to label the axes appropriately (42%) and to provide a chart title (48%). Proportions were lower among younger pupils, falling to under 5% at P3 for these skills.

There was no indication in the performance data of any difference in the ICT skills of boys and girls.

4.5 Summary

In total, 15 reading tasks, 15 writing tasks and six ICT tasks were administered in the survey, all set in mathematical contexts.

The reading tasks were level-based, with three different tasks per level, and over 5500 pupils in just under 800 schools were assessed. Attainment results were produced by applying the usual 65% cut-off score to the pupils' total task scores, as described in Chapter 1, and then averaging the proportions of 'secure' attainers over the three tasks at each level. At all levels a clear picture of stage progression emerged. Around two-thirds of the P3 pupils demonstrated secure attainment at Level A, and over half did so at Level B also. At P5, just over 80% of the P5 pupils demonstrated secure attainment at Level B and over half also at Level C. At P7, 80% of the pupils were classified as secure at Level C and 60% at Level D also. At S2, 85% of the S2 pupils were classified as secure at Level C, just under three-quarters also at Level D and just under 30% at Level E. The girls produced better performances in general than the boys at all stages.

In total, across the four stages, over 3000 pieces of writing were evaluated, using a best fit scheme. Each script was independently evaluated by three different raters, all practising teachers, and the 'majority' inter-rater agreement rate (same judgements offered by at least two of the three evaluators) was just over 80%. According to the evaluators, the majority of the P3 pupils produced writing at Levels A and B, the majority of the P5 and P7 pupils produced writing at Levels B and C, while the majority of the S2 pupils produced writing at Levels B, C and D. The stage progression was particularly striking between P3 and P5: whereas more than 60% of the P3 pupils were deemed to be working at Level A or below, the corresponding proportion at P5 was under 25%, at P7 under 15% and at S2 under 10%. The proportions of pupils deemed to be working at Levels D or E at P5, P7 and S2 were 5%, 15% and just over 30%, respectively. There was a very clear gender gap in favour of the girls at every stage.

The ICT tasks covered a range of relevant skills and knowledge, and were administered in the schools by trained field officers, with typically four pupils assessed in each 'practical' school (40% of the survey schools), pupils working individually on their assigned tasks. In total, around 1300 pupils were assessed in almost 350 schools, working on laptop computers provided by the field officers. On the basis of the pupils' self-reports, high proportions of pupils at all four stages were familiar with use of a keyboard and mouse, but there were much lower levels of familiarity among the P5 and P3 pupils with the use of a PC with Windows. As to the software that featured in the ICT tasks, the majority of P7 and S2 pupils claimed to be familiar with Word (80-90%), but familiarity rates were lower for Excel, at under 60% of the pupils at S2 and around one-third at P7. Among the younger pupils familiarity with the software was even less widespread: half the P5 pupils and over two-thirds of the P3 pupils claimed not to be familiar with Word, while almost 90% of the P5 pupils and virtually all the P3 pupils claimed to be unfamiliar with Excel.

At all stages, high proportions of pupils were able to open a folder and file and scroll the file (90% or more at P5, P7 and S2, and two-thirds at P3). While high proportions of the older pupils could also save the text under a new file name, just 30% of the P3 pupils could do this. When asked how they might work on the file in a different location, the majority of pupils at all stages suggested printing it, although high proportions of the P7/S2 pupils (80-90%) also suggested copying the file onto a floppy disk or CDROM, or sending it as an email attachment (65-75%). Just 40% of the P5 pupils and 20% or fewer of the P3 pupils suggested one or both of these methods. Around a third of the S2 pupils and a quarter of the P7 pupils suggested accessing the file over a network; at P5 the proportion was just under 15% and at P3 under 10%.

High proportions of the S2 pupils (85% or more) and most pupils at P7 and P5 (70-90%) demonstrated competence in basic word processing skills: adding new text and changing the appearance of fonts, adding emphasis through italics, etc. At P3, almost two-thirds of the pupils added text, and just over a quarter demonstrated the ability to alter the appearance of text by changing font, font sizes and/or font colour. Over 15% of the P3 pupils knew how to emphasise text through use of italics, boldface and underlining. Under half the S2 pupils, a quarter of the P7 pupils and lower proportions of pupils at P3 and P5, justified the text and/or added bullet points or indentation. The proportion of pupils who were able to move an existing graphic in a text file rose from 55% at P3 to over 95% at S2, whereas the corresponding proportions able to resize a graphic ranged from a lower 30% at P3 to over 90% at S2. Just over 10% of the P3 pupils were able to introduce a new picture into a text, rising to over 80% at S2. Over a quarter of the S2 pupils successfully wrapped text around a graphic: this compares with 15% at P7 and under 5% at P5 and P3.

Two-thirds of the S2 pupils were able to enter data into a spreadsheet table, dropping to around one-third at P5 and P7, and to less than a fifth at P3. One-third of the S2 pupils were successfully able to create a required chart using the tabulated data, falling to under 5% at P3 for these skills.

There was no indication in the performance data of any difference in the ICT skills of boys and girls.