1. Overview
1.1 Survey objectives
The 2008 Scottish Survey of Achievement (SSA) was required to:
assess and report nationally on pupils' attainment in mathematics and numeracy by 5 to 14 levels at the P3, P5, P7 and S2 stages - referred to as "understanding mathematics"
investigate and report nationally on pupils' ability to use numeracy skills in everyday contexts at the P3, P5, P7 and S2 stages - referred to as "using mathematics"
assess and report nationally on pupils' core skills (ICT, working with others and problem solving) applied in a mathematics context at the P3, P5, P7 and S2 stages
gather information and report nationally on pupils' and teachers' experience of learning and teaching in mathematics, along with their views about this experience
gather and report nationally on teachers' judgements of pupils' reading, writing and mathematics attainment levels
compare attainment and progression in mathematics and numeracy across the four stages and between girls and boys
compare attainment and progression in mathematics and numeracy in relation to deprivation
assess changes in performance over time where possible
and, as secondary objectives, to:
- assess and report, for 'opted-in' local authorities, pupils' attainment in mathematics and numeracy by 5 to 14 levels at the P3, P5, P7 and S2 stages
- moderate a national sample of class-based writing (in a mathematics context, where possible) at the P3, P5, P7 and S2 stages
- provide information to inform mathematics education policy and support opportunities for the enhancement of mathematics teaching in Scottish classrooms. For example, by modelling new approaches to the assessment of mathematics in line with the Curriculum for Excellence.
The following practical constraints were imposed:
the duration of an assessment session was to last no more than 30-40 minutes at P3/P5 and 50-60 minutes at P7/S2
the maximum that any individual pupil would be asked to undertake was two written booklets from either understanding mathematics or using mathematics, plus a questionnaire and either one of the three core skills elements of the practical together with a mental mathematics assessment or a piece of class-based writing
the schools that had been invited to participate in the pre-testing of assessment material for the survey would not be selected for survey involvement, unless unavoidable
the total number of pupils selected for testing in an individual school was to be in proportion to the size of the school roll, and was to average 13 for primary schools and 30 for secondary schools
a maximum of twelve pupils per school were to be selected for participation in the practical elements of the survey
1. 2 The scope of survey enquiry
1.2.1 Pencil and paper assessments
Pencil and paper assessment was employed for the assessment of understanding mathematics and using mathematics. In the first case, test booklets comprised conventional 'atomistic' test items, spanning three consecutive levels and covering the 5-14 mathematics curriculum for the stage concerned. A total of 120 different test booklets were administered, 30 per stage. Individual survey pupils were randomly allocated two booklets containing items at levels appropriate to the stage. Further details are given in Section 3. In the case of using mathematics, which was essentially a pilot study to inform future task development under Curriculum for Excellence, a handful of multi-item tasks were administered at each stage. These offered pupils an everyday context in which to demonstrate their ability to apply their numeracy skills. Again, individual survey pupils were randomly allocated two test booklets containing tasks at appropriate levels for the stage.
1.2.2 Practical assessments
A target of 300 pupils per stage were required for each of the three elements of the practical assessments (working with others, ICT, maths investigation) as well as completing a mental maths assessment. Sampling was further constrained by the practicalities of using field officers to carry out the assessments. Although clearly not an ideal methodology from a statistical point of view, the main motivation for the practical assessments was to provide exemplification and professional development in practical assessment methodologies for schools. The information to be gathered was indicative only.As in previous years, the practical part of the SSA was carried out with a sub-sample of pupils in a sub-sample of the schools in the main survey (cost and logistics being too challenging for all of the survey schools to be included).
1.2.3 Writing
All survey schools were invited to be involved in this survey component, by submitting a piece of class-based writing for each of a stratified, randomly selected 25% of their survey pupils, along with a level judgement for the writing. Schools were given two options for selecting pupils' writing examples:
a short or extended piece of functional writing in the context of mathematics, from the pupil's folio or generated specifically for the survey ·
an extended piece of writing of any genre or context from the pupils' folio.
A subset of submitted and rated writing was evaluated independently by teachers from other schools. All useable submitted and rated writing was independently evaluated by a team of teachers recruited from across Scotland as part of an inter-rater study.
1.2.4 Teachers' level judgements
Continuing previous survey practice, the survey schools were invited to submit class teachers' level judgements in reading, writing and mathematics for each of their sampled pupils.
1.2.5 Teacher and pupil questionnaires
In order to provide a context against which to reflect on and interpret the attainment findings, both teachers and pupils were invited to complete questionnaires about their mathematics teaching/learning experiences and subject attitudes.
1.3 Reporting strategy
In May 2008, following the field work for the 2008 survey, the Scottish Government confirmed that the SSA provided statistics of national importance and as such should become a National Statistic. As a first step towards this the survey was designated as an "official statistic". This meant that, for the first time, the SSA would be fully managed within the National Statistics Code of Practice.
The practical implications included the following decisions about publication of 2008 survey findings:
understanding mathematics (i.e. mathematics and numeracy attainment), teachers' judgements for mathematics, and questionnaire results would be published in the main 'headline' report;
teachers' judgements for reading and writing would not be included in the headline report, but would be published within the supporting evidence;
the using mathematics findings would not be published alongside the official statistics, but would be published later in another form, given its experimental nature looking forward to Curriculum for Excellence;
the results of the practical investigations would also be reported separately, given their primary role in providing exemplification and professional development through the involvement of practising teachers as itinerant field officers;
the results of the writing assessment would also be published separately, given their specific importance in informing future assessment practice in this area under Curriculum for Excellence.
For these reasons, the survey headline report includes selected findings from the principal assessment of mathematics and numeracy, from teachers' judgements, and from the teacher/pupil questionnaires. Other aspects of pupil performance assessed in the survey are reported separately.
The following sections document the sampling strategy used for the selection of school and pupils for survey participation, describes the strategy employed for the paper-based assessment of mathematics and numeracy ( understanding mathematics), and overviews the methodology applied for attainment estimation.