Programme for International Student Assessment (PISA) 2003: Initial Report on Scotland's Performance in Mathematics, Science and Reading
Chapter 1: Introduction
The OECD established its Programme for International Student Assessment (PISA) to assess the performance in reading literacy, mathematical literacy and scientific literacy of 15 year old students in its member countries. The results from the studies contribute school outcome measures to the OECD's educational indicators programme 2. Since PISA's inception its scope has broadened to include non-OECD countries as well. These non-OECD countries are referred to as "partner countries".
The first PISA survey took place in 2000 3, the second in 2003, and work has started on the third, to take place in 2006. Each study focuses in turn upon one of the above three literacies, while allocating a subsidiary role to the other two. Consequently, the whole series has a major cycle of nine years. In PISA 2000 the majority of assessment items were in reading literacy and the minority in mathematics and science literacy, while in PISA 2003 the majority are in mathematical literacy, the other two domains playing the minor roles. In PISA 2006 the majority are in scientific literacy. To provide continuity between studies and a measure of change over the three-year intervals, a proportion of the test items from previous studies are re-used in subsequent ones.
The scope of PISA has grown. In 2003 an additional, subsidiary, domain of problem solving was added to the other three, though this is not to be carried forward to 2006. (The results of the problem-solving tests are not reported in this initial report but will be reported on in 2005.) A trial of computer-based assessment will form part of PISA 2006.
The domains covered by PISA are defined in terms of:
- The content or structure of knowledge that students need to acquire
- The processes that need to be performed
- The contexts, or situations, in which knowledge and skills are applied.
PISA examines young people's ability to use their knowledge and skills to meet real-life challenges rather than whether they have mastered a specific school curriculum. For instance, PISA defines reading literacy as the ability to understand, use and reflect on written texts in order to participate effectively in life. Its concern is not one of making a parallel assessment to those provided by school examinations.
PISA provides a broad assessment of comparative learning outcomes towards the end of compulsory schooling to guide policy decisions and provide insights into factors that contribute to the development of knowledge and skills, and the extent to which these factors are common to different countries.
Design of PISA 2003
In 2003 over 275,000 students took part in PISA from 41 participating countries 4. All 30 member countries of the OECD participated and 11 non-OECD, partner, countries. One OECD country, the United Kingdom 5, failed to meet the rigorous sampling criteria for the study, meaning that its results are not considered sufficiently reliable to be reported in full. Scotland and Northern Ireland did, separately, meet the criteria. Consequently, this report gives results for Scotland and 29 OECD countries only. The UK and the partner countries are not included in this national report.
Pencil and paper assessments were used to assess students, with two hours of assessment for each student. Various questions types were used, including multiple choice questions and questions requiring students to construct their answers. Each student's particular assessment tasks were drawn from a total of six and a half hours of assessment items, with different students taking different combinations of items from this pool. This combination spanned all the topics being assessed in the 2003 survey, namely: mathematics, reading, science and problem solving. Students also completed a 30 minute questionnaire and senior teachers completed a questionnaire about their schools. Test sessions were supervised by external administrators in most countries , including Scotland.
PISA 2003 International Report
The OECD is to publish the first results from the 2003 study in 'Learning for Tomorrow's World - First Results from PISA 2003', (OECD, 2004). The results will also appear in the OECD's annual compilation of educational indicators Education at A Glance. Subsequently, a number of thematic reports will be produced. These will primarily concentrate on the findings from the mathematics section of the study. All these reports can be found on the OECD web-site at " http://www.oecd.org/".
Following OECD convention, the report gives two OECD values for the various results it tabulates, eg, two values for the overall mean scores are tabulated, not just the single value that might be expected. The first of the two values (the OECD average) is the average of each of the individual country averages. The second value (the OECD total) is weighted to take account of the different numbers of 15 year-old students in each of these countries. Thus, while the USA contributes equally to the first value, it makes a much larger contribution to the second. Iceland on the other hand while also contributing equally to the first value, contributes much less to the second. The first index is useful for comparing how a country compares with a typical OECD country, the second for comparing the performance of students in any one country with all students in the OECD. We have tried to ensure that, in determining which value to use in the following analysis, we are consistent with the approach that is used in the PISA 2003 international report.
PISA in Scotland
Scotland is not a member country of the OECD, but a part of the United Kingdom. For the PISA 2003 study it operated as an independent National Centre, meaning that it participated fully in all PISA activities as though it were a full country. This allowed for greater input into the planning stages of the programme as well as separate quality monitoring and adjudication of test administration. Structural differences in secondary schooling between Scotland and the other parts of the United Kingdom were more readily managed by treating Scotland as a distinct National Centre, and the results were more readily analysed by the International Study Centre as a consequence. Additionally, results for Scotland will appear in tables in the annex to the international report on the study, enabling the standing of Scotland to be readily compared with that of other countries, unlike PISA 2000, where results were listed for the whole of the UK only.
The intended sample for Scotland was of 108 schools and 35 pupils in each of these. Similar numbers were required for every participating country. The school sample was a stratified, random sample representative of all mainstream secondary schools in Scotland (local authority, grant-aided, and private schools). Special schools were excluded. The pupil sample spanned S4 and S5, as 15 year old pupils are found in both stages, and the required pupils were selected randomly by date of birth. All pupils except those with any of a defined set of severe learning difficulties, and those who explicitly refused to participate or whose parents refused for them, were eligible for assessment.
Of the 108 Scottish schools, 98 returned completed tests and questionnaires, a response rate of 90%. Just over 2,700 students completed tests and questionnaires. These response rates met the OECD's strict criteria for acceptability. The SCRE Centre and the Scottish Executive Education Department is very grateful to the schools and pupils that agreed to take part in PISA 2003. We understand the many demands that are placed on schools and on 15 year olds and that participation in PISA creates some disruption at a time when many are preparing for exams. We are pleased that schools see PISA as an important and worthwhile study and hope that this initial report and, in particular, the further reports planned for 2005 helps highlight the value of participation.
The Scottish Report
As already noted, this report presents results for Scotland and the 29 OECD countries that fully met the OECD criteria for acceptability. Comparative results for the UK are not given, as the OECD considered these too unreliable ( see Annex A for explanation). No results for the 11 non-OECD partner countries are given: readers wishing to know about any of these countries should refer to the main, international reports. Similarly, those wishing to know how Northern Ireland fared should refer to its national report.
The decision to limit the report to Scotland and the OECD was one of expediency. To have reported results for all 41 countries, would have meant excessively long tables and charts. As the 29 OECD countries included most countries whose results would be of interest to a Scottish readership, including most European Union countries, this was the best compromise. It also seemed to be an approach that was consistent with the presentation of the results in the international report where, as already mentioned, international averages are based on the performance of OECD countries only.
This present report is concerned with the main set of test results from the study. Further themed reports planned for 2005 are likely to report on the questionnaire data gathered from schools and pupils, the relationships between these data and the results obtained in mathematics, and the domain of problem solving (which was additional in PISA 2003).
This first Scottish report has been written at the same time as the international report and has drawn heavily on the results presented in that report. Clearly, one unavoidable consequence of this is that it may not be possible to carry through to the Scottish report all the last minute editorial revisions made to the international report. Some of the values reported in the Scottish report may, consequently, differ from those in the published International report. Any such discrepancies should be slight and should not change any conclusions drawn about the Scottish performance.
Trends in Mathematics and Science Study (TIMSS) 2003
At the same time as taking part in PISA 2003, Scotland also took part in the Trends in Mathematics and Science Study (TIMSS) which is run by The International Association for the Evaluation of Educational Achievement (IEA). TIMSS measures mathematics and science performance at P5 and S2. This study reported on 14 th December 2004 and the results show that our performance in mathematics is not significantly different from the international average at P5 but is significantly higher than this average at S2. In science, our performance was significantly above the international average at both P5 and S2.
When drawing analysis between the TIMSS and PISA results one must take into account the differences in the nature of the PISA and TIMSS approach. The assessments used in TIMSS were constructed on the basis of an analysis of the intended curriculum in each participating country so as to cover the core material common to the curriculum in the majority of participating countries. By comparison the assessment material used in PISA cover the range of skills and competencies that were considered to be crucial to an individual's capacity to fully participate, in respect of the assessment domains, in a successful modern society (in other words, to apply their skills in 'real life' situations). The other important difference between PISA and TIMSS is the stage at which students are assessed. While PISA assesses pupils towards the end of compulsory schooling (and therefore assesses the cumulative impact of schooling), TIMSS assesses pupils in middle primary and in early secondary.
For further detail about TIMSS and Scotland's performance in this study, see http://timss.bc.edu/.
Domestic Survey of Attainment
The results of PISA 2003 are also more encouraging that those of Scotland's domestic Assessment of Achievement Programme (AAP). In 2003, AAP examined student attainment in science. While PISA reveals that (although not our strongest subject), towards the end of compulsory schooling, students in Scotland perform relatively well in science compared with their international peers. AAP suggests that our younger secondary school students are not reaching the attainment targets in science that are set nationally. At secondary school level, AAP tests a sample of pupils from S2. The results show that fewer than 10% of this sample was 'secure' at the target attainment level (Level E) and a high proportion failed to show even 'basic' attainment at this level.