Scottish Government

Chapter 1: Introduction

What is PISA?

1.1 The Programme for International Student Assessment ( PISA) is an international assessment of student attainment in reading, maths and science at age fifteen. PISA is run by the Organisation for Economic Cooperation and Development ( OECD) and takes place every three years. Scotland has participated in PISA on each occasion since its inception in 2000.

1.2 Each cycle of assessment emphasises one of the three specific domains included, although the others are still tested. In 2009 the emphasis was on reading literacy. Students also complete context questionnaires providing socio-economic data alongside data on attitudes and motivation.

1.3 Worldwide, about 470,000 students from 65 countries took part in the study in 2009, including all 30 (or 34) OECD countries ¹ and eight states which had not participated in previous rounds.

What does PISA measure?

1.4 PISA focuses on testing the knowledge and skills required for participation in society and assessing the extent to which students can apply skills gained in school in everyday adult life, thus moving beyond the student's ability to master the school curriculum.

1.5 The reading assessment seeks to measure the capacity to understand, use and reflect on written texts in order to achieve one's goals, to develop one's knowledge and potential, and to participate in society. The survey assesses three broad aspects of reading, namely "access and retrieve", "integrate and interpret" and "reflect and evaluate". Various skills and strategies are needed to answer questions from each of those groups. For example "access and retrieve" involve skills associated with finding, selecting and collecting information, "integrate and interpret" involves processing what is read to make internal sense of a text and "reflect and evaluate" tasks involve drawing on knowledge, ideas or values external to the text. Readers relate their own experience or knowledge to the text and make a judgment about it.

1.6 The mathematics assessment seeks to measure the capacity to analyse, reason and communicate effectively as they pose, solve and interpret mathematical problems in a variety of situations involving quantitative, special, probabilistic or other mathematical concepts.

1.7 The science assessment seeks to measure an individual's scientific knowledge and use of that knowledge to identify questions, to acquire new knowledge, to explain scientific phenomena, and to draw evidence based conclusions about science-related issues, understanding of the characteristic features of science as a form of human knowledge and enquiry, awareness of how science and technology shape our material, intellectual, and cultural environments, and willingness to engage in science-related issues, and with the ideas of science, as a reflective citizen.

1.8 Analysis of the results of PISA allows the education community to examine factors associated with high or low performance. The OECD produce a number of research reports based on the survey findings. These are available on the OECD's PISA website www.oecd.org/pisa

1.9 Item Response Theory was used to estimate the performance of students in tasks not included in their booklets. This is a modelling technique whereby the probability of a student's response being correct in a test item that they did not attempt is estimated using their success rate in questions they have attempted, the relative difficulty of the questions they have attempted, and certain underlying student characteristics e.g. gender.

1.10 Strict international quality standards set by the OECD are applied at all stages of the survey.

PISA in Scotland

1.11 While the main OECD publication reports on the United Kingdom as a whole, Scotland participates as a distinct adjudicated area enabling results to be published separately. A subset of this information is reported by the OECD in their report. England, Wales and Northern Ireland also have boosted samples as non-adjudicated areas within the rest of the UK in order to produce separate analysis.

1.12 The survey was carried out in Scotland between 2 ^nd March and 3 ^rd April 2009. Given the participating age of 15, most students were in S4 and had been in school education since the time of devolution (1998/99).

1.13 The Scottish fieldwork was undertaken by the National Foundation for Educational Research ( NFER).

1.14 A representative sample of 106 publicly funded secondary schools and four independent secondary schools in Scotland was randomly selected by the OECD for participation in the survey. The sampling frame was stratified on the basis of exam performance (split into five categories), urban/rural location and school size. Special schools were excluded from the sampling frame.

1.15 From the main sample 87 schools participated, giving an initial weighted school response rate of 80 per cent. As this was below the required 85 per cent, replacement schools, also sampled by OECD, were asked to participate. This however meant that the required response rate was increased from 85 per cent to 88 per cent. This requirement was met with the participation of eleven replacement schools.

1.16 From the 98 participating schools a total sample of about 3,400 students was then randomly selected to participate in the survey. Students participated on a voluntary basis. Schools were able to withdraw students from the sample where it was considered that assessment would cause undue stress; this may have included students with additional support needs. This left 3,169 eligible students.

1.17 The overall exclusion rate for Scotland was therefore 4.4 per cent, made up of 1.7 per cent school exclusions (special schools) and 2.7 per cent within-school exclusion. This was within the maximum allowed of five per cent.

1.18 Student non-response then also included those not willing to take part, and those absent on the day of the assessment. Overall 2,631 students participated, resulting in an overall unweighted response rate of 83.0 per cent. Following weighting for age and gender there was an overall student response rate within the participating schools of 83.6 per cent, above the target rate of 80 per cent.

1.19 The assessments lasted two hours. Each student attempted one of 13 different booklets. The booklets contained questions in reading, maths or science, or a combination of the three. Approximately half of the tasks across the booklets were in reading, with the remainder split equally between maths and science. Students also completed a questionnaire giving information on their background and on their views about reading, with head teachers also completing a questionnaire about the school.

Interpreting PISA results

1.20 The true wealth of information obtained from PISA is contained in the analysis of the results against background information from national systems and the student questionnaires. However, much focus is given to the summary measure of the overall performance of students in Scotland. Before focussing on this though it is important to understand the way these results may be accurately reported.

Sampling error

1.21 As in all sample surveys, the results of PISA are affected by sampling error. By this we mean that, since PISA is based on a sample rather than the whole population and the students sampled may not reflect the population exactly, the results are only estimates of how the whole population would perform. The size of the uncertainty is reflected in the standard error. This uncertainty is why it is incorrect to give absolute rankings to each country. Scores are often given with confidence intervals within which we can say we are, for example, 95 per cent confident that the true value lies.

1.22 When comparing the performance of two groups e.g. Scotland and other countries, males and females, or 2009 and 2006, it is necessary to take into account the uncertainty caused by sampling. Small differences may not represent actual differences in the populations, but may be simply due to the sample participating. The ability to identify real differences will depend both on the size of the difference between the two groups and on the size of the standard error. Statistical methods are used to determine whether differences are significant.

Multiple comparisons

1.23 A complicating factor is that these significance tests still only give us a statement that we are, for example, 95 per cent confident that there is a difference. In the case of comparing Scotland against other countries this means that we would expect our (95 per cent confident) answer to be wrong five per cent of the time, i.e. for one in twenty countries. So when comparing against the 33 other OECD countries and three other UK countries, it is highly likely that at least one of our comparisons will be incorrect. In order to adjust for this expected error a "Bonferroni adjustment" is sometimes used. This sets the bar slightly higher for a difference to be significant, with the size of the adjustment based on the number of comparisons that are being made.

1.24 However this adjustment of course increases the likelihood of incorrectly stating a country is similar to Scotland. Also, since the size of the adjustment is dependent on the number of countries in the comparison, how Scotland compares against a particular country will depend on whether this is set in a table of OECD countries, UK nations, or some other group of countries, which does not make sense. For these reasons such adjustments have not been used in this report ². The main OECDPISA international report also does not use Bonferroni adjustments.

Change over time

1.25 The assessments in mathematics and science, and the way in which they were scored, were changed radically in 2003 (maths) and 2006 (science). It is therefore not possible to compare scores in maths and science with assessments from before these dates. Assessments in reading can be compared back to 2000. The following table shows OECD average and Scotland estimates for each year. Prior to 2003 (maths) and 2006 (science) it is not possible to say whether Scotland's performance changed, though it is possible to compare changes in Scotland's performance relative to the OECD average or other countries. So for example in maths, Scotland's performance fell between 2003 and 2006, whereas between 2000 and 2003 it fell relative to the performance of other OECD countries, though OECD performance may have fallen or improved.

Changes in the OECD

1.26 A further complication arises in the 2009 report because of the inclusion of four new countries (Chile, Estonia, Israel and Slovenia) who were invited to join the OECD in 2010. This means that the OECD average calculations are no longer based on the same number of countries as in 2006. Changes in Scotland's performance relative to the OECD average may therefore be affected by this change in OECD composition, rather than a change in the performance of any country. This report highlights where this has been the case by also calculating the OECD average without the four new countries.

PISA publications

1.27 The OECD has produced the following reports based on analysis of the results against background information from national systems and the student questionnaires. These are available from their website www.oecd.org/pisa

• Green at Fifteen? How 15-year-olds perform in environmental science and geoscience in PISA 2006
• Top of the Class - High Performers in Science in PISA 2006
• Equally prepared for life? How 15-year-old boys and girls perform in school
• The High Cost of Low Educational Performance
• Against the Odds
• Attitudes to Science

Other surveys of achievement in Scotland

1.28 In recent years Scotland has participated in two other international surveys, TIMSS (Trends in International Maths and Science Survey) and PIRLS (Progress in International Reading Literacy Study). Scotland is not participating in the 2011 TIMSS and PIRLS surveys.

1.29 The Scottish Government, in partnership with Her Majesty's Inspectorate of Education, the Scottish Qualifications Agency and Learning and Teaching Scotland, also conducts the Scottish Survey of Literacy and Numeracy (previously the Scottish Survey of Achievement), an annual survey which from 2011 will assess student performance in numeracy and literacy in alternate years.

Results for all of these surveys can be found on the Scottish Government website using the following link: www.scotland.gov.uk/schoolstats