CHAPTER 1 INTRODUCTION: AIMS AND OBJECTIVES OF THE OPTIONS APPRAISAL
1.1 The Scottish Government Schools Directorate and Lifelong Learning Directorate commissioned the Centre for Educational Sociology and BMRB to carry out an appraisal of the options for meeting the needs for longitudinal data on young people in Scotland, in particular in relation to their post-school transitions.
1.2 The two Directorates are responsible for policies that have an impact on young people through their school years and beyond into their post-school experiences of education, training and employment, and need the capacity to evaluate the impact of their policies on the post-school transitions of young people. They recognise that data on young people's transitions are also required by a wider user group of policy makers and practitioners, academics and researchers.
1.3 The Scottish Government (and previously the Scottish Office) has funded the Scottish School Leavers Survey ( SSLS) since the mid 1970s. The SSLS is a longitudinal study of a nationally representative sample of young people in Scotland. It collects data on young people's experience and attainment in compulsory secondary education and their subsequent transitions through post-compulsory education, training and the labour market including related information such as family background. The young people are first surveyed in the year after the end of compulsory education and several times thereafter - as young people's transitions have become more protracted, the age range of SSLS has been extended and recent cohorts have been followed up to the age of 23-24 (Appendix 1 gives further information about the SSLS). The intention has been that SSLS should provide data not only for the Scottish Government but also for provider agencies as well as the academic and research community.
1.4 The SSLS has provided a rich source of data over the years and has given Scotland an enviable resource for the longitudinal study of young people's transitions. But much has changed since its basic design was developed including significant changes in the nature of young people's transitions, in survey methodology and in the policy environment.
1.5 In recent years the SSLS has also suffered, in common with other postal surveys, from declining response rates at the first contact with consequent issues of bias in subsequent sweeps. In particular, low response rates and attrition have been an increasing problem in respect of less qualified young people and those not in education, training or employment, and those from lower social classes. An additional problem is the small cell size for groups of particular policy interest such as those young people not in education, employment or training, and looked after children. There are also issues about the age range covered by SSLS; the extent to which it captures the processes of transition; the quality of information about training and employment; and the retrospective nature of some of the attitudinal data.
1.6 The most recent contract for the SSLS ended in 2007 and it was seen as timely to review how the evidence needs on young people's transitions could best be met in the future.
1.7 The options appraisal was therefore commissioned to undertake a critical examination of a range of options on how to meet the needs for longitudinal data on young people, and make recommendations on any future longitudinal study. While the continuation of SSLS in some form was not ruled out, it was envisaged that the options appraisal would examine different approaches, taking into account the current and developing data collection environment. At an early stage of the project, it was agreed in discussion with the Scottish Government that the project would focus on developing, in some detail, different options for a potential new longitudinal study rather than investigating the full range of possible alternative approaches.
The options appraisal
Aims
1.8 The stated aims of this options appraisal were as follows:
- Analyse the needs for longitudinal data of existing and potential users in the Scottish Government and the wider community on young people's experiences in secondary school and subsequent transitions to further/higher education, training and employment.
- Examine different options for collecting, analysing and reporting longitudinal data about young people, and evaluate their potential for meeting the needs of users.
- Clarify the costs and benefits of each option, and make recommendations.
1.9 Specific objectives of the options appraisal were:
- to examine and recommend options for meeting data needs, giving consideration to:
- subject coverage
- study design
- sample sizes, including sub-samples for certain target groups
- sampling strategy
- lower and upper age limit of participants
- data collection methods
- data issues
- frequency
- uses for the data
- outputs and dissemination
- to provide a cost-benefit and a risk analysis for each option;
- to examine the extent to which the SSLS meets evidence needs in its existing form, could meet these needs in a revised form, or should be replaced by a new study or studies;
- to examine the extent to which there are other sources of data available now, or under development, that meet the evidence needs of policy-makers and other users;
- to consider the potential benefits and practical issues involved in matching individual records from any longitudinal study commissioned by the Scottish Government with records from other data sets;
- to consider the opportunities of working with other UK Government departments, Research Councils (particularly the ESRC), or others, to create an infrastructure for longitudinal data collection in Scotland or across the UK, including opportunities for co-funding future longitudinal research;
- to provide recommendations on the way in which the Scottish Government can set and adjust strategic priorities for any longitudinal study over time.
Methods
Consultation with external stakeholders and the Scottish Government
1.10 A key element of the options appraisal was to consider the needs of the Scottish Government and those of the 'wider user group' in respect of longitudinal data on youth transitions. The Scottish Government had already undertaken an initial information gathering exercise internally about the evidence needs of policy teams and this information was made available to us. This was supplemented by two seminars that the research team held with policy teams and analysts from the Schools and Lifelong Learning Directorates as well as discussion in the research advisory group. Three interviews or meetings were also conducted with Scottish Government staff on issues in respect of administrative data.
1.11 A number of external users and potential users of longitudinal data on young people's transitions were identified in consultation with the Scottish Government. These external stakeholders included academic users, national organisations and bodies in Scotland as well as the Department for Education and Skills ( DfES) 1 and the ESRC. A total of 38 organisations and individuals were consulted (Appendix 2 provides details).
1.12 Chapter 4 summarises the views expressed by policy makers, analysts and stakeholders. Most of those interviewed had heard of the SSLS although their knowledge of it varied widely from those who were aware of its existence but had never drawn on it, through to those who had an in-depth knowledge. We have not attempted to weight the views of stakeholders, for example, by their knowledge of SSLS, or of youth transitions or by the type of organisation concerned; this approach was agreed with the Government. Given the large degree of commonality in stakeholders' responses, this would not have been a productive approach or made any substantial difference to the findings.
1.13 It was evident that the consultation process itself raised awareness of the SSLS, the existence of relevant reports and papers and its potential for analyses of interest to the stakeholders interviewed. A small number of those consulted were in a position to make comparisons about the use and value of SSLS over the last decade or so with the position earlier, in the 1980s. While the limitations of poor response rates in the most recent surveys were acknowledged, stakeholders specifically noted the decline in the visibility of the SSLS and the apparent decrease in the number (and value) of publications coming across their desks.
1.14 Depending on stakeholders' level of awareness of SSLS, information was sent to them in advance of the interviews (see Appendix 1). Stakeholders were interviewed using a mix of telephone and face-to-face interviews. The average length of interview was 50 minutes.
1.15 Appendix 3 contains the topic guide used for the interviews. As well as discussing stakeholders' views and preferences in respect of a longitudinal study, stakeholders were also asked about the value of cross-sectional approaches and administrative data and the extent to which these approaches would meet their data needs. In summary the interviews covered the following topics:
- views on the purpose and design of a longitudinal study of young people's transitions, including whether a longitudinal study is needed and a comparison of longitudinal vs. cross-sectional approaches;
- coverage of the survey, in terms of which young people they would like included;
- the timing and frequency of surveys;
- the topics on which stakeholders need information;
- their uses for the data;
- the extent to which their data needs and uses have changed over time;
- their views on the need for comparability of data - over time and nationally and internationally;
- their preferences in respect of how the data might be analysed, on access to the data, on dissemination of findings and preferred types of outputs.
Investigation of administrative data
1.16 The remit of the options appraisal included consideration of the potential contribution to a longitudinal study of the administrative data that are increasingly available. More generally, it was also expected to consider the development of options in the context of a changing data environment, for example, the "Managing Information Across Partners" ( MIAP) strategy which aims to develop systems for sharing data across education and lifelong learning providers in Scotland. The MIAP strategy is part of the Open Scotland Information Age Framework ( OSIAF) which provides common frameworks for sharing person-level data between government departments.
1.17 The research team investigated a number of potential administrative data sources through a review of websites and published documents and a small number of meetings or telephone interviews with relevant personnel. The administrative sources that can provide linkable data for a longitudinal study are listed in Appendix 4.
1.18 The administrative data were considered in relation to their use in:
- providing a sample frame;
- enabling the tracing of non- respondents;
- enabling the linkage of a range of additional information on respondents.
1.19 At present the legal position regarding administrative data sharing and linkage is not well defined. There is considerable impetus within the UK government and Office for National Statistics ( ONS) to clarify these issues and move forward the use of administrative data.
Investigation of longitudinal studies
1.20 Other longitudinal studies in Scotland, the UK and internationally were reviewed in relation to design and methodology, and approaches to analysis, dissemination and user engagement (see Chapter 3).
Development of options for a longitudinal study
1.21 In addition to the activities outlined above, the research team also undertook a review of the latest survey design literature and a focused review of the youth transitions literature. Based on all of these activities, and the experience of members of the research team, several options for longitudinal data collection, analysis and reporting were developed, assessed and costed (Chapters 5 and 6).