Scottish Government

CHAPTER TWO METHODOLOGY

Desk research

2.1 The evaluation began with desk based research to profile participating schools in terms of:

• Mapping negative destination trend data over the past 5 years using SLDR data
• Mapping key performance indicators (covering attainment, free school meals, and negative destinations) using data from Scottish Schools Online
• Selecting potential comparator schools in each local authority, with similar key performance indicators to the pilot schools (Appendix F).

Case study visits

2.2 Case study visits were conducted during March and April 2007 and again in March and April 2008 with each of the 13 ² pilot schools. This was the optimum time to conduct the visits as it was nearing the end of the academic year, allowing interviewees to reflect on the activities of that year, yet avoiding exam times where school staff and pupils are less able to dedicate the time required to be involved in research. In each year these visits involved:

• Qualitative interviews with all 7 ³ERP Advisers and the designated Senior Management Team representative within each pilot school. A semi-structured topic guide was used to lead discussion around:
• Location of the pilot within the wider school
• Changes that have occurred over the course of the pilot
• Perceived success of the pilot
• The potential impact of the end of the pilot in each school
• Sustainability of work and outcomes achieved as a result of the pilot (contained in Appendices B and C).
• Qualitative focus groups with ERP participants where possible in each of the 13 pilot schools (in 2006/7 we spoke with 47 pupils in 9 of the schools and in 2007/8 we spoke with 65 pupils in 11 of the schools). A semi-structured topic guide was used to facilitate discussion around:
• The nature of activities undertaken with the ERP adviser
• Enjoyment and perceived usefulness of these activities
• Potential impact of activities on awareness and planning of post-school options (contained in Appendix A)

2.3 In December 2007 a short telephone consultation was carried out with each of the ERP Advisers to assess the extent to which the pilot was bedding in during its second academic year.

Telephone survey

2.4 A telephone survey was carried out in May 2007 with 81 ERP participants and 83 parents and again in May 2008 with a new cohort of 86 ERP participants and 91 parents (questionnaires contained in Appendices D and E). The principal aim of the surveys was to isolate the impact of the pilot by focusing on the aspirations and attitudes of young people prior to and following engagement in the pilot. The survey explored:

• The support received
• Impact of support on their school career
• Impact of support on career aspirations
• Applications to employment, education or training
• Personal development.

Data Analysis

2.5 In October 2007 and October 2008 Careers Scotland provided the consultants with School Leaver Destination Returns ( SLDR) data for pilot and comparator schools. This was analysed to assess:

• Any changes in the proportion of pupils moving into negative destinations in pilot schools, comparing figures with the previous year and monitoring progress towards individual school targets of reducing by 50% the proportion of pupils entering a negative destination
• The extent to which these changes were also evident in comparator schools

2.6 At this time Careers Scotland also provided SLDR tracking data that showed the sustainability of the destinations of ERP participants. This was analysed by the consultants to assess the extent to which ERP pupils were sustaining positive outcomes.

2.7 An interim report was prepared in November 2007, drawing together the findings of the first round of case study visits, the first telephone survey and analysis of the 2006/7 SLDR data. A second interim report was prepared in July 2008, drawing together the findings of the research carried out in the first year of the pilot, the 2008 case study visits and the 2008 telephone survey. When the 2007/8 SLDR data became available in October 2008, the final report was prepared, incorporating findings from across the 2 years of the pilot.