Scottish Government

2 THE EVALUATION

Aims and objectives of the evaluation

2.1 This chapter sets out the aims and objectives of the evaluation and summarises the approach, timetable and methods employed.

2.2 The Scottish Government Health Directorates set multiple aims and objectives for the evaluation of the pilot. The initial overarching aim of the evaluation was to:

Evaluate the implementation, operation and potential impact of employer-led regulation of HCSWs in Scotland.

2.3 In relation to the implementation of the pilot, the evaluation was to assess:

• Early communication with pilot sites and the usefulness of initial guidance for participating NHS Boards
• The need for local/national coordinators
• The ease with which the standards fitted with existing HR practices
• The time taken to establish the pilot within local sites
• The initial training required for Workplace Supervisors
• The time taken to establish monitoring criteria as part of quality assurance measures

2.4 In relation to the operation of the pilot the evaluation aimed to:

• Describe the process of recruiting HCSWs into the pilot and identify reasons for non-participation
• Explore stakeholder views on the definition of 'healthcare support worker' and groups to be included
• Assess whether Disclosure Scotland was able to meet the pilot demands for 'enhanced' Disclosure Scotland ( DS) criminal record checks
• Identify whether systems are in place for recording and dealing with previously unknown convictions uncovered through DS checks
• Identify the processes used to assess achievement of the standards and entry to the list
• Assess whether the process of personal development planning and review ( PDPR) is fit for purpose for assessing HCSWs against the standards and entry to the list
• Assess whether the process used to demonstrate and document the achievement of standards has the potential to be linked to the KSF foundation gateway of Agenda for Change
• Assess how onerous the process of achieving the standards is on the HCSW and the Workplace Supervisor
• Investigate the interface between the standards required of HCSWs within this pilot and those for support workers in social services and independent healthcare
• Assess the technical aspects of setting up, operating and maintaining a list of HCSWs
• Assess whether an appeals process is required for HCSWs who disagree with their Workplace Supervisor's assessment of their fitness to practise and whether there should be a process for the reinstatement of names in any future regulatory framework

2.5 In relation to the potential impact of employer-led regulation the evaluation aimed to:

• Assess whether rolling out the pilot across Scotland has the potential to enhance public protection
• Assess whether the SWISS database has potential to support future safe recruitment practice through 'flagging up' when a new entrant to the NHSS workforce is already on the SWISS database or has been dismissed by another employer for fitness to practise reasons
• Assess whether rolling out the pilot has potential to improve working practices amongst HCSWs
• Provide an indication of the resources (including staff time) required for potential roll-out of the pilot
• Identify how employer-led regulation could most effectively fit within existing/forthcoming human resource policies and procedures
• Assess whether the regulation of HCSWs needs to be underpinned by legislation/statutory regulation
• Provide recommendations outlining how, and whether, a system of employer-led regulation and listing of HCSWs could most effectively and efficiently be rolled out across Scotland
• Assess the added value that a future list would bring over the implementation of national standards alone
• Outline the measures that will need to be put in place prior to rolling out the pilot across Scotland

Methodology

2.6 The evaluation comprised both formative and summative aspects and adapted to follow pilot developments. A range of methods was used, including:

• Stakeholder and key informant interviews
• Individual HCSW case studies
• Surveys of HCSW volunteers, workplace supervisors, and non-participant HCSWs
• Desk research - pilot documentation and monitoring data
• Consultative Workshop

2.7 The SGHD tendered the evaluation in October 2006 and commissioned it in late 2006. The evaluation, originally planned to run for a full year (2007) in parallel with the pilot was subsequently adapted, in design and timescales, to capture the slower than anticipated pilot implementation. An initial formative stage (December 06 -March 07) explored the early implementation of the pilot and gained key informant perspectives to help scope aspects of the evaluation. Following a formative report to the 4 Country Steering Group in April 2007 and at the commissioner's request, an additional formative phase was undertaken (into summer 2007). The evaluation, following the extension of the pilot itself was extended to the end of December 2008. The summative stage, obtaining participant and stakeholder feedback ran from June to September 2008, to gather data to inform the consultative Workshop.

2.8 Throughout, the evaluation team has liaised closely with the SGHD commissioner and the National and Local Pilot Coordinators to track pilot developments. The lead researcher attended a range of pilot meetings; evaluation observations of these meetings inform this report. Meetings attended included the 4 Country Steering Group, NHSQIS National Project Group, a sample of pilot briefings, meetings and training sessions. Interim evaluation reports were presented to the 4 Country Steering Group (April and October 2007, September 2008) and the Workshop event.

2.9 The Scottish Government invited the independent healthcare site taking part in the pilot, Ross Hall Hospital, to take part in the evaluation. However, because of funding issues, Ross Hall declined.

2.10 The possibility of including patients' perspectives in the evaluation was considered at the outset, but the commissioners and evaluation team agreed that the pilot would be at too early a stage for this to be either possible or meaningful. However, the lay perspective was included on the 4 Country and NHSQIS National Project Steering groups, and their views are included in the evaluation.

Stakeholder and key informant interviews

2.11 Formative stage interviews (December 2006 to summer 2007) and end stage interviews (June to September 2008) were conducted, comprising a total of 54 interviews with 45 participants, with some participants interviewed twice. The planned number of interviews was extended (from 32 to 54) to capture the complexity of the pilot. Around 25 were stakeholder interviews and 29 were with key informants (some respondents held dual roles, being both directly connected to the pilot and also holding a role on a national group or body). Pilot stakeholders were purposively selected to capture pilot experience from a range of roles across the three sites but not all roles were represented at all sites. Pilot stakeholder representation included National and Local Pilot Coordinators, pilot site lead officers, human resources, facilities and partnership representatives (including some from the withdrawn Glasgow site). Key-informant representation included Disclosure Scotland, NHS Quality Improvement Scotland ( NHSQIS) professional staff and lay reviewers, NHS Education for Scotland ( NES), Ross Hall Hospital, Manager of Scottish Workforce Information Standard System ( SWISS), Health Facilities Scotland, Nursing Directors, Scottish Social Services Council ( SSSC), 4 Country Steering Group, KSF implementation lead, HCPL, Director Council for Healthcare Regulatory Excellence, UNISON, SWAG/ AMICUS/ UNITE, the Scottish Government pilot lead officer and the Protection of Vulnerable Groups policy officer.

2.12 The majority of interviews were conducted face-to-face with some additional end stage interviews by telephone. Separate topic guides were developed for the two stages of the pilot and were used flexibly and adapted to the particular area of interest for particular informants to pick up on emerging issues. Topics included:

• Informant role and involvement in pilot
• View on HCSW definition and inclusion in pilot
• View on guidance, approaches taken, processes and materials developed
• Key barriers, facilitators and emerging issues
• View on potential roll-out of pilot

Individual healthcare support worker case studies

2.13 Individual healthcare support worker case studies had two purposes:

• to provide rich data on HCSW and WPS perceptions of pilot processes
• to inform development of questionnaires and topic guides

2.14 Prospective case studies explored individual participants' experiences of recruitment, clearance by Disclosure Scotland, assessment, achievement of the standards and listing, and their perceptions of the pilot's development, timescales and burden. These individuals, tracked over time, helped inform the emerging picture of different experiences across worker groups and pilot sites as the pilot developed.

2.15 The twelve case studies represent a range of participants, from those who completed the process fairly easily to those for whom it went much less smoothly. They include people working in hospital and community mental health service settings, older people and children's services. The case studies include representation from the three sites (Ayrshire & Arran 5; Lothian 3; Lanarkshire 4), and from clinical ( 6 Nursing assistants; 2 Allied Health support workers) and non-clinical groups ( 3 Domestic assistants and 1 Medical laboratory assistant).

2.16 The research team briefed the Local Pilot Coordinators on the range of case studies sought and several potential participants were identified. The research team then contacted HCSWs by telephone or letter to request an interview. Each HCSW was interviewed by telephone, initially soon after recruitment and again some months later towards the end stage of the pilot. At first interview, we requested the HCSW's permission to contact their Workplace Supervisor and this was again confirmed at the second interview. The WPS was not contacted by the research team until after the second HCSW interview and 11 were interviewed (1 HCSW could not be contacted for a second interview). Separate topic guides were developed for HCSWs and WPS and topics included HCSW perceptions of information received, assessment, Code of Conduct and listing; WPS perceptions of preparation, training, support and the learning and assessment process.

2.17 In this report, case study material is used to illustrate themes. Given the small number of recruits in some categories, complete individual case studies are not presented to preserve confidentiality.

Surveys of HCSW volunteers, WPS and non-participant HCSWs

2.18 A questionnaire survey was sent to all HCSWs who had agreed to take part in the pilot and evaluation, to gain an understanding of their perceptions and experience of the process. A questionnaire survey was also sent to Workplace Supervisors to gain understanding of their experience of training and support received during the pilot and their views on supporting HCSWs through the pilot. A brief questionnaire was sent to all eligible HCSWs who had not taken up the invitation to get involved with the pilot.

2.19 The three questionnaires were drawn up in consultation with the NHSQIS Project Steering Group. ScotCen provided survey packs to Local Pilot Coordinators who generated name and address labels and mailed the surveys on our behalf (June-August 2008). Very recent HCSW recruits, who would have had little chance to progress, were excluded. The survey went to all WPS involved at the time. Reminders were issued to participating HCSWs and Workplace Supervisors only, two to three weeks later. The timing of the surveys was determined by the requirement to provide early evidence to the consultative workshop ³².

2.20 The evaluation team was keen to capture the views of healthcare support workers who opted not to get involved with the pilot. Early on, individual letters offering research interviews were sent to non-participants who had given consent to take part in the research (Lothian). However no-one responded. Later, we set up two 'drop-in' sessions to enable reluctant HCSWs to speak to a researcher, but again there was almost no take up (February 2008 Lothian). Therefore in order to maximise the possibility of returns, we extended the scope of the non-participant survey and issued it to all 2101 eligible HCSWs identified at that point.

2.21 All healthcare support workers and workplace supervisors were made aware of the evaluation at local site briefings on the pilot and via the evaluation information leaflet in individual information packs. HCSWs gave signed consent to Local Pilot Coordinators for participation in the pilot and/or evaluation. All mail outs were issued by LPCs. Informed consent was again sought at the start of any HCSW or WPS interview.

Survey Response

2.22 Participant response rate ranged from 38% for Workplace Supervisors to 43% for HCSW-participants, a reasonable figure for a self complete postal survey (Table 2.1). The HCSW-participant survey response rates were consistent across the three sites, but the responses to the WPS survey were not. Only a 19% return from Lothian was achieved, suggesting that there was possibly less engagement from that site's WPS at that point. The 12% response rate from non-participant HCSWs was higher than anticipated with over 240 valid returns. The majority of returns came from the much larger pool of Lothian eligible HCSWs 86% ( 208) with 10% ( 25) from Ayrshire & Arran and just nine responses, 4% of the total, from Lanarkshire non-participants.

Table 2.1: Survey response HCSW participant & WPS, non-participant HCSW

2.23 The response from all three surveys represented a reasonable spread across services areas (children's, mental health and older people's services); across worksites (hospital and community based) and across working patterns (full and part-time). There was little feedback from new employees. Full details of survey response rates are provided in Annexe O.

Desk research

2.24 The basic requirement for routine monitoring data was clarified at an early meeting between the ScotCen team, national SWISS lead officer, NHSQIS project officers and pilot site lead officers. Data tracked HCSW participants' progress from recruitment to eventual listing upon SWISS. LPCs forwarded an anonymised, agreed dataset to the National Pilot Coordinator who subsequently forwarded an amalgamated national data set (Excel format) to the evaluation team. It refers to the 470 HCSWs who volunteered to take part in the pilot. Data on the pool (2961) of eligible HCSWs, including participant and non-participant HCSWs is taken direct from the local datasets forwarded to the evaluation team. All monitoring data in this final evaluation report is as available at 31/12/2008.

2.25 NHSQIS conducted a number of Quality Assurance activities throughout the project. The NHSQIS Project Report ³³ and associated independent consultant reports ³⁴ inform this final evaluation report. The ScotCen team and NHSQIS National Pilot Coordinator worked together to try to avoid duplication and to minimise the evaluation burden upon pilot participants.

2.26 The Scottish Government Health Directorates Professional Adviser to the Pilot regularly forwarded updates from relevant Scottish and 4 Country policy and practice developments. The NPC and LPCs forwarded local and national pilot steering group minutes and updates.

Consultative workshop

2.27 The SGHD and evaluation team initially planned to hold a consultative workshop (October 2008) on pilot and evaluation findings and to debate policy recommendations. As the national regulation agenda developed, this planned pilot event became the focus of the second day of SGHD's first Regulation Event. A wide range of policy and practice representatives, pilot stakeholders and participants, partnership and service user representation and representatives of patients and the public attended ( Chapter 10).

Data management and analysis

2.28 All interviews were digitally recorded and transcribed in full. Transcripts were then summarised for analysis using 'Framework'. Framework, developed by the National Centre for Social Research ³⁵, provides a consistent method for organising and condensing qualitative information to enable robust analysis. It facilitates both between case (looking at what different people said on the same issue) and within case (looking at how a person's opinions on one topic relate to their views on another) investigation. A charting 'matrix' of key topics was developed following familiarisation with the transcripts. Every transcript was then summarised under these key themes. These summary charts were then investigated to map the range and diversity of peoples' experiences and views and to explore the reasons for particular opinions.

2.29 Quantitative, questionnaire and monitoring data, was entered into and analysed in SPSS statistical package.

2.30 Verbatim quotations, in italics, are anonymised to protect respondent identity, and are used to illustrate, amplify and clarify findings.

Summary

2.31 The two year evaluation (2007-2008) ran parallel to the pilot and aimed to evaluate its implementation, operation and potential impact. The mixed method approach captured both early process issues and early outcomes. A rich range of perspectives was gained via over fifty stakeholder interviews and three surveys to healthcare support workers (participant and non-participant) and their workplace supervisors. Individual support worker and supervisor case studies illustrated key themes. Local site data was collated to monitor support worker progress from recruitment through to assessment and listing. A final national level, consultative workshop, provided an opportunity for key stakeholders and pilot participants to comment upon the pilot and early evaluation findings.