EXECUTIVE SUMMARY
Introduction
1. In order to measure the outcomes and the impact that housing support services have for service users, in 2006 the then Scottish Executive commissioned research to review existing models to assess outcomes. This review recommended that the distance travelled model developed by City of Edinburgh Council would be suitable for application more widely. 1 After some refinement it was decided to pilot the Supporting People (Housing Support) Outcomes Framework and to evaluate that pilot. The evaluation was undertaken by Craigforth.
2. The main aim of the research was:
' To assess whether the Outcomes model being piloted could be introduced consistently at local authority level to capture useful information, which in turn could be used as a means of public and ministerial reporting at national level'.
3. The new Scottish Government is committed to focusing public services on outcomes, rather than inputs or outputs. In taking forward this agenda the Government has committed to simplifying and reducing the level of scrutiny of local government. This has included the removal of ring fencing from certain budgets from April 2008, including the Supporting People budget, with ring fencing removed from other budgets planned for future years.
4. A focus on outcomes requires the development of methods which capture the experiences of those who receive these services. The Outcomes Framework has therefore potentially increased, not reduced, in relevance.
The Pilot
5. The Supporting People (Housing Support) Outcomes Framework is structured around 4 summary categories: Accommodation; Health; Safety and Security, and Social and Economic Wellbeing. The Framework identifies 'elements of support' within each of these categories. The needs of service users are assessed on a five point scale in relation to those elements relevant to the individual's circumstances. Over time it is expected that the position of the service user will change: this will then be reflected in a changed assessment with the difference between assessments providing the 'distance travelled'.
6. In order to measure the preventative benefits of housing support services, providers are also asked to assess the position if the service had not been in place (the 'counterfactual'): this can be undertaken with or without the involvement of the service user.
7. Seven local authority areas volunteered to be involved with the pilot. Providers were then recruited to provide a spread of housing support services and client groups. A total of 33 providers were involved, 6 did not complete the pilot. The client groups covered by the providers included; domestic abuse; homelessness; learning difficulties; mental health; older people; physical disabilities; substance abuse and young people.
8. At the outset of the pilot, an evaluation framework was developed for the research. It was structured around 4 broad objectives, underpinned by a series of success factors. The objectives against which the Framework was evaluated were the extent to which it
1. Measures outcomes for individual clients
2. Measures impacts of the programme across local authorities
3. Does not place excessive data collection burdens
4. Provides useful information for national public and ministerial reporting.
9. A combination of quantitative and qualitative research methods were used to evaluate the Framework:
(a) Surveys of providers participating in the pilot at the outset of the pilot and then again at the end of the pilot. All providers responded to the first survey and 17 (65%) of the exit survey. Telephone interviews were also undertaken around the mid point of the pilot with all providers.
(b) Visits to 14 providers, two from each of the 7 local authorities. During these visits, assessment interviews were observed, staff from the service providers and service users were interviewed. Overall 33 service users and 35 staff were interviewed.
(c) Visits to the 7 pilot local authorities to interview Supporting People staff.
(d) Feedback from participants provided on ad hoc basis during the pilot on issues or difficulties.
10. The intention had been that the final dataset of cases assessed during the pilot would be reviewed as part of the evaluation but the value of this data proved limited because of inconsistencies and inaccuracies.
Measuring Outcomes for Individual Clients
11. Providers had concerns about the accuracy with which they, or their staff, were completing the Framework. In part, the concerns related to unease about the judgement involved in placing a client against a particular category within the matrix. Some providers would have been more comfortable with a system which involved tighter questions (tick box style). Some providers also commented that the wording of the matrix compounded the difficulties and that tailoring the wording to the particular client groups and their needs would be helpful.
12. There were particular concerns about accuracy in relation to the counterfactual (the 'current situation without support') score. Providers were not confident about their skills in assessing the likelihood of particular situations arising if services had not been in place.
13. Accuracy issues also arose where providers and service users did not agree about the appropriate 'score'. This raises a wider issue about whose 'score' the matrix should reflect and who 'owns' the information entered into the matrix.
14. Overall, in the services which were observed during the site visits, there was a high level of consistency in completing the Outcomes Framework by different support workers within a service. However the evidence from the dataset of cases and from the local authorities was that there was significant variation between services in the process used to complete the Outcomes Framework: this variation applied to services catering for the same client group or providing similar types of housing support service, as well as to those operating in different local authority areas
15. There were differing views about the extent to which the Framework was suitable in its current form for all client groups. Some of the concerns related to the current structure and terminology within the Framework, with a view from providers that it needed to be tailored more to the experiences and circumstances of particular groups.
Measuring Impacts across each Local Authority
16. For the Outcomes Framework to be capable of assessing outcomes across different client groups and across the local authority, a pre-requisite is that there is consistency and accuracy in the data collected by providers; there was limited evidence of this in the pilot.
17. Only one local authority felt that there had, overall, been any consistency of approach in their area and that this was because of the high level of support they had given to providers throughout the pilot. Local authorities commented that there were obvious inaccuracies in the data received from providers. There were also comments that both accuracy and consistency may have been improved if there had been clearer guidance to the service providers from the beginning of the pilot.
18. Most local authorities reported that the data from the Outcomes Framework added value as there was evidence collected in the Framework which was not available elsewhere. One of the main benefits was the picture it provided of the 'distance travelled' for individual clients. Most local authorities said that they would use the information provided from the Framework for service reviews. Some local authorities saw the Framework as having the potential to be useful as a cost/quality tool which would help them to link future funding decisions to service quality.
19. There was no evidence of information being shared across service providers and between services, although there was a willingness to look at how this might be possible.
20. Several of the local authorities had thought about how the information from the Framework could be aligned with datasets they were currently using. There was also scope for linkages with information held by social work departments. Others mentioned that there should be integration with the Single Shared Assessment and Community Care outcomes which were being developed at the same time as the Outcomes Framework model was being piloted.
Data Collection Issues
21. One of the key issues for providers and local authorities in considering the value of the Outcomes Framework was the extent to which it might increase or decrease current data collection burdens.
22. A number of the local authorities already collected additional information from services, as part of their internal monitoring systems. This meant that there were mixed views about whether the Framework could become the sole data request to providers. There were no issues about duplication or overlap with other information as the Framework brought a new set of evidence on services.
23. A perceived downside of the Framework was that the qualitative information from the service user was translated through the Excel spreadsheet into a series of numbers with a lack of contextual text information, which several local authorities said they also needed.
24. There were a number of detailed technical issues which the local authorities raised in relation to the software and the resulting data collection and transfer. Some of the issues had been raised with DTZ (who managed the pilot) by the providers and the local authorities during the course of the pilot. Others came to light as the local authorities attempted the final collation of data for analysis.
25. Both the local authorities and the providers seemed able to separate out the value of the Framework from any technical glitches. Equally, all were clear that if the framework was to be utilised more widely, the issues they had identified would have to be addressed ahead of any wider roll out.
26. Providers reported that by the end of the pilot it took an average of 20 minutes to complete the Framework, although the time varied for individual providers from five minutes to more than an hour. Longer interviews tended to be associated with particular clients, who required more time to engage with the process. For most providers the time required reduced during the course of the pilot as they became more familiar with the Framework.
27. Providers which saw the Framework fitting into their current processes for supporting planning and on-going service user review were much less likely to see the addition of the framework as a burden.
28. All of the local authorities had found the collation of data at the end of the pilot time consuming although they also had positive suggestions on how this could be alleviated.
29. A few providers had looked at how the model could be integrated with other datasets which they used already within their organisations. Where managers and other workers saw it fitting into the current support plans and reviews they had in place for clients or into datasets they currently used for monitoring service performance and quality, they were much more likely to be supportive overall.
30. The services where it was thought the framework might work less well were: longer term services; smaller services (where the resource issues might be too burdensome) and some specific client groups. In this last instance, services for people with physical disabilities were specifically mentioned as being one set which the framework might not work for.
31. There were mixed views about the resource implications of using the Framework in different settings (for example, in an office or the service user's home).
Information for National Public and Ministerial Reporting
32. This final objective sought to establish the extent to which the Outcomes Framework would provide useful information capable of being used for the purposes of reporting the impact of housing support services funded through Supporting People at a national level to ministers, other stakeholders and the general public.
33. The pilot did not produce evidence to reach any conclusion. This is very largely because of the level of inaccuracies and inconsistencies within the data collected. However, the evaluation of the pilot did not conclude that lack of accuracy and consistency was inherent to the Outcomes Framework.
34. In principle it remains possible that sufficient accuracy and consistency could be achieved to enable aggregation of this data across Scotland and analysis nationally about outcomes for particular client groups receiving housing support services. This particular objective may no longer be relevant in its current form: however the Supporting People (Housing Support) Outcomes Framework offers a methodology to deliver evidence about the impact on individual service users of particular service interventions
Conclusions and Recommendations
35. The role of the Framework should be seen as a means of measuring the impact of housing, care and support services, particularly those which are low level and have an emphasis on prevention.
1. The Outcomes Framework should be promoted as a tool which offers the potential to measure the impact of service intervention, particularly where those services are low level, non specialist and have an emphasis on prevention.
36. While the Scottish Government should no longer have the principal ownership of the Framework, it still has a very important role to play, principally in making improvements so that it becomes a more effective tool.
2. Ownership of the Outcomes Framework should lie at a local level, either with local authorities or with individual service providers. Ownership by local authorities will have the benefit of delivering a greater degree of consistency across providers.
3. The Scottish Government has an important role in further developing the Framework to make it into a more effective tool, and to explain its potential to local authorities, service providers and other stakeholders.
37. The 'counterfactual' proved difficult to implement; it caused anxiety for some service users and providers also felt that they lacked the necessary skills to identify what the 'pathway' might have been in the event of no service being in place.
4. The Scottish Government should explore alternative approaches to the assessing the preventative benefits of services and incorporate this within the revised Framework.
38. Increased consistency and accuracy in the completion of the Outcomes Framework would be necessary if the data was to be of value. Recommendations in relation to achieving greater accuracy and consistency are:
5. Development of a detailed guidance providing interpretation of the terms within the Framework and on use of the Framework. This should be undertaken nationally, but involve both local authorities and service providers.
6. Development of a viable IT platform ( e.g. using Access), accompanied by detailed guidance.
7. The provision of training before providers begin to use the Framework, the responsibility for which would depend on the agency driving its introduction
8. Consideration of the possible use of a personal identifier for services users which could be used to track their progress across a range of voluntary and local authority services.
39. Overall there was evidence that the Framework was acceptable to users of housing support services, although greater explanation of its purpose should be provided by providers. Recommendations in relation to service users are:
9. Development of the terminology used in the Framework (in conjunction with providers) so that it is more appropriate for client groups but does not lose the 'read across' between different services
10. Alternative formats of the Framework and the matrix in particular, should be available so that it can be understood by those with differing communication skills.
11. The frequency of assessment should differ, depending on the client group, with greater frequency for those whose needs change quickly and reduced frequency for those whose needs change more slowly.