Scottish Government

DEVELOPMENT OF TOOLS TO MEASURE SERVICE USER AND CARER SATISFACTION WITH SINGLE SHARED ASSESSMENT

CHAPTER TWO REVIEW OF EXISTING PRACTICE IN RELATION TO ASSESSING USER AND CARER SATISFACTION

Aims

2.1 The review of existing practice aimed to map developing practice across Scotland. We looked for evidence of how practitioners were responding to particular communication challenges and different approaches to obtaining views about "assessment of community care needs".

2.2 It was intended that the review would inform and engage a wide range of practitioners and managers in the project, and would in turn inform the project.

2.3 The information hoped for included:

• the range, styles and types of local consultation with service users

• the development of single shared assessment practice in different areas

• the nature of effective practice on which the project could build, particularly in the areas of enquiry seen as challenging

• contacts for taking forward the project.

2.4 Research literature had suggested that whilst obtaining views about satisfaction with public services was problematic generally ¹, feedback on assessment processes was particularly difficult to obtain. There were a variety of reasons for this, the primary one being that many people did not realize they had been "assessed". Further, several groups of people who presented communication challenges had been left out of major studies into determining user satisfaction with public services.

Survey method

2.5 A 6-part questionnaire was posted on 28 August to health boards and trusts (42), local authorities (32), health councils (15), community care forums (33), service providers (43 in the voluntary sector, 20 in the private sector) advocacy (13) and other organisations working with disabled people (3). Replies were requested by 15 September. Respondents were invited to send examples of their work.

2.6 The questionnaire sought retrospective information about agencies' practice in seeking the views of service users and carers. It also asked about current and future arrangements for carrying out single shared assessment. The questionnaire is reproduced at Annex 3.

Responses

2.7 Few responses had been received by the deadline; following telephone reminders a total of 49 questionnaires were returned by 30 September, with a further 8 arriving in the following 10 days, bringing the final total of responses to 57. A number of joint responses were received: therefore, the total response represented 36% of the organisations contacted. .

2.8 Respondents are listed in Table 1, using NHS areas as the primary unit of geographical distribution. All of the responses listed described experience of seeking service users' views and in some cases carers' views.

2.9 Given the location of the project in the Joint Future agenda and the multi-purpose nature of the questionnaire, it was unsurprising that statutory authorities chose to respond in different ways. The majority of statutory authorities sent joint responses.

2.10 As well as differences of department and perspective amongst health and social work authorities, there were also very different approaches to giving information between all respondents, ranging from generalized to detailed. Many sent supplementary information, or offered to send it, which was of great practical value to this project.

2.11 Some respondents were evidently more affected than others by the tight timescale, and several targeted respondents said they were unable to reply even within the extended timescale.

Analysis

2.12 Responses were analysed using the questionnaire as a framework. An overall picture emerged, across geographic areas and types of organization, of practice in seeking service users' and carers' views.

Table 2.1 Distribution of respondents

Findings

The big picture

2.13 The survey confirmed that:

• a culture of seeking service users' and carers' views about community care is well developed in most areas of Scotland, both by the responsible authorities, by service providers and of course by agencies which represent service users' interests as their primary function

• most areas have a mixture of consultative structures and methods, used for different purposes

• some authorities and organisations stand out in having a strong culture of engagement with service users and commitment to finding effective ways of meeting the challenges

• seeking the views of older people and of carers is well established; approaches to other groups is less universal, although the development of Partnerships in Practice for people with learning disabilities has stimulated much recent good practice with this group of people

• most feedback from service users is obtained for service development purposes, but several respondents also reported how they checked whether individuals' needs were being met satisfactorily

• much reported research activity was related to recent policy requirements such as the Best Value regime, and in some providers' work, to new National Care Standards.

2.14 Recent studies ² and our own knowledge of specific areas of Scotland indicate that much more service user feedback work has been carried out than was reported to us.

People whose views are sought

2.15 From the responses to the survey it is clear that a wide range of people are asked for their views on the services they use.

2.16 Most respondents in health and social work services gave examples of types of exercise to find out service users' views of different services. Some of these exercises, such as Best Value reviews of older people's services or of services for people with learning disabilities, related to particular groups of people.

2.17 Other reviews, for example of home care services, covered several groups of needs, as did studies of new services, such as a Rapid Response Team.

2.18 Health respondents reported exercises to obtain patient views of different aspects of their experience of hospital.

2.19 Regular individualized means of seeking views- in the process of assessment itself- were also described as applying to all people being assessed for services.

2.20 Several social work and joint respondents referred to the many consultative groups in their areas, most corresponding with sets of needs recognized in Community Care Plans, such as learning disability and mental health issues.

2.21 Older people were the group most frequently mentioned by health and social work respondents, for all types of enquiry. There were few reports of consultations with people with dementia, homeless people, and people with drug and alcohol problems.

2.22 2 respondents gave a comprehensive rather than illustrative answer, one listing reviews ranging from HIV/Addictions Services to Transport Services for People with Learning Disabilities (Edinburgh joint response), and the other (North Lanarkshire Council) setting out their extensive framework for consulting service users across services.

2.23 Service users' views of services provided by health and social work authorities have been further canvassed by health councils and by community care forums, in some cases through commissions from the providing authorities, in others in response to concerns of their members. Amongst the 4 health councils and 5 community care forums who provided information, a wide range of service user groups and purposes of investigation was covered (one health council did not provide this information). These included:

• evaluation of a new Rural Homecare Service (Argyll and Clyde Community Health Council)

• annual visits involving users of services such as dementia assessment and stroke rehabilitation (Grampian)

• a network of local forums working regularly with service users to feed-back their views (Highland Community Care Forum)

• a comprehensive list of surveys and evaluations across most types of social care service (Borders Voluntary Community Care Forum)

• a relatively new forum (Mid-Argyll) had concentrated on seeking the views of people in a psychiatric hospital and of people with learning disabilities and chronic illness using a local resource centre.

2.24 A further significant channel for service user views on statutory services was the advocacy organisation for people with learning difficulties in the Stirling Council area - Quality Action Group. This organisation also reported on how it self-monitored. Other advocacy respondents reported only on their experience of seeking views of their own organisations as did the Lothian Centre for Integrated Living.

2.25 These advocacy organisations together with providers in the voluntary and private sectors gave an indicative picture of the contribution of service user views to non statutory services. Most private care providers reported regular reviews and feed-back from their service users with needs across the community care spectrum. Voluntary sector respondents provided for people with learning disabilities and for people with physical and neurological disabilities. Between them they provided a wide range of service type. For several of them, empowerment of service users to control how the service worked for them was a primary objective.

Who asks people for their views?

2.26 It is widely acknowledged that the status of the person asking the questions can strongly influence the answers given. However, striving to avoid bias through, for example, using independent enquirers can be expensive and often does not fit into some enquiry routines and purposes.

2.27 Across all respondents there was wide involvement of their own staff in at least some aspects of enquiries.

2.28 Within health and social work, attempts to create some distance between questioner and respondent varied between authorities and between purposes of enquiry. At one extreme, authorities who had evaluated assessment processes had employed independent researchers, or in one case a staff member supervised by a university. At the other extreme, several authorities appeared to involve staff with whom service users would associate the service they used. More often, "HQ" staff, or staff with a "user involvement" remit were involved in individual and group enquiries.

2.29 Routine enquiries about service impacts on individuals were carried out by assessment and care management staff.

2.30 Independence was promoted by several authorities by commissioning health councils and community care forums (see above) or other local voluntary or service user groups. In one area (Grampian) there was a high incidence of involving service users in carrying out monitoring, including an Older People's Consultation Monitoring Group and a Mental Health Services Evaluation Group.

2.31 Community care forums and health councils also trained or supported their members to carry out enquiries of service users, as well as using their own staff or occasionally commissioning consultants.

2.32 External inspections and audit were another means of independent enquiry (e.g. Quality Improvement Scotland, the former Scottish Hospitals Advisory Service, and Audit Scotland). Several authorities reported that Best Value reviews had been carried out alongside Audit Scotland.

2.33 Amongst respondents reporting on their own practice in the independent sector, private providers all used their own staff. Some used managerial or supervisory staff to make the enquiries rather than front line staff.

2.34 Voluntary sector organisations, including the advocacy organisations, aimed to avoid bias by using staff not known to the person, or at least not directly involved in the service being discussed. A further factor for several organisations was meeting the need of some service users for support from a familiar member of staff. Almost half of the organisations had used independent researchers, and some had brought in independent facilitators at certain stages of an enquiry.

Approaches and methods

2.35 Many respondents demonstrated keen awareness of the essentials for getting a valuable return on their research efforts by:

• getting the balance right between quantitative and qualitative approaches to achieve the purpose of the exercise

• asking relevant questions clearly and responsively taking account of targeted communication needs

• facilitating responses

Health and social work authorities

2.36 Amongst health and social work respondents many methods were employed, singly or in combinations. These included:

• postal questionnaires for larger scale surveys

• individual interviews

• focus groups.

2.37 Most authorities had well established "user group" structures or worked through community care forums to reach groups of service users and carers (for example in Highland and Scottish Borders). A few (for example East Ayrshire) also set up projects which brought in service users to advise on particular developments. There was wide-spread use of residents' councils and other groups attached to particular services.

2.38 Although focus groups were the most commonly employed means of bringing people together, major events (e.g. an older people's conference of 220 delegates in East Ayrshire) and smaller local events and seminars were also reported. There was some evidence of development of informal means of listening to their service users (for example Falkirk Council's "listening lunches").

2.39 It was more difficult to get a clear picture across all public authority respondents of the elements that went into their enquiry processes.

2.40 7 out of the 24 health and social work respondents reported that they took expert advice on conducting enquiries, and 6 that they involved service users in designing their enquiries. Expert advice included internal research departments, NHS Audit Teams (e.g. Aberdeenshire), Audit Scotland (e.g. on the service user involvement element of Falkirk Council's quality management programme). It also included advice on communication modes, for example when seeking views of people with learning disabilities.

2.41 Of the sample questionnaires sent in, some appeared to us much clearer and more inviting than others. Expertly developed enquiry methods were used by several respondents, e.g. Servqual Gap Analysis (for groups and for postal questionnaires). Some respondents had added incentives to large distribution questionnaires (e.g. a prize draw for a meal in a local restaurant in Orkney).

2.42 Most authorities reported on how they had given information on the purpose and background to their request for feed-back, usually by letter and often by using local groups as a channel for giving explanations. Some enquiries had proceeded by giving information and inviting participation.

2.43 A few respondents reported publication or other type of feedback of the results of the enquiry exercise as a regular part of seeking service users' views. This sometimes included the influence the findings would have on future service development.

2.44 Response rate information from authorities was sparse and general. More specific information included:

• questionnaires about a new home from hospital service 42% - 58% (North Glasgow NHST)

• response rate across wide range of enquiry modes 20%-40% (Edinburgh)

• high for independently conducted, multi-mode review (Clackmannanshire)

• low for postal questionnaires (several respondents)

• high for events, focus groups (several)

• low for postal questionnaires and events (Highland)

• postal questionnaire 17% response = 334, 10% of 65+ population (Orkney)

• 100% agreed to interviews in locality review (Shetland).

Health Councils & Community Care Forums

2.45 Working within a smaller compass, most health councils and community care forums were able to give clear information about the steps involved in the different approaches they employed, and were also clear about what was effective. The 2 health councils (Grampian and Fife) and 2 community care Forums (Borders and Highland) which had most experience had used a variety of methods, including postal questionnaires, telephone and personal interviews and focus groups.

2.46 Health council information focused on how they recruited respondents and on site visit enquiries, whilst community care forums conducted enquiries through their service user members and contacts, as well as other research methods. Both types of organisation reported using focus groups and events. Community care forums laid emphasis in all their work on keeping close to service users, involving them in the design and conduct of the studies. Health councils reported high response rates, particularly to exercises which were carried out on-site (e.g. clinic waiting areas) and to events.

2.47 The advocacy group which carried out consultative work about issues affecting people with learning disabilities is reported more fully below.

Service Providers

2.48 Reports of practice and methods from voluntary sector providers ranged from standard questionnaires to detailed approaches based on putting person centred standards at the heart of the user's experience of the service, with checking out processes adapted to individual communication needs. One organisation reported on service users' involvement in a Care Commission inspection.

Across this range most types of method were used, with a higher incidence of questionnaires administered in face to face interviews and in many organisations supported by graphics and other communication tools.

2.49 Standing out were the comprehensive approaches of 6 projects run by the Action Group. Annual reviews were based on questionnaires checking on the standards of service, which reflected national standards, were endorsed by service users and were an every day part of people's dealings with the service. Reviews which were voluntary were supported by a service employee and communication was adapted to the person's needs which varied between the projects as well as between individuals.

2.50 An advocacy organisation led by service users emphasized the importance of place to enable people to be "open and honest", together with a wide selection of communication modes and explanatory tools (Quality Action Group). (This organisation also carried out consultative work about issues and services affecting people with learning disabilities.)

2.51 Most respondents reported a high response rate, except to some questionnaires.

2.52 Private service providers reported a wide range of approaches to seeking feedback;

• most reported regular, informal checking with service users

• several referred to annual and periodic surveys

• most had annual reviews

• many had more frequent reviews.

2.53 Methods included

• questionnaires by post and telephone

• one to one meetings

• 2 providers reported having or introducing anonymous surveys.

2.54 With one exception, response rates were reported as high.

The value of feedback exercises

2.55 Most information about service users' perceptions of the effectiveness of feedback exercises came from respondents outwith the statutory sector. More respondents overall were able to give their own assessments of effectiveness.

2.56 Private providers reported that service users' views of feed-back exercises were generally favourable:

• "most felt it was effective and their opinion mattered"

• being given choice of response method was welcomed

• a preference for "face to face discussions with managers" was reported

• it was also recognised that "some do not like to say anything negative - the approach has to be right",

• responses varied from "it's ... a waste of time . . to . . good idea, pleased it's being done"

2.57 In the voluntary sector, services which had empowered people to expect person centred standards inferred from the level of involvement in annual evaluations that people approved of them. Respondents were nonetheless concerned to improve and simplify their approach further. They and other voluntary sector providers also received comments that exercises had been "useful".

2.58 Several respondents across all sectors said that service users were more concerned with services being effective than whether the feedback method had been effective. One respondent answered the question in terms of the action produced by feedback exercises for contributors, rather than giving comments made at the time (Borders Voluntary Community Care Forum).

2.59 The few health and social work respondents who provided information on service users' perceptions referred mainly to events which had been enjoyed and well rated.

2.60 Orkney received appreciative comments in response to postal questionnaires as well as events:

"Thank you for asking my views".

2.61 Edinburgh emphasized the importance service users and carers attached to feedback on feedback:

"Most people have been very happy to be asked their views, but like to feel their time has been spent for good reason. It has been very important for them to receive information about results of the consultation exercise and about what will happen with the information gathered, how it will be used etc. . . . . . some consultations have very speedy impact and that is important for people to see"

2.62 North Lanarkshire's detailed analysis of its comprehensive programme of involving service users and carers also supplied their views on most of the exercises they had contributed to. Well rated approaches were:

• a series of information events which sought their views prior to drafting of the Community Care Plan

• in depth development of questions for people with learning disabilities

• one to one interviews, focus groups and questionnaires, when used for appropriate purposes.

2.63 Several questionnaires had received a mixed response - approved by frail older people, but not by people with any degree of dementia. Respondents' own assessments of their feedback exercises gave fuller information both of the overall value of the exercise to them and/or service users or carers, and about what worked well and what did not.

2.64 Perceptions of the overall value of feedback exercises included:

• "Bathing services: the questionnaire enabled us to determine need, cost and provide a service at an appropriate level" (Orkney)

• The outcomes of an independent review of a shared assessment process and recording system in 2000 was "the building block for the single shared assessment" (Clackmannanshire)

• "It is reassuring to all concerned when service developments and changes can be demonstrated to be soundly based in the wishes of users and carers" (Edinburgh)

• "Good experience. We generated feedback to staff and patients/carers. Changed practice, attitudes, delivery at service" (West Lothian NHST)

• "Both exercises provided health and Social Care with key areas for action planning and agreeing the way forward for a range of services" (NHS Lanarkshire)

• "Feedback is instructive. Our person centred planning process puts the service user "in charge" so it is natural for them to comment." (British Red Cross)

• Private providers - most approaches worked well (with some provisos about simplicity and the need to give reassurance about criticism) and gave valuable insights into how they should be running their services

• Varies depending on mandate for action produced -

"User/carer led consultation projects can bring pressure to bear on the authorities . . . we try to work in partnership with Health, Social Work and other agencies to address the issues which emerge" (Borders Voluntary Community Care Forum).

2.65 Perceptions of approaches that worked and did not work showed some consensus across all respondents and some that were particular to organisations' own focus.

What works well

2.66 Comments included:

"This is often about choosing the right methods and offering alternatives. Not everyone wants to fill in a questionnaire. Not everyone wants to speak into a video camera. If it is appropriate for 'some' people (especially if those people are not used to being listened to) then it is a success" (Quality Action Group)

"What works most effectively is to take time to find the right people to involve, make sure they are comfortable and well briefed. Talk to them in a setting which suits them and ensure that they get feedback on the process and finally to make sure that they have the support they require whatever that is." (Highland Community Care Forum)

"The personal touch, conversations carried out in a non-threatening environment, over a cup of tea, facilitated the gathering of data with a breadth and depth impossible with a simple survey/questionnaire tool." (Argyll and Clyde Health Council)

• Independent researchers, independent interviewers, independent organizations which "are trusted to pass on information" (many respondents)

• "The most effective consultations so far have used simple questionnaires followed up with well-planned and organised focus groups (Edinburgh and others)

• Staff commitment to ISO standard for involving service users and carers in day and residential services has encouraged good practice (Clackmannanshire)

• Focus groups, conferences, meetings, if well targeted, well organised (several respondents)

• Feedback sheets following provision of service

• Convenient location or captive audience

What has not worked well

2.67 Specific examples given included:

• Sample too small

• Questionnaires too long (several respondents. One external inspection questionnaire had 90 questions!)

• A public consultation process with people with learning disabilities

• Questionnaires and public meetings have not often worked well (several respondents)

• "Over-consultation", without feedback on results

• Questionnaire given to people following assessment.

2.68 Several respondents who were generally satisfied by their own and service users' assessments of their enquiry approach considered that they had to persevere with finding approaches that engaged all of their service users and enabled them to give unbiased comment (for example, the Action Group). This was considered essential if they were to deliver person centred services which were fully responsive and engaged the service user.

Experience of assessing satisfaction of service users and carers with assessment procedures

2.69 Replies were received in the following 7 categories, together with supporting documentation from most of the respondents. These were all extremely helpful in developing our "assessment of assessment" methods.

• Organisations which had not conducted assessment of assessment exercises but had encountered people's views about it through other service enquiries

• Organisations which were preparing for single shared assessment and had experience of involving service users in their preparatory work

• Organisations which advised service users about assessment rights and aimed to educate and empower people to use assessment procedures effectively

• Organisations which had carried out earlier assessment of assessment exercises

• Organisations which had evaluated pilots of single shared assessment

• Organisations which were preparing for evaluations

• Organisations which commented on documentation.

2.70 Between them their evidence confirmed:

• A range of issues which people considered important

• A gap between what professionals see as a priority for the process and what people see as priorities in their lives

• People's tendency to be more concerned with services and their availability than with assessment

• The need for extensive education and information of service users and carers if single shared assessment is to meet its aims of empowering people to be better supported by services

• Some of the individual ways in which information needs to be exchanged

• Evidence of the comparative effectiveness of different conventional "measurement" approaches

• The need to have different ways of coming to conclusions about whether the process is working, since people respond in different ways to different approaches (e.g. focus groups provided more extensive and some contradictory responses to a questionnaire survey.)

Working with communication differences and challenges

2.71 A wide range of resources is in use to assist communication, although awareness and use of them appears to be uneven across different areas. Examples given by several respondents included graphics, communication systems, IT, video, drama, role play, BSL signers, induction loop, and large print.

2.72 Implementation of The Same as You (Scottish Executive, 2000), appears to have increased physical resources, and training and staff resources. A number of authorities were now working directly with groups to develop skills and confidence. Careful preparation to make consultation events accessible were described by several respondents and comprehensive documentation provided. This included a striking report, What we think, of a conference, Shaping the Future - the Forth Valley Wide Learning Disability Strategy. Organised by the Council and Health partners in the area, facilitators for the day were trained by the Quality Action Group.

2.73 Large authorities are more likely than smaller ones to provide access to a range of specialist staff, such as equality officers, speech and language therapists, specialist organisations such as Alzheimers Scotland/Action on Dementia, and specialist agencies such as translating and interpreting services. Several respondents reported using advocacy organisations.

2.74 Organisations providing services for people with different communication needs have extensive expertise and commitment to extending it further, as reported above. They have generously shared their knowledge with the project and given us valuable insights.

2.75 Amongst smaller organisations community care forums have a repertoire of communication aids including Forum Theatre and art therapists and work closely with colleagues organisations concerned with specific conditions. Grampian Health Council demonstrated exhaustive ingenuity in finding communication modes for hospital consultations and in the range of visual and dramatic approaches employed.

2.76 The last word in this section should go to the Quality Action Group:

"We use a range of settings and techniques. We find that making people feel comfortable and feel fully briefed, is the best way to make them feel confident that their views are valued. Sometimes, though, the questions might seem like they are silly, or missing the point; sometimes people are very clearly telling us about a service, but people who run the service don't listen properly. These, often very clear, views, are sometimes missed because the person is labelled as having profound learning disabilities, or challenging behaviour."

Summary

2.77 This review of existing practice in relation to assessing user and carer satisfaction sought the views of a wide range of statutory, voluntary and private sector organisations with an interest in community care. A total of 201 questionnaires were sent out: the return rate was 38%.

2.78 Key findings from the review were:

• a culture of seeking service users' and carers' views about community care is well developed in most areas of Scotland, both by the responsible authorities, by service providers and of course by agencies which represent service users' interests as their primary function. Less work has been undertaken on evaluating user and carer satisfaction with assessment processes

• most areas have a mixture of consultative structures and methods, used for different purposes, with some authorities and organisations standing out in having a strong culture of engagement with service users and commitment to finding effective ways of meeting the challenges entailed

• seeking the views of older people and of carers is well established; approaches to other groups is less universal, although the development of Partnerships in Practice for people with learning disabilities has stimulated much recent good practice with this group of people

• most feedback from service users is obtained for service development purposes, but several respondents also reported how they checked whether individuals' needs were being met satisfactorily. One respondent demonstrated how people had been empowered to make best use of its varied services through a person centred engagement with all aspects of their service experience

• much reported research activity was related to recent policy requirements such as the Best Value regime, and in some providers' work, to new National Care Standards

• many agencies used their own staff to elicit feedback, but there were instances where independent evaluators were engaged

• postal questionnaires generally produced the lowest response rate of all methods used by agencies

• the gap between what service professionals see as a priority for the process and what people see as priorities in their lives

• people's tendency to be more concerned with services and their availability than with assessment

• the stage of implementation of single shared assessment varied significantly across Scotland

• a wide range of resources were available to assist communication, but awareness and use of them appeared to be uneven.