Section Two: Case Study Progress and Initial Successes
2.1 This section provides a short summary of the initial successes to date in each of the case studies against the local success factors. In each case, it is too soon to expect to see much of a demonstrable impact on service outcomes. However, even after the relatively short duration of the case studies, a number of conclusions can be drawn about the nature of the process and the opportunities and constraints that have influenced the CitiStat Pilot in each area.
Summary of the distinctive features of the case studies
2.2 Table 2.1 below summarises the key distinctive features of the case studies.
2.3 Each case study has adopted a different structure for the Panel. Each has involved Chief Executives although the role of elected members/non-executives has varied. The case studies have adopted a different substantive focus ranging across environmental quality of life, hospital bed capacity, access and treatment issues and delayed discharge from hospital. None of the case studies adopted a particularly adversarial approach to the Panel process, with some taking a more deliberately collaborative approach to suit their organisational culture and style of working.
2.4 In each case study, the process has produced better quality data and this has provided the basis for improvements in performance. There have also been other outcomes such as generally enhancing clarity, understanding and focus in decision-making. The process has also highlighted some weakness in the performance management of partnership arrangements. The CitiStat process has also had varied unanticipated outcomes across the case studies, generally related to attitudinal and organisational culture.
2.5 Each case study has different existing performance management and accountability processes; in general CitiStat has compared well with these approaches and has been seen as a useful model.
2.6 Staffing resources for the process have been significant, although some found ways of focusing resources for analysis and the support from the Scottish Executive PIU Team was important in all cases.
2.7 High-level support for the process has been an important enabling factor in all cases. The case studies have experienced a number of constraints on the CitiStat Pilot process including reorganisation, a lack of integration into wider structures and processes and wider buy-in to the process.
2.8 There is continuing interest in CitiStat across all four case studies involved in the Pilot. One is continuing with the Pilot process with a reframed focus and Panel structure; two are continuing beyond the Pilot phase extending and integrating it in a more formal manner; and one still has to consider recommendations on the future use of the CitiStat model, resource implications, scope for integration and applicability for cross-agency working.
Table 2.1: Summary of the distinctive features of the case studies.
| City of Edinburgh Council | Aberdeen City Council | NHS Tayside | NHS Ayrshire & Arran (A&A) |
|---|
Structure | Panel chaired by Chief Executive. Council Leader sat as observer | Panel co-chaired by Chief Executive and Leader. | Dual structure: high level Panel chaired by Chairman of Board with Chief Executive on Podium. 2 nd 'Bedstat' Panel chaired by Chief Executive. | Partnership structure: Panel comprises CEOs from A & A NHS and Chief Executives from 3 CHP areas. |
Role of elected members/non-executives. | Very limited involvement. Mixed views on value of this. | Considerable involvement of the Council Leader has been critical. | Leadership of Chair and CEOs vital to making process work. Non-executive Board members able to gain a better understanding of the business. | Non-executives not involved in Panel. |
Focus of Pilot | Environmental quality of life in two Departments. 'Liveability issues' seen as key public issues in the City. | Environmental quality of life across three Neighbourhoods. Extended to include roads & lighting. | High-level Panel: SEHD ' HEAT' targets. Bedstat: bed capacity/ access and treatment issues | Delayed discharge/ better care for the elderly. |
Timetable | 6 cycles completed Sept 05- Feb 06 | 6 cycles completed Sept 05- Feb 06 | 6 cycles completed Sept 05- Feb 06 | 2 cycles completed Nov 05-January 06. |
Experience of the Panel process | Not particularly adversarial. Process has been challenging, but largely welcomed by Podium. Standard of questioning improved over time, with clearer focus. | Deliberate collaborative approach adopted by Panel. Semi-formal and participative, although challenging. | The High-level Scrutiny Panel was more hard-hitting, although not particularly adversarial. The Bedstat Panel was more 'round table'. Both Panels successfully adapted to the styles of the Chairs. | Significant reframing of CitiStat to suit partnership basis and preferred style of working. Achievement to bring parties together, but absence of Chief Executives from all 3 local authorities, despite formal commitment. |
Use of data | Process has produced better data. Highlighted gaps in information and data that is not particularly useful. Drilling down evident - discussion of wider influences and corporate responses. | Improved data and analysis - data more consistent, useful and rigorous. Still a need to further refine performance indicators and data. Highlighted issues of indicators for reporting purposes, rather than performance management. Early focus on refining data - later focus on driving real performance improvements. Improved understanding of performance management. | Process helped to refine accuracy, validity and quality of data for performance management. Briefing process supported appropriate and effective questioning by Panels. The use of time-series data facilitated discussion about trends and the linking of data sets. | Recognised limitations of data and need for qualitative data too. Recognised need to develop shared ownership of the targets around Delayed Discharge. Shifted from 'hot spot' of delayed discharge to whole system view of better care for the elderly. Initial efforts focused on resolving data concerns in short term. |
Use of data to improve performance | Specific improvements in sickness absence and other indicators. | Use of data led to new action taken to improve performance eg. fixed penalty notices, graffiti policy, litter round schools, staff absence. Improved performance outcomes in PAC ratings and sickness absence. | Strong action-orientation and relentless follow through on agreed actions; solutions focused rather than descriptive. Time series data allowed discussion of prospective management of capacity and demand. Provided overview of performance and focus on hot spots. | Agreed new target for delayed discharge of over 6 weeks. Process focused on getting better data. Subsequent use of improved data to identify 'hot spots'. |
Other outcomes | Panel process helped to convey high level priorities and clarified Corporate/Departmental responsibilities for implementation of policy. Some evidence of better working relationships between departments. | Greater clarity, understanding & focus to decision-making Ripple effect - Pilot extended to two new areas; roads & lighting and housing voids. Buy-in at highest level important. | Better and quicker decision making. | Highlighted weaknesses in partnership arrangements and performance management. Recognised tacit assumptions about governance and accountability arrangements. |
Relationship with other performance management and scrutiny processes | Some duplication - although rigour of CitiStat, analysis and use of data provides useful lessons for scrutiny processes. CitiStat seen as good model to align scrutiny processes and be more strategic about performance management. | CitiStat allowed involvement in design of performance framework to produce more meaningful indicators. Potential recognised for CitiStat to add value to existing performance management and reporting. | CitiStat compared very favourably to previous performance management and scrutiny processes, such as the six monthly Performance Assessment Framework. CitiStat seen as a good model around which to build a health board's governance structure. | Highlighted lack of any joint performance management of partnership agreements. Issues around reporting requirements on HEAT targets to SEHD. |
Unanticipated outcomes | Highlighting attitudinal differences amongst managers. | Signs of shift in organisational culture. Challenge process welcomed by respondents as an effective vehicle for learning. | Provided useful forum where Executives can seek support of non-executives and CEO. The additional time spent by CitiStat team members on internal and external communication. | Reframed role of CitiStat Panel to take on a high-level monitoring role. |
Staffing resources | Significant additional burden for analysis team, service managers and senior managers. Support from SE ( PIU) crucial. Seen as too early to expect to see return on investment of resources | Recruited CitiStat administrator to support data analysis. Support from SE ( PIU) critical to success of Pilot. | Significant internal input into data analysis and management of process - central to success. Level of input by CitiStat team only possible in short term as an add-on to current workloads. Engagement in co-facilitation model of evaluation. Support from SE ( PIU) critical to success of Pilot. | Much of analytical work 'outsourced' to Discharge and Capacity Planning Group with support from A & A NHS Strategic Planning and Performance team and SE ( PIU). High-level input to Panel proposed to be on less frequent basis than original plan. |
Contextual factors with have enabled the value of the Pilot | High level support from CEO crucial. | High level support from CEO and Leader crucial. | High level support from Board and CEOs crucial. Strong pre-existing commitment to performance improvement. Involvement of managers in setting their own targets promotes ownership and accountability. | Partnership model distinct from other case studies. |
Contextual factors with have constrained the value of the Pilot | Restriction of indicators used in the Pilot limited focus. Ambiguity about role of elected members limited potential impact and opportunity to influence and inform councillors. Lack of involvement of service Department Directors limited impact at Director level. Departmental reorganisation. | Pilot not integrated into wider structures and processes. Restructuring of Council. Limited attendance of Corporate Directors on the Panel. | Need to engage others in the organisation to understand the model and what it is trying to achieve. | Later start than other case studies. Need to get 'buy-in' to focus on delayed discharge - meant need to prioritise the development of a joint and genuinely shared agenda. Absence of local authority CEOs at Panel sessions, despite formal commitment. No representation from one local authority. Targets for delayed discharge hold the NHS to account rather than joint accountability with local authorities. |
Future interest in CitiStat | Pilot being assessed internally and consideration being given to resource implications, integration with other performance management and governance arrangements and applicability to cross-agency working. Recommendations on the future use of CitiStat in the Council will be considered during the early summer 2006. | Process will continue in the service areas beyond Pilot phase. Proposal to extend CitiStat to other service areas and integrate with existing performance management structures. | CitiStat is continuing beyond the Pilot phase - proposed to formalise it within Board and Committee structure. | Further internal meetings planned and may also be meetings with Local Authorities to look at specific issues that have arisen. |
Other comments | Process empowering for some managers. Opportunity to get credit and feedback welcomed. | Process provided Panel with good news stories. | Patient's voice needs to find a place in the model. There are some recent plans to incorporate qualitative data from public/ patient involvement information and the Scottish Household Survey. | To make CitiStat work on a partnership basis requires clarification of the joint accountability of the various organisations and shared prioritisation across the partnership. |
City of Edinburgh Council
2.9 The City of Edinburgh Council tested the applicability of CitiStat in the context of environmental quality of life. The implementation of the Pilot ran from September 2005 to February 2006. It focused on selected operational performance information relating to two Departments: Environmental and Consumer Services and City Development.
2.10 There have been some demonstrable actions and changes to improve performance arising from the CitiStat process. The CitiStat process gave more immediate feedback to managers and allowed for quicker actions to be taken to address performance. It has been a more proactive, analytical and evolving process than other performance management processes used in Edinburgh such as reporting to Scrutiny Panels.
2.11 The CitiStat process highlighted a number of significant gaps in information available to Departmental Managers. The process has helped to generate better data and has shown that the Council collects some data which is not particularly useful; available data which may be used for reporting on Performance Indicators, but which is not necessarily actionable data in terms of performance management. The process started to "drill down" into the data and go beyond more superficial answers. Both the Panel and the Podium members developed a more detailed understanding of service issues. The process has evolved and developed a clearer, more manageable focus. The participants have got used to how it works and have been able to adapt it to suit their circumstances.
2.12 Initial efforts to refine key data on sickness absence appear to have allowed managers to more accurately pinpoint where the issues lie and take appropriate actions to influence performance. However, improvements in sickness absence were not sustained in the following month and a longer period of monitoring would be required before drawing robust conclusions about these outcomes. There have been early signs of improvement in a number of internal processes and operational issues which may support progress towards longer term environmental outcomes.
2.13 The process has shown the need for greater clarity of responsibility for actions: where Departments are responsible and where issues require a corporate response. The management of sickness absence is an example of where the process made the respective responsibilities very clear and this may have contributed to performance improvement.
2.14 There is some evidence of the development of better working relationships between Departments. This is attributed to the involvement of the Chief Executive and the high level priority given to the process.
2.15 CitiStat has been part of wider organisational changes which impact on performance management. The involvement of the Chief Executive and two Directors has been important in giving a sense of clear and high priorities. The pressure this brings has been useful to Departments seeking to implement change.
2.16 The CitiStat model provided some important challenges to established cultures and ways of working. It has been useful in raising performance management issues within some Departments and highlighting limitations in their knowledge about their activities and areas of weak performance. Most participants have been comfortable with the approach although acknowledge it has been challenging, and are pleased that it has opened up debate on improvement actions. One of the unexpected outcomes of the process has been to expose some attitudinal differences amongst managers in terms of how they have responded to the process.
2.17 The experience of the Pilot is being assessed within the Council and consideration given to resource implications, integration with other performance management and governance arrangements and applicability to cross-agency working. Recommendations on the future use of CitiStat in Edinburgh are to be considered during the early summer 2006.
Aberdeen City Council
2.18 The Aberdeen CitiStat case study initially focused on environmental service issues such as graffiti, litter, dog fouling and waste collection, across three Neighbourhoods. This focus was later extended to include roads and lighting. The process ran from September 2005 until February 2006. A later development was the separate establishment of a small discrete project to apply CitiStat to housing voids, although this has not been part of the formal CitiStat evaluation.
2.19 In Aberdeen the CitiStat process has resulted in improved understanding, better communication and analysis and improved speed of decision-making. Key elements in the process have been improvements to data to ensure it is consistent, useful and rigorous, better analysis to test the validity of data and allow the development of key questions and subsequently a more focused and challenging discussion. The discussion worked in two directions; both to communicate the priorities of the Panel and the challenges facing staff in their attempts to improve performance. This debate was seen as the quintessential element of CitiStat that distinguished it from other performance management approaches.
2.20 A number of specific actions have been taken to improve performance. These were attributed to the pressure from the challenging timetable and frequency of sessions. The strong focus of the Panel on results and clear action points arising from the meeting for the next session has also been crucial. The attendance of key staff at the sessions has been critical to motivating staff to implement the actions. The evaluation sessions themselves have also supported the process and enabled reflection, discussion and refinement of the CitiStat model for the local context.
2.21 There has been an observed ripple effect across the organisation although the impact of the model has been limited to a few service and performance areas. This was due to the support for the process by the original participants, including the Chief Executive. This effect may have been limited due to the short duration of the Pilot, resistance to change in some areas of the Council, limited attendance of Corporate Directors on the Panel and the ongoing re-structuring of the Council.
2.22 There is some early, limited evidence of improved performance even though noticeable impacts were not expected in a short time frame. There had been significant improvement in two key performance indicators, including sickness absence. This particular success was attributed to the clear targets and deadlines set by the Panel which focused managerial attention and resources to address particular hot spots. Here, CitiStat appears to have driven an improved adherence to existing policies rather than anything more innovative or radical.
2.23 CitiStat has revealed early signs of a shift in organisational culture, now less tolerant of poor performance. Respondents welcomed the opportunity to account for their performance at the CitiStat sessions and found the process both challenging and an effective vehicle for learning. CitiStat has facilitated a review of some existing performance indicators and data sources within the participating services. This has led to the emergence of a new, more relevant and meaningful set of indicators and targets in which participants have been involved; this enhances the effectiveness of that data and makes it more suited to effective performance management.
2.24 The level of support from the staff from the Scottish Executive PIU working closely with Council staff at all stages of the CitiStat process has been critical to the success of the Pilot.
2.25 After the Pilot phase, Aberdeen City Council have confirmed that they will continue using CitiStat within the service areas used in the Pilot and may roll-out CitiStat to other service areas where such an approach could be beneficial. The intention is to ensure the approach is fully integrated into the Council's programme of Continuous Improvement and Performance Management.
NHS Tayside
2.26 The NHS Tayside CitiStat Performance Improvement Pilot started in June 2005 and has worked through a dual structure. A high level Scrutiny Panel focused on the Scottish Executive Health Department HEAT targets which are required within the Board's local delivery plans. A second 'Bedstat' (later re-named Taystat) Panel was established to take a more operational focus on the delivery of access and treatment services.
2.27 The NHS Tayside case study has been able to use the CitiStat process constructively to identify and respond to challenges presented in relation to the delivery and improvement of access and treatment services. The process has enhanced the existing commitment within NHS Tayside to performance improvement. NHS Tayside have successfully adapted the CitiStat model for their own purposes, using the more direct elements of the model to differing degrees at the high level Scrutiny Panel and Bedstat Panel to suit their organisational culture and individual chairing styles. The CitiStat process has been subject to continuous monitoring and improvement through an action learning/rapid cycle change model within the CitiStat team itself, and through co-facilitation of the evaluation sessions at the end of Bedstat Panel meetings.
2.28 The high level commitment to the process, the leadership of the senior executives and the commitment of the non-executives have all been essential to the success of the process. This level of involvement has helped to embed the processes of improvement, scrutiny and accountability within the organisational culture and there is a commitment to continue the process after the Pilot period has ended.
2.29 The process has enhanced the quality and analysis of the data to generate service improvements. Key to this success has been a more individualised and focused approach to data needed for service improvement around hot spots, rather than using that data which can be measured or is available because it has been collected historically. The co-generation of these hot spots across acute and primary care boundaries both demanded and promoted dialogue and collaboration across the wider system. This has improved communication and working together and prospective performance management in the short term. Progress has also been made in the partnership arrangements with local authorities and non-executives are better informed about performance issues.
NHS Ayrshire and Arran
2.30 NHS Ayrshire and Arran piloted CitiStat by focusing on delayed discharges from hospital, across the three Community Health Partnership ( CHP) areas and the acute sector. The case study commenced in November 2005 and will continue after the formal end of the Pilot in March 2006; the evaluation process ran alongside the formal CitiStat process, up to March 2006. This case study offers some interesting lessons for the development of CitiStat on a cross-agency or partnership basis.
2.31 The process exposed a significant weakness in the knowledge base, since the data initially available for the process did not allow investigation of causes or measure outcomes. Initially, thinking also developed about the appropriate focus of the process, so that rather than concentrating on the 'hot spot' of delayed discharges, the case study should take a whole system view looking across a range of issues that could contribute to delivering the outcome of better care for the elderly.
2.32 CitiStat created a real sense of achievement by bringing the members of the partnerships together to have high level conversations on delayed discharge for the first time; this was seen as an energising and significant achievement, despite the absence of local authority Chief Executives at Panel sessions and the subsequent decision to refocus the process.
2.33 The case study prioritised the development of a joint and genuinely shared agenda. The process highlighted a number of weaknesses in the partnership arrangements, as well as in performance management which were ultimately to lead to a refocusing of the process. This was useful; it helped to highlight tacit assumptions about accountability arrangements and showed that whilst the four organisations do have mechanisms for joint planning, they do not have mechanisms for joint accountability. To make CitiStat work on a partnership basis would require clarification of the joint accountability of the various organisations for reaching the delayed discharge targets set by SEHD and shared prioritisation of the tasks required across the partnership. The practical outcome of this realisation is that it will take longer to tackle the issue of delayed discharges and that targets may not be met, but there are likely to be benefits in the long term.
2.34 Despite the relatively short term nature of the Pilot process, as in the other case studies, CitiStat has contributed towards improving the quality of data available; this has allowed the identification of a specific 'hot spot' and development of a targeted approach on one particular area, which although less ambitious and far-reaching than the original approach, is probably more realistic given the particular context.
2.35 Important changes were also made to the CitiStat process to 'reframe' it to work better in a partnership context. A number of changes, including reducing the number of observers, were made to make the process feel less artificial and confrontational and to promote dialogue.