Scottish Government

Section One: Introduction

1.1 This evaluation was commissioned by the Scottish Executive, Office of the Permanent Secretary, Performance and Innovation Unit ( PIU) and Scottish Executive Office of the Chief Researcher. The Performance and Innovation Unit is responsible for the Pilot of the CitiStat model of performance management in four organisations within the Scottish public sector. The organisations selected to undertake the Pilot in Scotland were City of Edinburgh Council, Aberdeen City Council, NHS Tayside Health Board and NHS Ayrshire and Arran Health Board.

1.2 The CitiStat Pilot commenced in three areas in September 2005 and completed in February 2006. The NHS Ayrshire and Arran case study ran from November 2005 to March 2006. The evaluation process has operated to the same timetable in each case study.

Background to CitiStat

1.3 CitiStat is a performance management system which originated in Baltimore, Maryland, USA which uses recent operational data on key performance indicators as the basis for discussions between the Leader of an organisation and service directors. CitiStat meetings are intensive and formal; service directors are asked to account for their Department's performance and offered support to improve performance where necessary.

1.4 In Baltimore, CitiStat is used to:

• improve the quality and reliability of services
• raise performance by establishing communication between the office of the Mayor and key officials
• provide a forum for immediate operational direction from the Mayor
• resolve cross-cutting issues affecting performance
• foster a culture of accountability throughout the organisation.

1.5 An independent review of CitiStat in Baltimore ¹ found that the programme had contributed to:

• increased customer friendliness
• integration of accountability of resources from federal, state and local government
• improved ability of departments to share performance information and policy data quickly and accurately
• a higher level of agency performance in delivering critical goods and services.

1.6 The review also identified a number of key issues.

• The extent to which the potential myopia of a bi-weekly accountability system can obstruct longer range strategic planning.
• The need to avoid inadvertent micro-management of agency operations.
• The need to link data to the strategic vision and initiatives of the Mayor and the City.
• The challenge to CitiStat of the need to work with city agencies to orchestrate their activities with policy and administrative goals and objectives.

Preparation for the CitiStat Pilot in Scotland

1.7 A team from PIU visited Baltimore in October 2004 and March 2005 in preparation for establishing a CitiStat Pilot in the public sector in Scotland. The team reported that a single CitiStat model was not necessary for a successful programme and that CitiStat in Scotland should be shaped by decisions about the 'theatre' of the process itself, openness of information, ownership of data, scrutiny and accountability and the roles and responsibilities within Departments and the Chief Executive's office.

1.8 A number of key issues were identified which influenced the way the Pilot was developed in Scotland:

• The distinctiveness of the Scottish context: this includes the level of autonomy for the city of Baltimore, and its relationships with other parts of government, as well as the strong political leadership.
• The dynamics of the CitiStat sessions: cultural differences with the USA may cause some discomfort with the 'confrontational' nature of the CitiStat sessions, including the layout of the room and style of questioning which was similar to giving evidence in a court room.
• Existing performance management and scrutiny processes: in Baltimore, there was no independent scrutiny process. The efficiency agenda, the concept of Best Value, and the need to account for the use of public money through a paper trail were not evident in Baltimore.
• Engagement with service users: there was no engagement with service users in Baltimore and no consideration had been given to this issue. Sessions were held in private and there was little focus on outcomes for service users.
• Timescales: the Baltimore focus was on short term tactical change rather than long term improvement. Solutions were expected in very short time scales compared to those targeted in the Scottish public sector.
• Technical and operational involvement: staff at all levels of the organisation in Baltimore appeared to feel comfortable in contributing to the CitiStat process. The involvement of frontline managers and supervisors appeared to encourage the process and embed responsibility for compliance throughout the organisation.
• Analysis and use of data to target service improvement: the CitiStat team in Baltimore that undertake the analysis of the data are not necessarily experts in the service area. The focus on Baltimore is on continuous improvement rather than a direction of travel towards a specific target.

1.9 Based on these observations it was felt that the CitiStat model could be used in the four case study organisations to drive up performance, but also to test out how elements of the process would work in the Scottish context.

1.10 A further delegation from the Scottish Executive and from each of the four case study organisations visited Baltimore in October 2005 to allow for further reflection on how the Baltimore experience might shape CitiStat in Scotland and the specific way in which it might be used by the different case studies.

1.11 Subsequent meetings of the delegation and their contribution to this evaluation process endorsed the earlier conclusions of the Scottish Executive and highlighted a number of important contextual factors. These included the need for clear strategic drivers of the process. It was identified that CitiStat should fit within the culture of continuous improvement, rather than having a short term focus on the detail of the 'issues of the day' without any real prioritisation of these in terms of their significance. It was felt that the process could be adapted to tackle deep rooted organisational and operational issues, as long as it was embedded within the organisation, rather than being a stand-alone project or the 'big idea' of one high profile individual.

1.12 Whilst observers of the Baltimore process recognised the value of the role of the Mayor, in Scotland there is no single direct equivalent role and executive and non-executive functions are divided between appointed managers and board members/elected members respectively. In addition, many were not keen to replicate the more 'confrontational' approach used in Baltimore; indeed whilst the Scottish Executive did wish to see a more focused, direct or 'tougher' approach taken, there was no requirement that a 'confrontational' style should be adopted.

1.13 There were also concerns about governance and accountability issues. A number of people raised questions about management structures within the NHS and local authorities and whether these lend themselves to the CitiStat model, including what structures would be appropriate for membership of the Panel and Podium (see Figure 1). The appropriative role of non-executive NHS board members and elected members of local authorities and the role and value of observers were also concerns. The potential impact of the approach on the relationship between NHS Boards and the Scottish Executive Health Department ( SEHD) was also raised.

1.14 The model presented a number of challenges about the availability and use of data. It raised questions about whether information systems in the public sector enable monitoring of key indicators. In order to provide a quick response to the use of data for performance management, systems would need to adapt to provide data (which may be unvalidated); however, it was also felt important to be outcome focused rather than input driven and to look at trends over time, rather than snap shot data.

1.15 The process in Baltimore was clearly very resource intensive. The need for dedicated staff with the appropriate analytical skills was identified. It was also noted to be essential to recognise existing performance management and accountability processes and minimise duplication of effort.

1.16 Figure 1 shows the proposed model for the CitiStat Pilot in Scotland. This illustrates the links between the analysis of agreed performance indicators, the CitiStat brief to the Panel, the Panel process itself in which service Directors account for their performance and agree further actions, the evaluation process and the follow up brief which feeds into the next cycle.

Figure 1: The CitiStat Model in Scotland

Selection and structure of the Pilot areas

1.17 The two local authority Pilot areas initially focused on environmental quality of life. This issue was chosen by the Scottish Executive because it was felt to be a measurable activity, important both to the public and to policy makers. This is a challenging issue and is also an area where the responsibilities for the delivery of services in these areas often cut across a number of different departments and agencies.

1.18 The Pilot areas were selected in consultation with the authorities themselves and the Convention of Scottish Local Authorities ( COSLA). The Pilot areas were selected to reflect varied Council structures; in Edinburgh the Council has an executive structure in which the full Council has devolved responsibility for key areas of service delivery, including environmental issues, to a formal executive of elected members. In Aberdeen, a neighbourhood management structure is in place and three Neighbourhoods Directors are responsible for these issues within their designated Neighbourhoods.

1.19 Initially Orkney Islands Council was also included in the Pilot although did not proceed as it was felt that the preconditions for a successful Pilot were not present. These preconditions included: involvement in the selection of appropriate subject matter for the Pilot; the need for a long lead in time to ensure political and managerial buy-in; addressing concerns about a potentially adversarial approach; the absence of a high volume of data and the need for suitable IT systems to support data quality and analysis.

1.20 The two Health Boards were selected by the SEHD. The Tayside case study addressed the delivery of access and treatment services. NHS Ayrshire and Arran examined delayed discharge from hospital.. Access and treatment and delayed discharge are both part of wider SEHD targets in relation to Health Improvement, Efficiency and Governance and Access and Treatment (known as HEAT targets) included in Health Boards Local Delivery Plans from April 2006.

1.21 All case study organisations were supported by a CitiStat Manager from the Scottish Executive PIU available to work with the organisational management team in each area.

Aims and objectives of this evaluation

1.22 The general aims of this evaluation were to:

• Determine whether and how CitiStat can be effectively used in the Scottish public sector to improve quality and productivity in the delivery of public services.
• Determine whether CitiStat can provide a performance management model that is both robust and can be used for continuous improvement in local service delivery.
• To assess whether a focus on operational outputs can contribute to improvements in higher level strategic outcomes.

1.23 The specific objectives of the study were:

• To illustrate how CitiStat data are generated, displayed and interpreted across the organisations to accomplish better service delivery and employee accountability.
• To assess the extent to which the programme has affected managerial decisions and operational behaviour (intended and unintended).
• To demonstrate the effectiveness of CitiStat in allowing each case study organisation to meet their own success factors.
• To track and compare perceptions of earlier performance management models and early perceptions of CitiStat.
• To examine external, contextual factors which both enable and constrain the potential value of CitiStat in case study areas.
• To examine and learn from the implications of unanticipated outcomes.
• To make recommendations for improvements in procedures for wider CitiStat implementation across the Scottish public sector.

Evaluation methods

1.24 The focus of the evaluation has been to explore the experience of the case studies themselves in using the CitiStat model and to develop an understanding of the opportunities and constraints that exist within different public sector contexts in the implementation of CitiStat. This has taken place within the context of the use of other performance management tools by local authorities and the NHS in Scotland. The evaluation does assess any improvements in service delivery, although given the short duration of the Pilot these were not anticipated to be substantial.

1.25 Evaluation is rarely used to directly and actively inform service delivery. Many organisations are 'data rich, yet knowledge poor'; the information available is not used to full effect for performance improvement. A mechanistic approach to meeting targets which are often imposed, rather than reflecting local needs and aspirations, can hinder the effectiveness of performance management in improving the effectiveness of public service delivery. This evaluation addresses these issues through using an action research approach. ² This was felt to be particularly appropriate for a number of reasons;

• Action research would engage the case study teams and Scottish Executive CitiStat managers in the evaluation process generating useful, 'live', practical findings as it proceeds, as well as providing a summative evaluation on completion of the Pilot.
• Action research and CitiStat work on similar cycles of action and reflection and the two processes were felt to be complementary.
• Findings would be fed back into the on-going CitiStat process in each area and would provide tested evidence of good practice to inform the development of wider guidance on CitiStat.
• Given the short timescale of the Pilot phase, the flexibility of action research could provide a timely and adaptive feedback mechanism between evidence and practice.
• The two processes both require and cultivate an organisational culture conducive to learning from evidence. By involving participants of the CitiStat process in the evaluation, the approach promotes the skills of critical reflection on practice and potentially embeds and sustains an evaluative culture within the initiative after the formal evaluation has ended.

1.26 Each of the Pilot areas forms a separate case study. The detail of the approach to evaluation was decided in collaboration with each case study area. It is central to an action research approach that each CitiStat case study should decide its own indicators of success; this means the way in which data for the evaluation was collected depended on local decisions and allowed the evaluation to get to the heart of how the process worked in each particular place and provide a different and more positive kind of evaluation process.

1.27 All the case studies held evaluation sessions after CitiStat sessions where participants were encouraged to reflect on the sessions and performance to date. These used a number of different approaches including;

• visual inquiry methods
• interviews with key Panel members
• questionnaires to Podium members and observers (Edinburgh)
• maintaining personal diaries (Aberdeen)
• observations of the independent evaluators and co-facilitators.

1.28 A key element of the evaluation process was to bring the four case studies together to encourage discussion of the diversity and commonalities of experience and of key learning. A CitiStat Exchange Inquiry event was held in January 2006 and was a useful forum to share experience across the case studies and decide on the key lessons, both locally and nationally. This event used a variety of approaches including the visual presentation of the experience of each case study, as shown in Figure 1.1. The findings from this event, from the experience of Orkney Islands Council and from the experience of those who visited Baltimore have been integrated into the wider findings of this report. Fuller details of the methodology are included in Annex 5.

Figure 1.1 Visual presentations at the CitiStat Exchange Inquiry event

1.29 All quotations used in this report are unattributed to maintain anonymity.

Structure of this report

1.30 Section 2 provides a short summary of the initial successes to date in each of the case studies against the local success factors and summarises the key features in a table. Sections 3 to 5 identify a number of common themes and distinctions that are worth highlighting based on evidence from the case studies and from discussions at the CitiStat Exchange Inquiry event held in January 2006. Section 3 looks in detail at how the case studies have used data to improve performance. The CitiStat Panel process is the focus of section 4. Opportunities for and barriers to action and change are considered in section 5. Finally, section 6 considers the future for the CitiStat process in Scotland. Details of each of the case studies are reported separately in Annexes 1-4. Annex 5 provides fuller details of the methodologies used.