CHAPTER SIX: Meaningful Data Capture and Analysis
The need for performance data has been a consistent theme throughout this report. Data, or the performance information it becomes through analysis, underpins each of the constituent parts of an effective performance management system.
In Chapter 3, the importance of priority, indicator and target setting is stressed at all levels of the organisation, based on information covering the full range of policing activity. In Chapter 4, information needs are articulated around the requirements of accountability and empowerment, from individuals, through team to organisation level. Chapter 5 underscores the necessity of data to inform reviews of performance and processes, not only to recognise what might be improved but also to understand what works. Thus it is essential that such information is able to provide a complete picture of performance, and to do so from the local to the national level.
Performance data is also ultimately the source of evidence as to whether or not a performance management system is achieving its aims, that is, driving service quality. In order to answer that question, forces and scrutiny bodies must define what they mean by quality of service and have a clear understanding of what constitutes success in this regard. From this, valid and reliable indicators can be developed that will allow for more robust and focused assessment of progress towards achieving success, however this is defined.
6.1 Data Capture
It is probably fair to say that, across the Scottish police service, the capture of meaningful data is the least well developed aspect of local performance management systems. Nor is the issue always necessarily to do with a lack of basic data. Often the difficulties lie in ready access to the data and the ease with which it can be converted into meaningful information which can be used for performance management purposes.
Ideally, what is needed is real time access to data so that managers at all levels, are able to monitor progress on a regular basis. At the tactical level, managers need information to enable them to identify areas of concern and to make responsive tasking and co-ordinating decisions. Timely access to data is also a prerequisite for effective strategic performance management.
During inspection, HMIC noted that, across all forces, the most well established automated processes for data access, are those dealing with crime and incident data. All forces hold their crime and incident reports on electronic databases. Nonetheless, this does not necessarily enable real time, electronic access for analysis purposes. In the case of one force in particular, the data has had to be extracted manually and converted into a spreadsheet format before then being transferred to a statistical software package for analysis.
Moreover, even where electronic access to data has been automated, the ease with which it is subsequently analysed to produce performance information varies. This appears to depend in part upon whether or not these databases link directly with analytical software. HMIC notes that both Tayside Police and Lothian and Borders Police, for instance, use additional systems that facilitate pre-set queries to be run on crime data. A further crucial factor is how well the underlying software is structured to meet analysis requirements. Fife Constabulary is an example of a force which uses specially designed analysis software to provide the performance information required.
However, dealing with crime is just one facet of policing. As has been discussed already, at the local level most departments and command areas also record information, in some form, relating to other activities. With the implementation of new force contact centres, additional performance information is becoming available in this area too.
The scrutiny demands of the Accounts Commission and of HMIC, through its own annual statistical return, also require data from forces covering wider organisational performance. HMIC is aware of continuing difficulties experienced by some forces in collating this data accurately within established timescales. A contributory factor appears to be limitations in data capture and storage systems that restrict ready access to the data, either within the departments themselves or by the central units charged with compiling the returns.
In summary, performance management requires accurate, timely and meaningful data. If performance management is not to become overly bureaucratic then it must have strong IT support. Managers need to have performance information which informs their decisions allowing them to identify areas of concern and respond to them. The information presented must be concise and not overwhelm them, potentially resulting in useful findings being lost somewhere in an unwieldy statistical report. The performance information must be capable of being understood and interpreted, and managers should be able to obtain the relevant information they need from their own desktops.
Recommendation 9
HMIC recommends that forces and common police services establish systems whereby robust performance data can be captured easily and on a timely basis. Such data should be readily accessible, involving a minimum of bureaucracy in the process.
Clearly this is best achieved with the optimum support of IT. Ideally an overarching database arrangement should exist that links to existing systems and by which the required performance data and information can be extracted. This system should also be capable of producing meaningful contextualised reports direct to relevant managers' desktops.
HMIC notes that some forces, in recognition of the foregoing, have initiated enquiries with the Scottish Executive to secure funding for such a system through the Efficient Government Fund. HMIC considers that there is an imperative for such systems in all forces and the benefits of a single system for the Scottish police service cannot be overstated. Achievement of this aim, taking due account of the co-ordinating influence of the Scottish Police Information Strategy, appears vital to HMIC.
Recommendation 10
HMIC recommends that ACPOS, in liaison with the Scottish Executive, pursue the procurement of a suitable common IT solution to support performance management across all Scottish forces, as a top priority.
6.2 Performance Reporting
Generally, the more automated the data processes, the less time and effort is expended and the quicker the performance management information can be provided for a range of reporting purposes. Responses revealed a certain amount of variation in the purpose and frequency of collated performance reports.
This was true not only between forces but also within forces, where a number of different performance reports were produced. Not surprisingly, the more frequent reports were those reporting exclusively on crime trends. Dumfries & Galloway Constabulary and Tayside Police for example, both provided short, weekly reports showing crime figures down to local level.
All forces and common police services provide what could be termed their main corporate performance report on a monthly and/or quarterly basis, generally linked to the cycle of associated meetings. These tend to focus on performance against crime rates, local objectives, and statutory performance indicators where these do not already form part of force priorities. Because of this, the emphasis tends to be on operational and, in particular, crime-related, performance information. For the same reason, information relating to business support activity tends to be included in force reports only where this relates to force or national priorities.
Some forces, however, have extended the contents further to include additional information. These include graded responses to incidents and call handling statistics by Fife Constabulary, human resources statistics by Tayside Police and Northern Constabulary and local finance and asset management, also by Northern Constabulary.
The incidence of crime, offences, road traffic collisions and so on, are affected by numerous factors, some police-related, others not. Because of this, the presentation of top line statistics can be limited in the extent to which it can convey a more rounded understanding of apparent changes in policing performance. One way of adding meaning to statistical data is to set figures in a broader context. Tayside Police's 'Performance Profile' report, for example, includes brief explanatory comment around performance trends, provided by force intelligence analysts.
Another way is to contextualise data within a longer timeframe. A number of forces report recent trends against baselines derived from longer term average performance. Others are seeking to add more statistical rigour to their trend analysis. Whatever the approach adopted, recipients should be able easily to locate, extract and interpret all information relevant to them. Where too much detail is provided to the target audience, this is arguably worse than no information at all and can affect motivation. Most forces use colour coding in some form in order to highlight progress towards targets or objectives. Most also present data in graphical as well as numerical form.
Central Scotland Police's'Force Performance Report' contains crime control charts. Here, upper and lower control limits ( UCL and LCL) are derived from historical crime trends. Where subsequent crime rates fall outwith these limits or where consistent upwards or downwards trends emerge, these are likely to reflect causal factors rather than random or natural variation at work. In addition, the performance report shows whether or not the force and respective command areas are on target by colour coding for ease of reference. In relation to some targets, variance is shown in numeric terms rather than by percentages, making it easier to see what further action might be required. Examples are shown in the following charts:
Central Scotland Police
Strathclyde Police produces a monthly 'Forecast Bulletin' for selected crimes and crime groups. The report contains charts showing actual and average incidence of these crimes at monthly intervals. Forecasts, based on the last 60 months of crime data, are given for the following four months. These are updated on a monthly basis in order that emerging trends can be identified and so that any change in trends can be taken into account.
Tayside Police has a 'Points of Note' section at the front of its report, which provides a text-based summary of force performance against targets. All figures or text relating to progress towards targets are colour coded. In addition, performance management staff train all inspectors and above in how to access management information from the electronic system.
It was clear from the inspection that forces are generating a number of performance reports designed for different purposes and audiences, and which are presented in various formats. HMIC would not wish to be prescriptive about this, beyond underlining the benefits of regular internal review to identify opportunities to streamline the reports and of maintaining external feedback mechanisms to ensure that they remain fit for purpose.
Tayside Police in its monthly performance profile report retains a tear-out section for recipients to fill in with their comments and suggestions concerning the content and format of the report. In addition, a named contact and telephone number is given as an alternative means of providing feedback.
Tayside Police and Fife Constabulary ensure that Performance Management staff regularly attend Tasking and Co-ordinating Group meetings. This enables staff to gain, at firsthand, an understanding of how the information they produce is used in such fora.
It is vital that performance information be valid and reliable and takes a holistic view of a force's performance. It should be reported regularly and in a way that is meaningful and relevant to users. Having reliable data will ensure that subsequent debate will focus on the management of the information rather than be deflected over arguments surrounding its capture and reliability. HMIC is satisfied that, with improved access to a wider range of information as recommended by HMIC, forces will be well placed to achieve this.
6.3 Data Capture and Performance Reporting at the National Level
At the national level, performance information is desirable in order that the Scottish police service as a whole can gauge its progress towards its priorities and, in so doing, demonstrate to an external audience its ability to deliver a service to the public. As was mentioned in the Preamble, while the Scottish police service is subjected to scrutiny from a number of sources, HMIC has previously commented on the need for an overarching monitoring arrangement.
HMIC has also commented on the advantages of procuring a suitable single performance management IT solution for all Scottish forces. Such a system has the potential to be the 'engine room' for the collation of data at the national level.
The ongoing work in attempting to develop a performance assessment model for ACPOS, linked to its 'Policing Priorities' has previously been noted by HMIC. A report in this regard currently lies for consideration with the new Performance Management Business Area which, inter alia, contains proposals for an ACPOS performance management framework, what a resultant performance report might look like and what infrastructure would be required to support such a structure.
It appears to HMIC that this is a particularly pertinent and important area of work which links closely with a number of recommendations from this report. An additional benefit of collating information in this manner would be that HMIC and other relevant stakeholders would have access to the same information, thereby streamlining the current statistical requirement from forces. HMIC will monitor progress in this area closely.
In the meantime, forces remain subject to demands for regular statistical information from various bodies. For example, the Scottish Executive, Audit Scotland and HMIC all require some form of statistical return from forces, at least annually and sometimes quarterly. In all cases the same information is requested of each force. Some of the time the same information will be requested by the different bodies. Even so, inconsistencies both within and between individual force data sets are not uncommon. While long recognised, these matters remain of some concern to HMIC.
All forces have in place some form of auditing procedures through which system data can be checked. The more robust these mechanisms are the greater likelihood that decisions can be made on the basis of accurate information. Forces are also subject to varying degrees of external audit, primarily of crime and incident recording procedures, by the various scrutiny bodies:
- as part of its primary inspections, HMIC regularly conducts a crime/incident audit in each force
- a thematic inspection by HMIC is currently underway, examining force compliance with the Scottish Crime Recording Standard
- as part of its statutory responsibility, Audit Scotland carries out an annual audit of performance indicator arrangements in each of the forces.
In addition, on a more ad hoc basis, staff within the Scottish Executive Justice Statistics Department also perform short audits of crime statistics within the forces.
The emphasis of these audits is about presenting an accurate picture of policing performance to the public. Over the last three years, the Accounts Commission has highlighted unsatisfactory arrangements for producing certain statutorily required information in several forces. HMIC recognises and values the Accounts Commission's crucial role in driving improvement within the Scottish police service. HMIC also shares its concern where shortcomings are found in force procedures for producing the required information. The use of the term 'unreliable' in this respect and its possible implications has been raised with the Accounts Commission by ACPOS.
6.4 Comparison
In the absence of an overarching system to monitor the overall performance of the Scottish police service, valid and useful comparison between forces is difficult. One force produces a quarterly report which aims to show forces' individual and combined progress against national targets however local difficulties at the data capture stage mean that not all forces are able to provide this data on such a regular basis. Another force attempts to utilise comparative data from England and Wales. In this case incomparability of data between systems is unavoidable and the resulting analysis, therefore, can at times be incomplete or flawed. HMIC recognises the need which drives these initiatives but has reservations over their current effectiveness.
Other methods of collating information in such a way as to facilitate comparison do exist. iQuanta is a web based tool that has been developed to facilitate centralised data capture, dissemination and analysis across all 43 forces in England and Wales. Forces are required to provide the data, relating primarily to statutory indicators and measures but also including recorded crimes and offences, on a monthly basis. Data is available at force, basic command unit and crime and disorder reduction partnership level. In this way the system provides users with shared access to a unified set of policing data that is consistent.
There is no doubt that a system such as iQuanta can be a powerful facilitator for the provision of regular, robust and consistent performance data. By drawing together disaggregated data from the constituent forces in this way, a more reliable picture of national performance can emerge.
Where there is, perhaps, a need for caution, it is in how data is subsequently analysed and the resulting performance information reported. It is apparent that collating data in this way facilitates comparison between the constituent forces. Nonetheless HMIC recognises that league tables can be drawn from information currently placed in the public domain by forces and other related bodies. Indeed, there have been occasions where the media has constructed national league tables, particularly around recorded crime levels. To compare at such a simplistic level does little to help identify genuine areas for improvement.
The need to avoid simplistic league tables also finds support in a recent report 23 which examined the introduction of a Performance Assessment Framework ( PAF) for NHSScotland. The following quotes are of interest:
"Support for the Scottish PAF per se was strengthened by the absence of league tables and its importance as a stimulus for discussion at the Accountability Reviews"
"In order to retain the high commitment of NHSScotland, it is important to avoid the pitfalls of other similar systems, such as league tables and strong financial incentives".
In assessing performance data, particularly at a national level, the wider context should be considered, including demographic, geographic, economic and other influencing factors. Otherwise the potential for drawing misleading and unhelpful conclusions is very real.
To an extent these factors have been taken into account in the Most Similar Families ( MSF) groupings developed in England and Wales and used within iQuanta. The MSF concept is intended to offer a more rigorous approach to comparisons of forces, basic command units and crime & disorder partnerships. It applies a statistical technique to derive families that are different but whose internal members are as similar as possible, along demographic, socio-economic and geographical lines. To be more meaningful, comparisons would then be restricted to those between members of the same family.
HMIC acknowledges the value to be derived from comparing or benchmarking performance and is supportive of any drive towards establishing more valid processes for doing so. However a number of limitations with the MSFs remain. In terms of the concept itself, even those who constructed the revised families, acknowledge that there is no definitive and unchallengeable 24 method of deriving such groupings to enable like-for-like comparisons. In addition, the interpretation of findings associated with MSF are limited, particularly where families represent looser groupings of forces. Furthermore, the model in England and Wales is based on a crime reduction focus and therefore the constitution of each family represents factors particularly relevant to that issue, suggesting that different families might be needed for other matters.
Practically, with only eight, somewhat diverse, forces in Scotland the most similar families concept could only realistically be applied at a lower level. There are 36 force command areas of considerable diversity across the Scottish forces, covering 32 local authority areas. Sitting within each of the 32 local authority areas is a community planning partnership. There is nothing definitive in the statistical literature to suggest that this is too small a number to which to apply clustering but it is clear that the diverse map of Scottish forces and community planning partnerships offers little to suggest significant value can be derived from clustering these basic units into families.
Nonetheless, HMIC acknowledges that there may be a desire for comparison between local authority areas in respect of community planning based on the much wider range of information available at that level. If so, there may be alternative and potentially more meaningful ways of achieving comparison.
For example, the inspection team is aware of independent, academic research examining the MSFs, being conducted by staff at University College London. 25 As well as producing an evaluation of existing Home Office families, the research is also examining alternative models that are based on more localised geodemographic (geographical and demographic) area profiles. These profiles are used to describe socio-economic and socio-cultural behaviour and work by clustering smaller geographical areas on the basis of social similarity such as lifestyle types, rather than, more arbitrarily constructed administrative locations such as force areas.
Several such classification systems are already well established in the UK. 26 Though widely used by business, their application within the public sector to date remains rare. Some work in this field has however been conducted with several forces in England, Wales and Northern Ireland. 27 The main thrust of this has been to enhance understanding of small area crime and policing environments in order to aid resource management and deployment. The research underway may additionally show that, where force or other organisational level comparisons are desirable, clustering and comparisons between areas, on the basis of geodemographic classifications, could prove to be a useful and complementary informative strategy to that of traditional geographic comparisons.