Scottish Government

Main Findings

The recommendations exceed the original brief in so far as they deal with issues of process and procedures as well as monitoring and evaluation. Monitoring and evaluation options were chosen to have minimum resource implications and, as far as possible, to make use of existing information systems.

The key process recommendations are:

• The Compact and associated Good Practice Guides should be revised and updated. The revised versions need to be symmetrical, identifying the commitments of both government and the sector;
• An Annual Compact Implementation Action Plan should be drawn up;
• There should be an Annual Meeting which brings together the sector, senior politicians and civil servants and senior management from NDPBs and Agencies. This would discuss progress in implementing the Action Plan and associated issues. The outcome of the Meeting would feed into the next year's Plan; and
• Within each government organisation a Compact champion should be identified who will have responsibility for ensuring that the Compact is implemented.

The key monitoring recommendations are:

• A number of direct and indirect monitoring indicators should be derived from the Executive's information sources and additional information should be collected that relates specifically to the content of the Good Practice Guides;
• The key monitoring mechanism should be an Annual Monitoring Survey that would be distributed and analysed electronically. As far as possible this would ask identical questions of the sector and government. The survey of the sector should be based upon a sample derived following consultation between the Executive and SCVO. The government survey should be a census;
• The survey should be augmented by 3 Focus Groups that would be convened every 2 years and would cover different types of geographical area.
• Collating a baseline for the Compact is difficult, given the length of time that has passed since it was first introduced. Accordingly, the recommendation is that: The baseline should be based upon the responses to the first electronic survey, supplemented by information derived from the proposed indicators and contextual information collated from other sources.

The proposed approach to monitoring blends summative (outcome) and formative (ongoing) evaluation. Given this, it is felt that formulating a separate evaluation framework was of limited importance. Accordingly, it is recommended that:

• Evaluation should be based upon surveys and interviews which analyse the processes within a sample of voluntary organisations, government departments, agencies and NDPBs. These would be augmented by case studies.

Background

The Baseline Review was commissioned to develop recommendations for setting up a monitoring and evaluation framework for the Scottish Compact. This Research Findings summarises the main research report. The Compact sets out a framework of principles covering the working relationships between central government, its agencies and Non-Departmental Public Bodies (NDPBs) in Scotland and the voluntary and community sectors. These principles are outlined in the formal Compact documents, including the Scottish Compact and a series of Good Practice Guides. They aim to develop better mutual understanding and confidence between the Scottish Executive, its Agencies and NDPBs and the voluntary sector. As such, the Compact development process is about relationship building and its maintenance.

Comparative Analysis

The research includes comparative analysis which looked at the other national compacts in the United Kingdom and the Canadian Compact. From this it proved possible to formulate a generalised model of compact development. The start of the process is invariably discussion between the sector and government out of which comes some commitment to set up a compact. The formal compact is then produced. If this is not produced jointly, by the sector and government, then there is, at least, consultation between the 2 parties. The compact is then implemented through the production of good practice guides, the commonest of which invariably deals with finance.

All of the compacts analysed followed this model. Thereafter practices differ. Some, for example the English Compact, produce an annual action plan which details what needs to be done to implement the compact over the next year. This is usually linked to some form of review process, in effect monitoring. Some compacts then link the review into an annual meeting. This often brings together senior politicians (government ministers in the case of the English Compact) and senior representatives of the sector. Progress is discussed, a report of the meeting published and the next year's action plan formulated. The cycle of implementation then continues.

All compacts recognise the importance of monitoring and evaluation. Where monitoring has been undertaken, for example in Scotland and England, this has involved surveys of the sector and government. The extent to which the outputs of monitoring (or review as it is often called) feed into the compact implementation process varies. All compacts have a commitment to evaluation. None have yet undertaken any, nor do any seem to have firm ideas about either the process or the content of such an evaluation.

Areas for Monitoring and Evaluation

A variety of information was synthesised to identify the main areas that monitoring and evaluation should focus upon. Eight main areas were identified:

1. Awareness of the compact, which was seen as being the start of any relationship building process;

2. Dissemination - the extent to which the knowledge and awareness of the compact permeates through an organisation, especially to the front-line staff;

3. Usage was seen as being a critical measure, as it was likely that there would be limited impact, regardless of the extent to which there was awareness of the compact, if the parties to it made limited use of it;

4. Usefulness needs to be considered - the extent to which the partners find the compact of use in their dealings with one another;

5. The good practice guides, given that they are often the most tangible and detailed aspect of the compact, need to be considered in terms of the extent to which they have been used and the impact that they have had;

6. Given that the compacts are about relationship building, mutual understanding needs to be considered, that is the extent to which the partners have developed a better understanding and awareness of one another;

7. Confidence, the day-to-day functioning of the relationship, needs to be considered; and finally

8. Trust, which is the ultimate goal of the compact, needs to be assessed, with the development of trust being the culmination of confidence.

It is felt that the above areas cover all of the various ways in which the compact can be defined, that is: as state of mind; the formal document; the good practice guides; and, implicitly, through the supporting documentation.

Monitoring and Evaluation Information

A variety of information sources were evaluated to see if they could be used to provide information for monitoring and evaluation of the Scottish Compact. The majority of the sources had been set up for non-compact specific purposes. As such it was felt that they were of some, although limited, use in an unmodified form.

Several indicators were then identified that could be either drawn directly, or derived, from 2 main Scottish Executive data sources: the Direct Funding database (which contains details of voluntary organisations directly funded by the Executive) and the soon to be set up Consultation Registration and Evaluation System (which will log information about forthcoming consultations and then summarise the consultation process and its impact).

A total of 34 indicators were identified and each was linked to the key areas for monitoring and evaluation. For some of the indicators to be operationalised additional information needs to be collected whilst others require modifications to the data collection forms. The suggested changes are detailed in the full report.

Developing a Sampling Framework

Defining the partners to the Compact is easily done at a strategic level. However once one begins to unpick the "voluntary and community sectors", and "government", the situation becomes more complex.

Given that the voluntary sector is so large, it being estimated that there are in excess of 50,000 individual organisations, monitoring and evaluation needs to be based on a sample of the sector. To identify a voluntary sector sampling framework a variety of networks were analysed, and the advantages and disadvantages of each outlined. Although the Compact explicitly accepts that not all voluntary organisations will want partnership with government, the view was taken that the entire sector is affected by government. The intensity of this impact will clearly vary and some parts of the sector will be far more enthusiastic about developing close links with government than will others. However it was felt that the Compact is relevant to all voluntary and community organisations. If this is accepted, then the crucial factor for any sampling framework is that it is representative of the sector and does not give undue weight to various sub-groups, for example, national organisations or ones funded by the Executive. It was felt that the most cost-effective way of developing a sample framework was for the Executive to work with SCVO in order to make use of one of its data collection networks. The suggestion is that this be the Workforce Panel which is currently being set up.

The government parties to the Compact are more easily identified. However it can be argued that it would be difficult to identify individuals within large organisations who are aware of the Compact and are involved in its implementation. Whilst this may be the case, it was felt that this could reflect failures of internal management and an unwillingness to take on board the commitments contained within the Compact. Accordingly it was recommended that monitoring and evaluation should include all parts of government covered by the Compact, rather than being based on a sample.

About this Study

The Baseline Review was commissioned from GEN Consulting to develop recommendations for setting up a monitoring and evaluation framework for the Scottish Compact. The Compact was launched in 1998 and sets out a framework of principles covering the working relationships between central government, its agencies and Non-Departmental Public Bodies (NDPBs) in Scotland and the voluntary and community sectors. These principles are outlined in the formal Compact documents, including the Scottish Compact and a series of Good Practice Guides.

Research Methods

The Review involved a number of elements. These included: a literature review; a comparative analysis of other compacts; an appraisal of the various datasets held by the Executive and others; an assessment of the extent to which the datasets could be used for monitoring and evaluation purposes; and the formulation of a series of monitoring and evaluation recommendations. Fuller details of the Recommendations and the reasoning behind them are given in the complete report.