Scottish Government

5. Draft Monitoring and Evaluation Framework

Introduction

5.1 The brief required the researchers to:

consider and identify the information required for evaluation of the long term impact of the VDLF, including economic, environmental and quality of life outcomes.

5.2 We would suggest a framework which recognises both the short term outputs of the programme as well as the longer term outcomes referred to in the programme objectives. These longer term outcomes may not be achieved immediately. This is because in some cases, the site has been prepared for development, or exploratory studies have been carried out to support its development, and it is only when that development takes place that these outcomes (and in some cases the full outputs) will be known.

5.3 One way of dealing with this is to have a current actual and an anticipated monitoring pro- forma. This will deal reasonably well with the interim and provisional condition of many cases.

5.4 However, many of the more subtle area regeneration outcomes are unlikely to be dealt with effectively within the limitations of an annual monitoring form. While some account of these may be give in annual reports, the periodic evaluations of the Fund will be the best way of giving more extended consideration of the impact of the supported schemes on their immediate and wider locality.

Monitoring Framework

5.5 Our proposed framework is set out in Table 5.1 below. In the outputs column, the Table sets out the basic facts about the VDLF funding has purchased. The outcomes column attempts to set out the wider impacts or benefits that the activity funded by VDLF has achieved. The key issues are:

• Impact - if the site previously looked unsightly, the impact is likely to be much greater
• Visibility - sites which are located close to transport arteries or are in local centres or overlooked by large numbers of houses will have more impact than sites in out of the way fringe urban or rural locations
• Further investment stimulated - resolving derelict land problems often has a key role to play in increasing confidence in an area and encouraging further development. Although inevitably a subjective assessment and one which may not be capable of being made until some time later as part of a periodic evaluation, this can be an important benefit because of the externalities that a successful development project on a vacant or derelict site can create.
• Jobs - it has been traditional in the past to record the amount of employment created on such sites. We see this as perhaps one of the less useful parts of the monitoring and evaluation framework, mainly because displacement is often very high at a local authority level. However, if the site is in area of deprivation, local employment may be important.

5.6 While some of these measures require a degree of subjectivity, we feel that most could be recorded by project staff, using both a current actual pro forma and an anticipated pro forma for sites where the final land use or land uses are not yet fully in place.

5.7 At a minimum, the outputs should be recorded annually by partnerships. The outcome information may be felt to lend itself more to inclusion in periodic evaluations. Table 5.1: Draft Monitoring and Evaluation Framework

Table 5.1: Draft Monitoring and Evaluation Framework

Notes:

'Greened' space is space which has been grassed over as a temporary measure. Long term grassed space is treated as 'park/ landscaped'.

'Park/ landscaped' is space which is intended for permanent green use and includes tree and shrub planting, parks and play spaces.

'Sport/recreation' would include playing fields, golf courses, etc.

'Business' includes all land and buildings for employment use - it is difficult to separate industrial and office space because of Class 4.

Number of households benefiting from improvement: record all households with a direct view of the site.

Road /rail routes passing site: record the road classification number.

Numbers of passers by: traffic numbers on trunk roads and rail passengers can be obtained from the Scottish Transport Statistics: http://www.scottishexecutive.gov.uk/Publications/2006/12/15135954/205. For local access roads within the urban area, the local authority's transport department may have carried out counts.

Further investment stimulated: The form should simply record 'yes' or 'no'. Where there are catalytic effects, these should be recorded in a separate narrative account. The case worker should detail subsequent development in the immediate locality which appears to have been encouraged or supported by the VDLF project. The names and types of the developments for which VDLF appears to have had some catalytic effect should be recorded, and, where appropriate, the number of units. The strength of causal link should also be recorded - whether it can be stated that it would not have happened without the VDLF scheme, that it was unlikely without the VDLF scheme, that it was of a higher quality or did not need gap funding/grant aid - as a result of the VDLF supported schemes. In some cases, it may be more the case that the project was one a series of VDLF projects whose effect was mutually reinforcing. Many of these assessments will involve a degree of subjectivity, but the District Valuer, who collects evidence about land and property value trends may be able to provide an informed third party view. When periodic evaluations are being carried out, it would be appropriate to carry out some interviews with the developers of apparently linked projects.

Jobs created: The number of jobs located on the site should be recorded in the column. Only net new employment in the locality should be recorded. If there has been displacement at a local level - if these jobs had been relocated from the same area of the city or from elsewhere in the town - there may be no net gain. The key test is whether the relocation generates an increased number of jobs in a deprived area.

Indirect and induced employment: In this column we would be seeking to assess the indirect and induced effects at city or local authority level of any new employment attracted to the area - i.e. the further multiplier effects as a result of local buying patterns of the business or through the spend of the wage bill. (Local multiplier effects within the same area of a city are generally very small, unless the development is very large indeed, and can generally be disregarded) There will only be new indirect or induced employment at a city or local authority level if the employer is new to the city.

Periodic Evaluation

5.8 In addition to ongoing monitoring, there is also a case for periodic longer term evaluation, which would capture more fully the 'softer' outcomes such as the impacts on local residents, the local property market and business confidence, and to generate an overall evaluation of the impact on regeneration areas. We would suggest that periodic evaluation of all programmes should be commissioned by the Scottish Executive at three yearly intervals. We would recommend a single centrally commissioned evaluation, as it would facilitate a standard approach across all partnerships, make possible useful cross-partnership comparisons, and is likely to provide a more independent evaluation.

5.9 The periodic evaluations would require interviews with local residents to explore the impact on satisfaction with their neighbourhood, and for the impact on the property market and business confidence, a series of interviews with developers/ house builders, local property agents and the district valuer.

5.10 While we have suggested that there should be some assessment of catalytic effects in the annual monitoring forms, the number of considerations involved and the desirability of an independent view makes this an area which can only be dealt with fully and comprehensively in a periodic evaluation.

5.11 Similarly, the issue of the number of net, non displaced jobs, and the benefits of the employment created to the deprived localities in which they are created is quite a complex assessment and may be only capable of being dealt with fully in a periodic evaluation. The same is true of the overall image benefits to the city or local authority area. The periodic evaluations would also assess, ex post, the degree of additionality achieved by a sample of the projects.

Baseline

5.12 The issue of a baseline for the evaluation has also been raised by the Executive. As each of the partnerships has, to some extent, set out to achieve different things with the funding, there will be a need for individual baseline positions relating to the individual objectives prioritised - for example in Dundee, there may be a need for a district baseline for Stobswell, or any other districts strongly prioritised.

5.13 For all partnerships supported, however, the key baseline must be the stock of long term vacant and derelict land at the time that the funding was granted - i.e. at that date, the stock of sites on the VDLF register which were vacant or derelict for more than 10 or more than five years. These are the sites where the likelihood of market failure and of additionality is greatest.

5.14 The impact on sites which have been vacant for more than 10 years will be particularly relevant, and to a lesser extent, sites vacant for more than five years. In our view, this is the key baseline against which the effectiveness of the VDLF funded activities should be assessed.

5.15 The overall change in the amount of vacant and derelict land in the local authority area is a less satisfactory measure of the effectiveness of the use of the fund because overall change will be affected by 'new' vacancy and dereliction - i.e. land and buildings falling out of use - which likely to be outwith the control of the partnership. The change in the overall level of vacancy and dereliction is however also of interest, as it helps to assess the continuing need for funding.

5.16 It would also be helpful from a longer term evaluation perspective to create a photographic baseline of all vacant and derelict sites - and certainly those vacant for five or ten years and any other sites targeted for action, showing their impact in their setting.