6 Assessment methodology: limitations
This section discusses some of the limitations inherent in the approach adopted for the evaluation, explains how they have arisen and makes recommendations as to how they could be addressed in the future.
6.1 Summary of the approach adopted
The methodology used for this interim assessment of the impact of CEEF both now and in the future is discussed in detail in Appendix 3. This has followed, where appropriate, Defra's guidance: Greenhouse Gas Policy Evaluation and Appraisal in Government Departments, April 2006, which is in turn based on the Treasury Green Book.
Data quality
Project submissions to CEEF are carried out electronically through a web-based system which is available to all local authority applicants. Applicants are required to enter project details including costs and estimated energy savings (and hence financial savings). Projects which meet the eligibility criteria are then automatically approved. The pre-defined criteria within the web-based systems ensures that consistent data is collected for each of the individual projects.
An external consultant independently checks the quality of the data within the web-based system. Regular reviews are carried out on the progress of CEEF and the projects submitted. This enables progress to be tracked and data quality issues to be identified and corrected.
6.2 Data limitations
The main limitations of the approach adopted concern:
6.2.1 Uncertainty over the project start date
This interim assessment has based its understanding of the impact of CEEF on the information currently held on project submissions, including that on predicted energy savings. This requires an assessment of when the project started and how long it ran for.
The applicants are asked to fill in three dates on the CEEF website:
Project committed date - when the local authority commits to implementing the project
Project spent date - when the funding has been spent
Project implemented date - when the project has been implemented and savings should begin.
An assessment of how these dates were entered by applicants was made and the results shown in Table 5. The dates entered by the applicants have not been used consistently in completed application forms. A number of submitted application forms had the same value for all three dates - others did not have dates for implementation.
Table 4 - Analysis of Dates Entered on Applications
Status of dates | Number of Projects |
|---|
All three dates the same | 57 |
|---|
Implementation date left blank | 242 |
|---|
Implementation date entered correctly | 238 |
|---|
Total | 537 |
|---|
There is therefore uncertainty from the information available from the CEEF website whether successfully submitted projects:
- Are fully implemented
- Have been delayed or cancelled due to unforeseen circumstances.
This means that there is uncertainty about the lifetime of the savings arising, and in some cases whether the project has actually taken place.
How addressed in this evaluation
The accuracy of dates was checked for the six projects investigated in detail. Three of the projects had been successfully implemented and three remained incomplete. However the committed, spent and implemented dates had only been input for one of the projects. The three dates had not been input for any of the other projects including two completed projects. In regard to the incomplete projects, discussion with the local authorities suggested that these projects would go ahead, so as a minimum it would be expected that the committed dates for these projects could be input.
In addition to the dates above, the CEEF website automatically generates a project submission date, when an application is submitted. This is the date that the Scottish Executive indicates can be used as an indication that the project has, or is about to be implemented. In view of this, the "project submission date" is the only date that is provided in a consistent manner and it has therefore been assumed that energy savings start at this point and persist for the average lifetimes (listed by technology) given from the CEEF website. The data suggests that this is the most reliable approach to take, but it is likely to lead to overestimate savings.
Recommendations for the future
It is suggested that additional guidance is supplied so that the applicants provide information on when projects have been implemented in a consistent manner. At present this is not the case, as is shown in Table 4 above. It is also suggested that annual reports are made by the key local authority contact to provide an ongoing picture of savings achieved.
6.2.2 Predicted savings
This interim assessment is based on predicted savings, rather than measured data or estimates made after the project has been carried out. In a number of cases where the projects have been implemented it is currently too early to judge savings, as the projects have not yet given rise to savings over a sufficient time period for an accurate assessment to be made (this is particularly true for projects which have seasonal energy use). In addition, there are practical difficulties in estimating the realised savings (for example, insufficient resolution of metered data) and a variety of approaches have been taken in the estimation of these consumption values by the different local authorities.
Any inaccuracies from these estimates will feed into the assessment.
How addressed in this evaluation
To provide a sanity check on the data an assessment was made of payback periods for each of the types of technology funded under CEEF. Figure 10 shows the range of payback periods calculated within the completed application forms for each technology type and compares this with published information on typical paybacks (The Carbon Trust Good Practice Guide: GPG312: - Invest to Save: Financial Appraisal of Energy Efficiency Measures across the Government Estate). This is indicated by the minimum, average and maximum values. The wide range of payback periods can be attributed to the large variety of equipment in each technology type and to the variety of building types the projects have been installed in.
Figure 10 - Range of Payback Periods within CEEF Applications
The following table further compares these quoted payback periods and then compares them with an expected range of payback periods as presented within GPG 312. It is important to note that the simple payback period depends on the cost of the fuel at the time - the latter paybacks were based on data assembled over a ten year period, and the fuel prices may therefore differ from today's. In addition the data in GPG312 shows a wide range of possible projects, some of which would have been rejected from CEEF as the payback periods exceed the CEEF five year limit. It is therefore not possible to draw quantitative conclusions from this comparison, other than to note the data seems broadly consistent.
Table 5 - Comparison of quoted CEEF Payback Periods with standard data
Technology | Average Payback | Range of CEEF Paybacks | Range of Stated GPG 312 Paybacks |
|---|
Motors and Drives | 2.2 | 0.2 to 5 | 0.5 to 5 |
|---|
Pool Covers | 2.5 | 0 to 4.3 | 2 to 3 |
|---|
Pipe Insulation | 2.5 | 0.9 to 4.9 | 1 to 5 |
|---|
Cooling Controls | 2.7 | 0.1 to 5 | 1 to 2 |
|---|
Cavity Wall Insulation | 2.8 | 0.9 to 4.9 | 3 to 6 |
|---|
Other | 3 | 0.1 to 4.9 | - |
|---|
Draught Proofing | 3.2 | 1 to 4.9 | 1 to 3 |
|---|
Change of Fuel | 3.5 | 1.2 to 4.9 | - |
|---|
Low Energy Lighting | 3.7 | 0.8 to 5 | 1 to 12 |
|---|
Building Controls | 3.7 | 0.6 to 5 | 1 to 6 |
|---|
Roof Insulation | 3.8 | 2.2 to 5 | 2 to 4 |
|---|
Improving Boiler Plant | 4.1 | 1.9 to 5 | 2 to 6 |
|---|
Point of Use Hot Water | 4.2 | 4.2 | 2 to 3 |
|---|
CHP | 4.8 | 4.6 to 5 | 3 to 6 |
|---|
This showed wide ranges of behaviour which were nevertheless consistent with those observed by a range of other studies, and can reflect variations in project parameters as well as data analysis approaches.
Further to this, a number of individual projects have been evaluated in greater detail. This has enabled comparison in some cases of before and after estimates, but in the main this was not possible as the projects had only recently been commissioned. In addition it must be borne in mind that for most, dedicated sub-metering was not available and instead it was necessary to calculate estimates, usually by consulting derived data based on modelling. The methodology adopted was considered sound but a detailed assessments of outcomes was not possible at this stage.
Some further advice or guidance on how the implemented savings should be estimated is likely to be beneficial, and will ensure a more consistent approach to the estimates.
Recommendations for the future
It is suggested that applicants be required to provide information on the actual energy reductions obtained in subsequent years after project completion, together with the methodology used. This could be carried out as part of an annual reporting process. This would considerably increase confidence in the data.
6.2.3 Project lifetimes
When assessing project savings it is necessary to estimate how long the project will operate for. Estimates are available of the lifetimes of the technologies supported by CEEF, as shown in Appendix 3, and these have been used in this evaluation. Clearly there will be individual variation around these figures. Nevertheless, it is not clear if projects will run for the full technology lifetime, or whether this will be cut short, for example if the building they were installed in closed.
How addressed in this evaluation
It has been assumed that projects persist for the average technology lifetimes.
Recommendations for the future
In future assessments of the impact of CEEF it would be appropriate to test how long projects have continued to operate for and to compare this with the technology lifetimes.
6.2.4 Future projects
An assessment has been made of the savings arising from existing projects. However, the number of projects is increasing with time, and this will clearly increase the impact of CEEF. Therefore additional estimates have been made of the likely level of potential future projects, based on the current rate of project submission, and on the average payback, capital cost and energy and carbon savings per project. Clearly if this changes in the future, any inaccuracy about the assumptions will influence the accuracy of the results.
How addressed in this evaluation
The estimate of the number of likely projects was compared with the actual number at the end of December. This showed good agreement - however, this is over a short timescale so will not reflect any changes in how the local authorities promote CEEF. For example, those local authorities who have made least progress are understood to be developing action plans to increase uptake of projects: this could result in a significant increase in the number of projects.
Recommendations for the future
CEEF has operated for a relatively short period of time, during which the rate of uptake has increased significantly as awareness of the fund has grown and teething problems have been overcome. Nevertheless there is still a wide variation in uptake between local authorities. If some of the local authorities who have made relatively little progress to date are able to take full advantage of CEEF this could substantially increase the rate of project uptake. At the same time, as the fund progresses some local authorities may find it more challenging to develop projects within the five year payback criterion. Nevertheless, future evaluations will have a longer period of fund operation to draw on, during which these issues may be clearer.