ANNEX B - DATA AND MODEL UNCERTAINTY - HOW RELIABLE ARE THE RESULTS?
Background on uncertainty
B.1 Establishing uncertainty of model calculations is non-trivial. Sources for potential errors need to be known, the extent of variation in input data needs to be established and it needs to be understood how errors translate through to the final result.
B.2 It is important to distinguish between 'accuracy' or 'trueness' (systematic error) and 'precision' (indeterminate or stochastic error). Accuracy is the agreement between the modelled (or measured) value and the true value (which might or might not be known). Inaccurate results occur if there is a systematic bias resulting in values that are too high or too low. Precision relates to stochastic processes which generate a spread in the calculated (measured) values of a quantity. Better precision means less random error. Precision is independent of accuracy.
General uncertainty of SEI's calculation model
B.3 There are very few empirical studies on error estimates associated with (multi-region) input-output analysis so far; in particular, with environmental MRIO studies. Input-output models in general are based on the assumptions that there is proportionality between monetary and associated physical flows in an economy and that industries within a sector are homogenous, i.e. use the same production methods. Compared to bottom-up process-based methods this leads to less precise but arguably more accurate results as there is no truncation of economy-wide indirect impacts (see Minx et al., 2008 in the references).
B.4 In the course of the UK- MRIO project (Wiedmann et al. 2008a, see also Wiedmann et al., 2008b), SEI undertook the first comprehensive Monte-Carlo analysis of the uncertainties in a global multi-region input-output model, which is also the precursor of the two-region model REAP. The analysis of uncertainties of UK- MRIO model results tried to capture all possible variations of underlying data and calculation procedures. For the majority of data this takes account of the random uncertainty of data points due to statistical variation and random errors. In other words, we have determined the stochastic variation of the whole model system.
B.5 The results of this uncertainty analysis show that, with statistical significance, CO 2 emissions embedded in UK imports ( EEI) were higher than those for exports ( EEE) in all years from 1992 to 2004 and that EEI were growing faster than EEE, thus widening the gap between territorial (producer) emissions and consumer emissions. For aggregated results (CO 2 consumer emissions), the relative standard error has been shown to be between 3.3% and 5.5%. Therefore, the estimate of total embedded emissions can be regarded as robust and reliable. The authors emphasise, however, that on an individual sector level these errors are generally higher and that the precision is not sufficient for a robust footprint or life cycle analysis of individual products.
B.6 It is important to mention that revisions of underlying data (e.g. input-output or Environmental Accounts data from ONS) create a systematic shift of results. These revisions are usually carried out once a year and emissions data for individual sectors can change up to 10% for CO 2 and up to several hundred percent for the other GHGs (following revisions to activity data and emissions factors and improved quality in coverage). Results for embodied consumer emissions will therefore change accordingly whenever revised input data is used to update the model system.
B.7 Other possible systematic error sources include structural changes and sectoral price changes in foreign IO data over time, systematic over- and underestimation of carbon intensities of foreign industries due to the mismatch of sectors in UK and foreign IO and CO 2 data, change of import structure over time, or the choice of currency conversion factors. In principle it is possible to investigate and model these systematic errors methodically. Such detailed modelling of systematic uncertainty, however, was way beyond the scope of the UK- MRIO study and instead we have dealt with the uncertainties arising from systematic changes by allowing a stochastic variation large enough to capture anticipated systematic variation.
B.8 Based on the uncertainty estimates mentioned above we calculate that the time series of GHG footprint for Scotland has a relative standard error ( RSE) for annual estimates in the region of ±5% for CO 2 and in the region of up to ±10% for other GHGs. This gives an RSE for all GHGs in the region of ±7% (around 70-80% of GHG emissions are CO 2).
B.9 On the assumption that the errors are normally distributed there is a 68% probability that the true value for a particular year lies within ± 1 RSE of the estimate and a 95% probability that the true value lies with ± 2 RSE. Assuming that for any two years the estimates A and B are independent and the RSE are the same then the SE of the difference is given by ÷[(Ax RSE)2+(Bx RSE)2]. As it is likely that the estimates for any two years are positively correlated (in which case the SE would be smaller) this give a worse case estimate for the SE when determining whether the difference between two years is statistically significant. In the case of the change between 1992 and 2006 in the GHG footprint per capita, the difference is 1.95 tC0 2e and the estimated SE of the difference using the above formula is 1.56 tC0 2e. Assuming normality of errors, then this change is not significant at a 95% level of confidence but is significant with a level of confidence of approximately 80%.
Specific uncertainty and reliability of Ecological Footprint time series for Scotland
B.10 In addition to the error source for input-output modelling of GHG footprints mentioned above, the estimation of an Ecological Footprint bears additional uncertainty due to the fact that this is an composite indicator which combines several land types weighted with various equivalence and yield factors - each of which are associated with their own uncertainty (for the basic calculation method of the EF of UK production as used in this project see Kitzes et al., 2008). Due to these additional uncertainties we therefore cautiously estimate the relative standard error to be more than for CO 2, possibly in the region of other GHGs (i.e. in the region ±10%, see paragraph B.8). This is a rough estimate as we have not performed an actual uncertainty analysis of underlying data.
B.11 As described in section 4 our method of calculating the national EF of consumption for the UK differs from the one adopted by Global Footprint Network ( GFN). This is due to a fundamental difference way of attributing impacts to international trade (described in Wiedmann, 2009). This creates a systematic shift in values for the trade balance and therefore in the total footprint for consumption. The difference between the EF totals per capita from SEI and GFN can therefore be regarded as an upper limit of systematic errors that can occur between different model systems. For the UK data, this is up to 15% in the per capita results.