Scottish Government

4.0 Comparing JAC- SAF at Individual and Aggregated Level

This section meets Objective (iv): "To compare data from the JAC and SAF datasets at both the aggregated and individual unit level and to assess the suitability of SAF data for statistical purposes"

4.1 Introduction

The objective of of this and the next section is to assess the suitability of SAF data for statistical purposes. To this end two comparisons were desired.

1. An individual comparison on a holding-by-holding or business-by-business basis of the JAC and SAF values for each comparable Item ( CI) (see Section 1) - these individual comparisons are then aggregated to give overall national figures and breakdowns by size, Farm Type and NUTS4 region. This compare-then-aggregate ( CA) strategy can consider only those holdings or businesses common to both datasets - the overlap in Figure 11 (59% of holdings or 87% of area - see Section 2). The individual comparison is presented in Section 3.

Figure 11 - Datasets considered in the analysis.

2. An aggregated comparison considers the two dataset separately. For CI's in each dataset aggregated values are calculated for the national, size, Farm Type and NUTS4 regional groupings. These aggregated values are then compared. This is the aggregate-then-compare ( AC) strategy and is presented in Section 4.

Both analyses are required since they each have complementary advantages and limitations. Since CA uses individual comparisons both the absolute levels of error ( AE) ¹⁰ and the compensating errors ( CE) ¹¹ between the datasets can be estimated. CE's are smaller than the AE values but could be the more significant measure for national level statistics if overall totals are more important that specifying exactly what is where. The importance of AE's is increase for lower levels of aggregation where the smaller populations mean that there is less opportunity for compensating errors this is most significant at smaller levels of aggregation where it is necessary to specify more exactly what is where. The AC strategy since it aggregates first can only present the compensating errors, but AC is advantageous since it makes comparisons using data from all the holdings in each dataset, whereas CA uses only the subset of holdings which are common to both. Finally the CA strategy also has the advantage of allowing the generation of histograms plots presenting the count of holdings versus the size distribution of their errors.

4.2 Materials and Methods

The analysis uses the same datasets as section 2.0. These are the SAF and JAC datasets for 2005 to 2007 both held as Oracle tables. A schematic for the analysis is presented in Figure 2 below.

Figure 12 - Schematic for the analysis in Section 3 and 4.

Drawing on the meta-data in Section 1 (defining the corresponding categories in the two datasets and the nature or their match) and on the data structures of Section 2 (set up to compare the SAF and JAC populations) the analyses outlined in Section 1 were made. For the holdings in common (determined by matching the SAF and JAC database tables) the areas per item per holding were tabulated and the absolute and compensating differences calculated using SQL queries. The outputs from these queries were exported to indexed MS-Excel files to be compatible with further analyses by Analytical Services. The results were sorted and grouped as required for the tables in Section 3. Histograms showing the distribution of difference values for the year 2007 were also prepared in MS-Excel (one file per JAC item) - for examples see Section 3. For Section 4 the complete populations of both JAC and SAF holdings were classified and grouped separately by size, Farm Type and NUTS4 region (4 SQL queries) the difference values were then calculated and exported to MS-Excel files.

4.3 Results

4.3.1. Individual Comparison

The outcomes of the individual comparisons are in all cases presented by commodity grouping and then ordered by JAC Items. Additional summary items suggested by SG Analytical Services are italicised. For Section 3 each table presents the Item, Year, the JAC common total, the SAF( FDS or L) total, the difference SAF- JAC (Diff), the difference as a percentage of JAC common total (%Diff), the absolute difference ( ABSDiff) and the ABSDiff as a percentage of JAC common total. Positive figures for the difference values mean that the SAF is larger than JAC while negative values mean SAF is smaller.

Overall (National) All Data - 2007 Only

For Crops (Table 1) the range in Diff is from 1% (Spring Barley and Total Cereals) to 362% (Mixed Grains). The largest errors tend to be associated with Items with broader definitions e.g. Mixed Grains or Other/unspecified crops (362% and -79%) or for Items with smaller areas e.g. Lupins (107%) where one or two differences can have a big impact. The large disagreements between JAC and SAF for "Mixed Grain" may result from the use by JAC of a more restrictive definition, where only mixtures of two or more of wheat, barley and oats are acceptable. There are more Items with negative Diffs ( JAC> SAF) which may reflect differences in how area calculations and deductions are made in the two datasets. The AbsDiff values are larger than Diff, and in some cases much larger, e.g. 23% versus 1% for Spring Barley. In some cases where the Diff is large the AbsDiff shows a smaller rise. e.g. Lupins with 107% versus 181%.

Table 13 - Total Crops Data for 2007, Differences

Comparison of Spring Barley (ITEM18) reporting between JAC (All) and SAF (FDS) for 2007 - main holdings only

Figure 13

The distributions of the % Diffs for the individual holding-to-holding comparisons have been calculated for each of the Items at National level and with a breakdown for Main and Minor holdings. The distribution charts used for illustration in Section 3 are those for Main holdings only, since it was the view of Analytical Services that including the minor holdings in a presentation of holding counts would skew the patterns since minor holdings make "very small" contributions to the overall totals. The distribution charts show characteristics patterns for good and poorer matches. Spring Barley (Figure 3) is an example of a good match, with a strong "peak" around the "zero" difference value with a tailing off in counts as the size of the errors increase. There can often be an increase in counts at the edges of the histograms since the categories <-80% and >80% capture a wider range than the inner bins.

The outer bins are also unusual since they are designed to capture the specific circumstances when one of the datasets records an area while the other does not ¹². The holdings listed on SAF but not on JAC ( JAC=0) may contain a small number of holdings which have never responded to census, but the bulk of them seem more likely to be landless keepers submitting SAF. Landless keepers own no land, and rent no land under a full tenancy, and thus do not receive a census form. They may carry out agricultural activities on seasonally let land or on common grazings. Crop growing on seasonally let land should be reported by the land owner on their census form, but would not form a part of their SAF submission leading to a complementary mismatch for two holdings (contributing to larger absolute than compensating difference values).

Comparison of Spring Oats (ITEM20) reporting between JAC (All) and SAF (FDS) for 2007 - main holdings only

Figure 14

Where the match is poorer then the two outer categories become more prominent, e.g. Spring Oats (Figure 4). It is worth noting that for Spring Oats it is not one of the outer bins that is dominant (which would mean that information is being recorded in one dataset and not the other), but that there are cases for both datasets where individuals are recording data on one form but not in the other. Eventually the outer bins can come to dominate e.g. Bare Fallow (Figure 5). The histograms are also useful in identifying cases where one dataset is not reporting fully for some reason - e.g. Total Potatoes (Figure 6) shows 815 holdings where FDS=0 and JAC>0 versus 290 holdings where FDS>0 and JAC=0. There are also asymmetrical distributions of the inner bins, e.g. Total Crops, Fallow and Set-aside and this reflects the composite nature of the Items (Figure 7).

Comparison of Bare Fallow (ITEM39) reporting between JAC (All) and SAF (FDS) for 2007 - main holdings only

Figure 15

Comparison of Total Potatoes (ITEMS 24/25/26) reporting between JAC (All) and SAF (FDS) for 2007 - main holdings only

Figure 16

Comparison of Total Crops, Fallow, Set-aside (ITEM40) reporting between JAC (All) and SAF (FDS) for 2007 - main holdings only

Figure 17

While the Grass Items have similar magnitudes of Diff and AbsDiff to the Crops Items the patterns of the distributions of differences are significantly different. While Grass>= 5 years only has a Diff of 4% the AbsDiff is large at 60% and there are 2006 holdings with SAF values >80% larger than JAC (Figure 8). This may reflect the seasonal renting in of pastures, with renters reporting larger holdings via SAF than the land they own and report in the JAC. This is balanced by significant underestimations in the -40% to -20% and higher categories where the owned land reported in JAC is bigger than SAF claimed land. This phenomena can also be seen (though less strongly) for Grass <5 years and Total Grass. The seasonal renting issue may also contribute to the size of the end bins if a whole holding is rented in or out with returns for one datasets being matched by null values for the other. Where there are units that may not make claims via SAF (e.g. some deer estates) but do return JAC then the left hand bin of the distribution is enlarged - e.g. Rough Grazing (Figure 9). Woodland is heavily under-reported in SAF.

Table 14 - Grass, hay, straw, silage

Figure 18

Comparison of Rough Grazings (ITEM47) reporting between JAC (All) and SAF (FDS) for 2007 - main holdings only

Figure 19

The match for the Veg for human consumption items is poor with JAC consistently higher than SAF returns (Table 3). Notable are the 430 holdings with JAC returns or imputed values but no SAF (Figure 10). This reflects the relatively lower coverage of horticultural crop by SAF as seen in Section 2.

Table 15 - Veg for human consumption

Better matches are made for Soft fruit (Table 4) - but only when there are single crop comparisons being made - e.g. Raspberries and Blackcurrants. Where there are compound classes with several elements and small areas then the matches can be very poor (Mixed/other soft fruit at 708%).

Table 16 - Soft fruit

For nursery stock the numbers of holdings are small and the distributions this sparse. The match for Bulbs/Flowers/Bedding Plants is adequate - but with many cases of either JAC or SAF returns but fewer with both. The Hardy Nursery Stock is significantly underreported in JAC. This is a definitional issue (the match between the nursery stock items in JAC and the Trees Shrubs code in SAF is weak as it is compound and partial - see Section 1). This weakness is perhaps compounded by the ineligible status of the Trees, Shrubs and Bushes category in SAF that may mean the category is less stringently measured. A new SAF code for nurseries will be introduced in 2008 (since they will in some circumstances become eligible for SFP) and this should ensure a much better match between the two datasets.

Table 17 - Bulbs, flowers, nursery stock

Figure 20

Analytical Services have noted that since crop areas on JAC must be reconciled with the total farm area; this could encourage activities to be recorded under inappropriate items to make the form totals balance. This would occur mainly where it's not clear under which JAC Item a particular crop-growing activity should be recorded. They have also suggested that closer investigation of the differences for crop items may suggest where Census questions might be made less ambiguous or otherwise improved. SAF information, if of sufficient quality and if its relationship to JAC data were sufficiently well understood, might allow such changes to be made without damaging trend information.

For the livestock figures (Table 6) the comparisons are on a business-by-business basis and in most cases only the total figures can be compared since the SAF categories either do not match those of the JAC or the JAC categories are more specific than those recorded in SAF. The Diff figures for some of the Items are on first inspection very close, e.g. Total Pigs (0%) and Beef Cows and Heifers (-3%) - but the AbsDiff figures point to really serious problems with the comparisons (42% and 23% respectively). This is borne out by the distribution plots Figures 11 and 12. For pigs there are a significant number of occasions when for a business SAF has values but JAC does not. This is balanced by several occasions where the numbers of stock are larger in JAC than SAF (-40% to -20% is the peak). It is possible that these differences relate to when livestock numbers are recorded in the two surveys. For the headline figures of Total Cattle and Sheep (Figures 13 and 14) then there is a consistent pattern of larger numbers being reported in JAC (the peak in the -ve value bins) but also a significant number of businesses where there are SAF returns but no JAC. Note also that livestock kept on a seasonal let should be reported by the tenant, but if that tenant is a landless keeper, they will not receive a form and so their livestock will not be captured by JAC. This could be tested by looking at the holding numbers of SAF submissions not matched with JAC; landless keepers (and only landless keepers) will have holding numbers above 7000.

Table 18 - Pigs, Poultry, Cows, Sheep and Misc. Livestock

Figure 21

Comparison of Beef Cows and Heifers (ITEM0101/0103/0105/0107) reporting between JAC(All-b) and SAF(L) for 2007

Figure 22

Comparison of Total Cattle (ITEM0122) reporting between JAC(All-b) and SAF(L) for 2007

Figure 23

Comparison of Total Sheep (ITEM0145) reporting between JAC(All-b) and SAF(L) for 2007

Figure 24

Overall (National) All Data -2007 Only (Main and Minor)

This dataset again presents a national scale view of the individual holding to holding or business to business comparisons - in this case aggregated by main and minor holdings. Note that this analysis is not available for the livestock Items. The consistent pattern is that minor holding show larger and frequently much larger, Diffs and AbsDiffs than do major holdings. Contrast Figure 15 Spring Barley (minor) with Figure 3 for the distributions of %Diff.

Table 19 - Total Crops at the National Scale, Differences

Table 20 - Grass, hay, straw, silage

Table 21 - Veg for human consumption

Table 22 - Soft fruit

Table 23 Bulbs, flowers and nursery stock 1

Figure 25

Overall (National) All Data - 2005, 2006, 2007

The following tables present the time series of national data for the years 2005-2007 (note for some of the livestock Items the required information is not available in certain years). For crops there is a remarkable degree of consistency across time, with no flips from positive to negative except where the differences are very small e.g. Spring Barley between -1% and +1%. For the more extensive crops the changes are smaller but for some of the less extensive crops there can be larger changes - e.g. Triticale and Lupins. The year-to-year consistency is maintained for the Grass, hay, straw and silage classes in all cases except for woodland where there has been a decrease in the Diff and AbsDiff in 2007. The data is nosier for the remaining land based items with small changes having potentially large effects with patterns being difficult to discern or explain. For livestock Items there can be good consistency for simple overall categories e.g, Total Pigs, but for classes with more complex or compound definitions there can be significant "jumps" in the data series e.g. Beef cows and Heifers.

Table 24 - Total Crops at the National Scale, 005 to 2007, Differences

Table 25 - Grass, hay, straw, silage

Table 26 - Veg for human consumption

Table 27 - Soft fruit

Table 28: Bulbs, flowers, nursery stock

Table 29 - Pigs, Poultry, Cows, Sheep and Misc. Livestock

4.3.2. Aggregated Comparison

The outcomes of the aggregated comparisons are again in all cases presented by commodity grouping and then ordered by JAC Items. Additional summary items suggested by SG Analytical Services are italicised. For Section 4 each table presents the Item, Year, the JAC (All) total, the SAF( FDS or L) total, the difference (Diff), and the difference as a percentage of JAC (All) total (%Diff). These figures are generated independently for each dataset at the level of aggregation required and then compared.

Overall (National) All Data - 2007 Only

These tables show only minor differences from the Diff column in Section 3 with no discernable pattern to these differences.

Table 30 - Overall National Data, 2007 only

For cereals the overall picture has a difference of 1%, but as can be seen in Figure 16 there are some differences for particular Items, with SAF showing higher totals for Wheat and Mixed Gains but lower totals for Winter and Spring Barley.

Figure 26

For potatoes there is an apparent under reporting in SAF with JAC 16% or 4,800 ha larger.

Figure 27

Where single crop Items can be compared (see Figure 18) then the JAC and SAF show a degree of conformity (mid teens for percentage differences are typical and in some cases low percentages can be achieved e.g. peas at 2%). This conformity may tend to breakdown for Items with smaller areas such as Lupins where percentage errors can be large (110%). For compound or "catch all" Items such as Other/unspecified crops (see Figure 19) the correspondence between Items is poorer partly since the areas are smaller and any errors tend to be magnified, there may be differences in the reporting of such items and there may also be significant differences that result from how the categories are interpreted by those making the returns.

Figure 28

Figure 29

Differences between the grass and grazing Items are interesting since they give an insight into the differences in the holding populations for the two datasets. The most dramatic difference in terms of area is for rough grazings with a -845k ha difference (-25%). This almost certainly reflects the fact that there are holdings in JAC that are not returning SAF since they have not to date claimed "agricultural" payments - e.g. sporting estates. This is likely to change, however, with SAF used for administration of LMC and RDP measures. Grass leys are underrepresented in JAC by 29% and as noted in Section 3 this may be an artefact of how seasonally rented land is handled in the two datasets. The conformity for total crops and grass is good at 1% difference but differences in woodland remain significant at 58%.

Table 31 - Changes in grassland and others, 2007 only

Figure 30

For the Items with smaller areas there is generally poor conformity (Tables 13-15). This is the outcome of differences in the reporting populations, the greater effect of area estimation errors for Items with very limited total areas and the difficulties of interpreting and comparing compound Items.

Table 32 - Changes in root crops and pulses

Table 33 - Changes in soft fruits, 2007 only

Table 34 - Changes in Flowers, 2007 only

For livestock where there are comparable single Items then adequate conformity can be seen between JAC and SAF (e.g. Ewes and gimmers in Table 16 and Figure 21). In most cases, however, there is a poor match either since categories do not match exactly or because there is a relatively lower overlap between JAC(All) and SAF (L) that is achieved for SAF( FDS). This poor match is even the case for "Total" categories where issue of definitions do not arise. In this case the likelihood is that for some stock species (Total sheep and Total cattle) much of the difference may arise from the 3 month gap between SAF returns (March) and JAC returns (June), but this would not explain the differences of Total Poultry . Even when there is conformity between the two datasets it can be the case (as shown in Section 3) that the low Diff value is the result of cancelling out of large differences between individual businesses (e.g. Suckler cows).

Table 35 - Livestock Items

Figure 31

Overall (National) All Data -2007 Only (Main and Minor)

The tables below present the comparison between the complete 2007 JAC and SAF populations broken down by size classes (main and minor holdings). Where a holding exists in only the SAF then this has been recorded as Unspecified since the size class definitions are derived from JAC. As can be seen in the tables below the Unspecified class is usually small relative to main holdings but can be larger than the areas for minor holdings. The analysis does not present values livestock since comparisons between SAF and JAC can only be made at business level and the JAC size classifications apply to individual holdings. As with the comparison of common Holdings (Section 3.2) the minor holdings in general show the largest differences in percentage terms, since small differences are proportionately more significant.

Table 36 - Changes in cropping area