Scottish Government

Scotland's People: results from the 2001/2002 Scottish Household Survey
Volume 8: Technical Report

4.4 Achieved sample profile and weighting issues

Two types of weighting are potentially necessary with a random probability sample of this kind. The first are intrinsic to the survey design and are necessary to compensate/adjust for unequal probabilities of selection for individuals, households or other units of analysis. The second are extrinsic to the survey design but may be necessary to counteract the effects of non-response bias.

Weighting for analysis based on household data

The weights for analysis of household data have two main elements. Firstly, it is necessary to 'weight up' those local authorities which were under-sampled and 'weight down' those which were over-sampled (this is a weight of the first type mentioned above, which adjusts for unequal probabilities of selection). Secondly, to weight addresses any disproportionality introduced by differential response rates by local authority within quarters. The combination of these two elements is shown in Table 4-5. (The weights for some local authorities vary between one quarter and the next because the number of achieved interviews fluctuate between quarters - see Section 4.1). The final sample profile across the two years should, therefore, correctly reflect the distribution of households across Scotland's local authority areas, as estimated by the Scottish Executive ⁵.

Weights are calculated for each local authority each quarter and based on the quarter in which the interviews were undertaken (regardless of when the address was sampled). This reflects the need for the data to be nationally representative in each quarter and should allow any published findings to be reproduced by selecting the relevant quarter's data. In practice, however, it may not be possible to reproduce exactly some of the results from earlier publications if the data for that quarter were subsequently changed (e.g. to correct errors that were identified later).

Table 4-5 Weights to account for 'under'/'over' sampling and differences in response rates by local authority by quarter: SHS 2001/2002

No other weight is applied across all cases in order to compensate/adjust for the unequal probabilities of selection. Strictly speaking, however, a corrective weight should be applied in those cases in which the Multiple Occupancy Indicator (MOI) on the PAF is found to be inaccurate. The reason for this is that a property-type bias might otherwise be introduced. For example, if tenement properties were consistently found to contain multiple dwellings when the MOI used in the selection of addresses for the sample had indicated that they contained just one, each achieved interview at such an address should be given a weight proportional to the actual number of dwellings, to compensate for the reduced probability of selection for each dwelling at that address. All properties within that local authority area should then be weighted back down slightly in order that the actual and weighted sample sizes remain the same.

In practice, the MOI was found to be inaccurate in 2.6% of cases. The impact of weighting to correct for these would have been negligible so it was decided not to weight by the MOI in order to avoid additional complexity in the weighting scheme for the survey. This issue is reviewed on an annual basis.

Similarly, in theory an additional weight should be applied in cases where a dwelling contains more than one household, only one of which is interviewed, in order to adjust for the lower probability of selection for each of the households in that dwelling. In practice, however, as only a very small number of dwellings were found to contain more than one household, the use of such a weight would have made very little difference to the overall results, and it was therefore felt that it was not worthwhile introducing further complication to the weighting calculations.

Weighting for analysis based on individual (random adult) data

Using the Postcode Address File produces a sample of households, so for analysis of individual level data it is also necessary to weight the responses of the 'random adult' by the number of adults resident in the household who were eligible for interview ⁶. The reason for this is that individuals living in larger households have a lower probability of selection.

As a result of this, one would expect the unweighted profile of 'random adult' respondents to be skewed towards those sections of the population most likely to live in households with fewer adults (older people and older females in particular) and away from those likely to live in households with larger numbers of adults (younger people). Once the data are weighted by the number of eligible adults in the household, however, one should see the profile correct itself significantly. In most surveys of this kind, however, some under-representation of younger people and males, and over-representation of older people and females, is likely to remain because of the effects of non-response bias. Depending on the extent of the remaining skew, it may be necessary to adopt further corrective measures - an issue considered in detail in the following section.

Analysis of data based on the 'random adult' also requires a corrective weight to take account of differences between the number of such interviews completed in each local authority area and the actual adult population of such areas. Like the element of the household data weight which adjusts for differences in fieldwork outcomes by local authority, this is intended not to compensate for unequal probabilities of selection but to ensure that the final profile of 'individual' data correctly reflects the relative populations of the different local authority areas once variations in fieldwork outcomes have been assessed. This is not identical to the weight described for analysis of household data, since variation in response rates for the second part of the interview may have produced a slightly different distribution from that of 'highest income householder' interviews. The weights required for each local authority (which are then multiplied by the number of adults in the household to create the weight for each case) are summarised below.

Table 4-6 Average weighting factors to adjust for under and over-sampling and differences in random adult response rates by local authority

Weighting for analysis based on the 'random schoolchild'

There is one further weighting factor needed to adjust for unequal probabilities of selection, relating to the information collected about a 'random schoolchild'. For this information to represent correctly the population of schoolchildren resident within households, it should be weighted by the total number of eligible schoolchildren resident within each household. If not, it will proportionately over-represent the characteristics and experiences of 'only' children and under-represent those of children from larger families. The weight for the random schoolchild case is created by combining this weighting and the relevant local authority weight.

Weighting for analysis based on the Travel Diary

Examination of the SHS data suggests that significantly fewer interviews take place on Fridays, Saturdays and Sundays than on other days of the week. As differences in the proportions of adults interviewed on each day of the week will affect the Travel Diary data's representativeness of travel patterns for the week as a whole, it was decided to introduce a weight to compensate for this. This simply 'up-weights' interviews carried out on days of the week on which fewer than one-seventh of all interviews have taken place and 'down-weights' those carried out on days on which more than one-seventh of all interviews have been completed.

It is also apparent that the distribution of interviews by the day of the week differs for certain sub-sections of the adult population. For example, disproportionately more adults in full-time employment are interviewed at the weekend (due to their greater availability then), thus yielding an inaccurate picture of the travel patterns of those in full-time employment. The Travel Diary weighting factor is therefore refined to compensate for this.

The weight created for any analysis of the Travel Diary combines the above weighting factors and the existing 'random adult' weights. Further information about the Travel Diary, including a comparison to the National Travel Survey, is available in the Travel Diary User Guide ⁷.

4.5 Data quality and comparisons with external sources

We turn now to the issue of whether additional post-survey weighting is required to address any residual bias in the sample profile (arising, for example, from differential patterns of non-response across sections of the sampled population).

Age and sex profile of the 'random adult' sample

We saw earlier that the unweighted sample automatically under-represents those living in multi-adult households, since they have a smaller chance of selection for interview. As Table 4-7 shows, therefore, the weighting to equalise probabilities of selection has a significant effect on the profile of the 'random adult' sample. The data shown have been weighted both by the number of adults resident in the household and by the local authority weight described in the previous section. These two weights tend to act in the same direction, since those larger local authority areas which are 'weighted up' also tend to be ones with a higher average household size.

Table 4-7 Comparison of weighted and unweighted age and sex profile of 2001/2002 SHS data with Census estimates

* Weighted by number of adults and local authority size

The weighted sample for 2001/2002 still does not match exactly the profile of the adult population suggested by the Census estimates with, as expected, under-representation of younger people in general and 16-24 year olds in particular. Consequently, older people are over-represented in the survey.

However, there are some reasons for being cautious about seeking to 'correct' the remaining imbalances. Firstly, the survey estimates of the age and sex profile of the sample are - like all its estimates - subject to sampling error. The 95% confidence interval for the percentage of males aged 16 to 24, for example, is likely to be in the region of 1.3% - slightly less than the difference between the SHS sample percentage and the percentage suggested by the Census. Secondly, the age/sex profile of the random adult sample is not greatly different from the profile of all adults resident within the households at which an interview was carried out. This suggests that non-response to the second part of the interview has not contributed significantly to a skewing of the 'random adult' sample. Finally, there are good reasons to think that the population that is outside the scope of the survey - student halls, nurses homes, prisons, army barracks etc, will disproportionately contain adults of the age groups that appear to be under-represented. Conversely, older people will be disproportionately represented in the population in hospitals and nursing homes. Thus, although the sample might differ from the profile of the population as a whole, it might not differ greatly from the profile of the population in private households.

However, the Census does provide an opportunity to establish a more thorough weighting scheme for the survey by allowing comparison of the characteristics of responding and non-responding households. The SHS has been included in the Census-linked study of survey non-response that is being carried out by ONS. The results of this study will be used to develop further weighting for the SHS.

The following sub-sections examine this issue further through a comparison of other key household variables with information from other sources.

Household type, property type, tenure and number of bedrooms

We noted above in Table 4-7 that the SHS appears to under-represent young adults and over-represent older adults. This is also apparent when household types in the Census are compared with the Census (Table 4-8).

Table 4-8 Comparison of household types in the 2001/2002 SHS and the 2001 Census

* SHS data weighted by local authority size only

As Table 4-9 shows, the sample appears robust in terms of the variables associated with accommodation/property characteristics. There is a slight over-representation of outright-owners relative to the Census and under-representation of 'other' tenures. Other differences from the census are only one percentage point.

Table 4-9 Comparison of key variables between the 2001 Census, 1996 SHCS and the 2001/2002 SHS

* SHS data weighted by local authority size only ^‡ includes households in shared dwellings

^† Pays part rent and mortgage (shared ownership) included in 'Own with mortgage'