Scottish Government

Scottish Household Survey: Fieldwork outcomes 2003

4. Data quality

The issue of bias arises in every survey of the population. There are a number of sources of bias, some of which reflect aspects of the survey design (such as the sampling frame or who is deemed eligible for interview). However, bias is also a reflection of those aspects of fieldwork outcomes mentioned above:

• the quality of survey administration procedures
• whether potential respondents can be found at home at times when interviewers call
• whether they are able to participate in the interview, i.e. not restricted by ill health, disability or communication barriers
• the willingness of members of the public to participate in the research.

A high response rate is generally viewed as one of the key measures of data quality and, all other things being equal, a high response rate and a large sample should ensure accurate estimates. However, to the extent that non-response to the survey is not spread evenly, either geographically or between sub-groups of the population, the resulting bias will limit the accuracy of the survey's estimates. The question of bias is considered by comparing key results from the SHS with comparator data. Since the publication of the 2001 Census, this source is the most accurate comparator for population data and in spite of being two years behind the current SHS, population measures such as age distribution and household types change little from year-to-year.

Achieved interviews and data files

The following sections compare SHS data with other sources. These comparisons are based on the most recent processed data, which lags behind the number of interviews achieved. This means that the number of interviews reported on Table 2-3 at page 6 (15,023 household interviews) is greater than the number of interviews in the most recent data file for 2003, which contains 14,880 household interviews and 13,968 adult interviews. Interviews carried out on 2003 sample that were not included in the 2003 data are carried into the 2004 data files.

Household type, property type, tenure and number of bedrooms

As noted in Table 4-3 on page 18, the random adult sample appears to under-represent young adults and over-represent older adults. This is also suggested by the under-representation of single adult and large adult households when household types in the 2003 SHS are compared with the Census (Table 4-).

Table 4-1: Comparison of household types in the 2001 Census and the 2003 SHS

* SHS data weighted by local authority size only

As Table 4-2 shows, the sample appears robust in terms of the variables associated with accommodation/property characteristics. There is a slight over-representation of owners who own their property outright relative to the Census and under-representation of 'other' tenures. Other differences from the census are only one percentage point.

Table 4-2: Comparison of key variables in the 2001 Census, 1996 Scottish House Condition Survey (SHCS) and the 2003 SHS

* SHS data weighted by local authority size only
‡ includes households in shared dwellings
† Pays part rent and mortgage (shared ownership) included in 'Own with mortgage'
†† Although Scottish Homes no longer exists and had largely disposed of its rented housing stock the reference is retained in the questionnaire in case some tenants continue to think Scottish Homes is their landlord.

Age and sex profile of the 'random adult' sample

When a single adult is randomly selected within households, the unweighted sample of adults always under-represents those living in multi-adult households, since they have a smaller chance of selection for interview. As Table 4-3 shows, weighting to equalise probabilities of selection generally has the effect of bringing the profile of the 'random adult' sample closer to that of the adult population. The SHS data shown have been weighted both by the number of adults resident in the household and by the local authority weight described in the previous section. These two weights tend to act in the same direction, since those larger local authority areas which are 'weighted up' also tend to be ones with a higher average household size.

Table 4-3: Comparison of weighted and unweighted age and sex profile of 2003 SHS data with 2001 Census estimates

* Weighted by number of adults and local authority size
** Weighted by local authority size

However, even after this design weighting has been applied, the weighted random adult sample for 2003 still does not match the profile of the adult population suggested by the Census estimates with, as expected, under-representation of younger people in general and 16-24 year olds in particular. Consequently, older people are over-represented in the random adult sample.

Driving and transport

In relation to driving and transport, the survey results also look broadly in line with what one might expect from other sources such as the National Travel Survey and the differences which exist are, again, comfortably within the confidence intervals associated with the two surveys. Mode of travel comparisons with other sources are less conclusive, though methodological or classification differences may be playing a part here.

Table 4-4: Comparison of key variables relating to driving and transport

* SHS weighted by number of adults and local authority size
** SHS weighted by local authority size and number of school children in household
*** SHS weighted by local authority size only
† Census figures are for method of travel to place of study, age 5-17

Ethnicity

When comparing the ethnic composition of all household members with that of the population as a whole (as recorded in the 2001 Census), there is agreement in terms of broad categories. For example, in the Census, 98.0% of the population is recorded as White. In the 2003 SHS 98.4% of all household members are recorded as White. Even within the detailed categories the differences between the SHS and the Census are never more than 0.6%, although this can represent a large proportionate difference between very small categories.

Table 4-5: Comparison of ethnicity in Census 2001 and 2003 SHS

Urban/rural classification

Analysis of the Scottish Household Survey makes extensive use of the Scottish Executive's classification of areas into different degrees of urbanity and rurality. This classifies settlements according to their size and for settlements with a population of less than 10,000, their proximity to a settlement with a population of 10,000 or more. A full description of the urban/rural classification is given in the Glossary to the SHS Annual Report ( Scotland's People: results from the 2003 Scottish Household Survey).

Comparison of the households at which SHS interviews were achieved and the classification of all Scottish households shows that there is a good match between the two within local authorities although overall, large urban areas are under-represented. Table 4-6 compares the proportion of households in each local authority in each type of area.

Table 4-6: Comparison of 2003/2004 SE urban/rural classification and 2003 SHS

Row percentages, all Scottish households shown in bold, SHS in plain text

Rows may not always add to 100% because of rounding.

Economic activity

One area where the results of the SHS indicate significant differences from other sources is in relation to indicators of economic activity. As the following table shows, the most recent results from the Labour Force Survey (LFS) suggest that the SHS may be under-representing people in employment, and over-representing the economically inactive. It should be emphasised, however, that the information from the SHS shown here is based on the respondent's own classification of their economic activity (collected at the start of the interview), rather than on the full International Labour Organisation definition, which is not classified by the respondent and is the basis for official estimates of unemployment. The SHS is not an official source of statistics on employment (see Methodology, Section 4 on limitations of the data).

Table 4-7: Comparison of economic activity variables among adults of working age

* Weighted by number of adults and local authority size

Figures in this table have been calculated using all working age people as the denominator. Headline unemployment statistics are not calculated on this basis.