CHAPTER TWO. RESEARCH METHODOLOGY AND ANALYSIS
RESEARCH METHODOLOGY
2.1 The questionnaire for the survey was designed by NOP, working in close association with the Scottish Executive Health Department and the Research Advisory Group, representing a range of interested parties within the Executive and the NHS in Scotland.
2.2 The interview collected information on a range of subjects:
- Self-reported health status
- Views on the NHS in Scotland in the past and expectations for the future
- Awareness of emerging NHS initiatives and services
- Use of services
- Out-of-hours
- Telephone consultations
- Primary Health Care Team (PHCT)
- Home visits
- Outpatient services
- Inpatient admissions
- Overall satisfaction with NHS services used in the last year
- Public involvement in the NHS in Scotland
- Patient feedback
- Demographics
2.3 The questionnaire was designed to reflect the key survey objectives noted in Chapter 1 of this report - these issues were addressed within the structure outlined above, e.g. the section on inpatient admissions included material relating to choice, satisfaction, access and aspirations for improvement. The section on the PHCT included a check on which health professionals had been seen by the respondent in the last year. Where more than one person had been seen, only one of these contacts was selected for detailed questioning. Prioritisation was given to categories of health professional which would come up only occasionally in order to ensure a robust sample of cases for the respondent's own GP, other GPs at the surgery and for the Practice Nurse.
2.4 The draft questionnaire was tested via a two-stage pilot study with 35 respondents. The pilot interviewing was spread over two evenings to allow for some revisions to be made to the questionnaire during the test fieldwork. All of the interviewers working on the pilot survey were personally briefed by the project team. Respondents were asked about questions they found problematic and were also asked directly about particular issues in relation to understanding and answering certain questions. The final questionnaire, marked up with topline findings, is available electronically through the following link: http://www.scotland.gov.uk/socialresearch …
2.5 All fieldwork was conducted via Computer Assisted Telephone Interviewing (CATI), a technique which uses a computerised questionnaire in order to avoid the need for interviewers to manually check question routing. This leads to better data quality because of the reduction in filtering errors compared with conventional pen-and-paper data collection. All main-stage interviewers were personally briefed by the NOP executive team. The average interview length was about 23 minutes.
2.6 A total of 2600 interviews were conducted in March and April 2004 with a representative sample of adults aged 16 and over, from all parts of Scotland. The sample frame consisted of randomly generated numbers in proportion to the population distribution across Scotland. The sample profile was quota-controlled by sex, age, working status and region.
2.7 Eight out of ten of those interviewed said that they would be willing to take part in future surveys for the Scottish Executive - this gives some scope for follow-up research. The next planned survey in this series is due in 2006.
2.8 The 2004 methodology was very similar to that employed on the 2000 survey - this ensured that, where questions were consistent, comparisons could be made across time. Many in-home social surveys have quite long interview lengths but telephone interviewing has a shorter acceptable length of questionnaire. The use of telephone fieldwork meant that the interview had to be concise, so that sections such as improvements could not use open-ended questions without an adverse effect on the coverage of other issues. It was not possible to include all of the topics suggested for this survey without extending the interview length beyond the acceptable limit.
2.9 The use of a telephone survey by definition excludes the very small number of households that do not have this facility but the approach brings advantages as well, notably in the generation of an un-clustered sample but also in terms of cost-effectiveness. Telephone interviewing can reach isolated properties in very rural areas as easily as blocks of flats in the centre of major cities. The random generation of telephone numbers gives access to ex-directory homes, though the issued sample is screened to remove numbers that have been registered with the Telephone Preference Service. Respondents are told that their numbers have been generated randomly rather than obtained from a directory or other source.
2.10 The sampling frame consisted only of landline numbers and for practical reasons excluded mobile phones. The exclusion of non-landline households is not a significant factor at present but is likely to become a greater concern over time as more (younger) people move towards exclusive use of mobile telephones.
ANALYSIS
2.11 The question responses were cross-analysed by the following variables:
- Sex
- Age (7 groups - 16/24 through to 75+)
- IPA social class (4 groups - AB, C1, C2 and DE)
- Tenure (3 groups - own/buying, social renting and private renting)
- Use of the six key NHS service areas (Out-of-hours, telephone consultation, Home Visit, Primary Health Care Team, Inpatient and Outpatient)
- Overall satisfaction with NHS services used in the last 12 months
- Limiting illness/disability/infirmity
- Self-reported health status
- Age interlocked with sex (8 groups - 16/34, 35/54, 55/64 and 65+)
- Urban/rural split (6 groups, based on geographical remoteness and settlement size, defined by postcode and based on those used for the Scottish Household Survey or SHS - the four main cities, other urban areas, small accessible towns, small remote towns, accessible rural areas and remote rural areas).
- Deprivation (5 equal-sized groups, based on the 2004 Scottish Index of Multiple Deprivation or IMD data, ranging from the most deprived quintile to the least deprived)
2.12 The survey data was target weighted for sex, age, tenure and social class, using information from the 2001 Census (for sex, age and tenure) and NOP's in-house database (for social class). Sample profile tables, for both 2000 and 2004, are included in Chapter 3 of this report.
2.13 All of the sample differences highlighted in this report are statistically significant, at the 95% confidence interval. Across the overall sample of 2600, the maximum estimated sampling error was +/-2.2% - this means that for a finding of 50%, there were 19 chances in 20 that the real result (i.e. if all adults in Scotland had been interviewed) was between 48.8% and 52.2%. More information on sampling error is provided in the Annex.
2.14 Some variables appear regularly as key discriminators (sex and age, for example) while others are less routinely significant (for instance, whether the respondent lived in an urban or rural location). More information on the sample profile and the relative significance of demographic factors is given in Chapter 3. Throughout the report, where the data is presented in detailed table format, the most statistically significant results appear in bold italics. Base numbers (i.e. those answering a particular question) are included as un-weighted figures in all of the tables and charts and are sometimes used in the commentary.
2.15 Comparisons have been made where possible with the findings of the 2000 survey and also with other relevant research from across the UK. Differences in context, wording, means of data collection and weighting place limitations on the extent to which comparisons can be made with the 2000 survey or other sources. For example, while the methodology was almost the same, the social class profile of the 2004 sample was different to that shown in the 2000 survey. The development of modernisation meant that there were also some notable differences in the question wording used on the two surveys. The 2004 survey was designed to reflect the current policy context subsequent to implementation of the agenda set out in 'Our National Health'. For this reason, there is only limited scope for direct comparisons between the two surveys and the 2004 survey rather than that conducted in 2000 is designed to provide a consistent baseline for subsequent research into public attitudes.
2.16 Some comparisons are also made with data from the most recent English tracking survey commissioned by the Department of Health - this 2003/04 project also explores public attitudes towards, and perceptions of, the NHS. It uses a smaller sample than in Scotland - just over 1000 interviews - and is conducted face to face rather than via the telephone. There are also some comparisons made with the last published data from the Executive's Scottish Health Survey but this uses information from 1998 as the 2003 findings were not available at the time of reporting. Questions used in these other surveys are generally phrased differently to those used in Scotland in 2004 but they allow for broad comparisons to be made.
2.17 Professor Roy Carr-Hill from the Centre for Health Economics at the University of York conducted a series of secondary analyses in order to draw out more detail in the area of satisfaction. This analysis has been integrated into the main report. He carried out multivariate analysis to explore the relative contribution of socio-demographic variables such as age, sex, social class and educational qualifications compared with objective features of the contact such as waiting time to discriminate between those who were less satisfied and those who were very satisfied with services. Even where there were only relatively small numbers involved, this analysis can show statistically significant differences between groups in a way that is not always apparent in standard cross-tabulations - see for example, Chapter 5.2 with regard to the Out-of-hours service.