Scottish Government

CHAPTER TWO: RESEARCH METHODOLOGY AND ANALYSIS

Questionnaire design

2.1 The original survey questionnaire was developed by a multidisciplinary advisory group. The version used in the 2006 survey was revised by members of the project's Research Advisory Group, which comprised representatives from across the Scottish Executive, NHS Health Scotland, Edinburgh University, The Scottish Development Centre for Mental Health and representatives from the 'Choose Life' initiative, the 'see me…' campaign and the Scottish Recovery Network.

2.2 Specific topics covered in the questionnaire were as follows:

• general health
• length of residency in, and satisfaction with, the local neighbourhood
• informal support networks and civic participation
• perceived positive and negative influences on mental health
• control over factors affecting mental health
• a measure of positive mental wellbeing
• a measure of possible psychiatric morbidity
• experience of mental health problems
• telling others about mental health problems
• the social impact of mental ill-health
• recovery from mental ill-health
• sources of information about mental health problems
• awareness of adverts/promotions about mental health problems
• familiarity with key mental health campaigns, initiatives and promotional activity
• the perceived prevalence of mental ill-health
• attitudes to mental ill-health
• attitudes towards specific symptoms of mental ill-health

2.3 The latter topic area was probed using vignettes depicting people with symptoms associated with depression, schizophrenia and stress. The vignettes were adapted from a study by Link et al (1999) on public recognition of mental illness. Each vignette was constructed to meet the criteria of the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, for the disorder in question and related to a man (called Robert) or a woman (Shona). Respondents were randomly assigned one of the vignettes and asked questions about the person depicted and the symptoms they were displaying. At the end of the section, respondents were asked to say what condition they believed the symptoms described.

2.4 Given that the survey is used to track progress towards specific outcomes, the questionnaire was designed to be comparable with that used in the 2002 and 2004 surveys. However, as in 2004, a number of refinements and additions were made with a view to improving the quality and usefulness of the data gathered. Most notably:

• the wording of the preambles to some questions was clarified
• the Warwick-Edinburgh Mental Well-being Scale ( WEMWBS) ( NHS Health Scotland, University of Warwick and University of Edinburgh, 2006), a battery of questions designed to measure positive mental wellbeing, was added to the self completion section of the survey
• the section on recovery from mental ill health was reworked. Two new questions were added focusing on factors that hinder recovery and messages of recovery received from a) health professionals and b) friends/family. Additionally the wording of several of the pre-codes in existing items was refined
• in the vignettes section, the question concerning whether Robert/Shona should have the same rights as other people was rephrased as it was felt that the original version was probably affected by social desirability bias, that is, the tendency for survey respondents to give answers which they deem to be socially acceptable

2.5 The changes had significant implications for the length of the questionnaire. To keep the interview to the target length of 30 minutes, several questions which had proved to be of limited analytical value in previous surveys were deleted and the demographics section was revised and condensed.

2.6 A full list of all of the changes made to the questionnaire for the 2006 survey is provided in Annex D.

Methodology

2.7 The survey was undertaken among a random sample of the Scottish adult population (including the Highlands and Islands) between 16 October 2006 and 21 January 2007. All interviews were conducted face-to-face by experienced Ipsos MORI interviewers in respondents' homes.

2.8 An advance letter was sent to all sampled households from Ipsos MORI. The letter was printed on Ipsos MORI headed paper, with Scottish Executive logos, and signed by the Managing Director of Ipsos MORI. The letter was designed to provide basic information about the survey but to avoid giving prominence to the issue of mental health. A copy of the advance letter can be found in Annex A.

2.9 Households were sampled as follows:

• the postcode address file ( PAF) was used as the best available source for Scotland's household population. Addresses were sorted into five geographic groups by amalgamating NHS Board areas. These were:
  • Central Belt/West (Greater Glasgow & Clyde, Lanarkshire and Forth Valley);
  • Lothian & Fife;
  • Borders & South (Borders, Ayrshire & Arran, Dumfries and Galloway);
  • Highlands and Islands (Highland, Western Isles, Orkney & Shetland);
  • North East (Tayside, Grampian).
• The target number of interviews was allocated to each area in proportion to the adult population of the area.
• With each area, Output Areas ( OAs) ⁵ were selected with probability proportionate to the population to provide clusters of addresses within which interviews would work. A total of 99 Output Areas were selected.
• Within each of the sampled OAs, 24 addresses were selected at random.
• Interviewers made at least six calls at each sampled address if there were problems making contact, including calls during weekends and evenings.

2.10 All fieldwork was conducted using Computer Assisted Personal Interviewing ( CAPI), where data is collected on laptop computers. The General Health Questionnaire ( GHQ12) and WEMWBS components of the survey were administered using CASI (Computer Assisted Self Interviewing) whereby respondents are invited to enter their responses directly into the CAPI machine. Eighty percent of respondents agreed to complete the CASI module, with interviewers providing assistance to those for whom computer literacy was an issue.

2.11 The target number of interviews for the survey was 1,200 and the total number of addresses allocated was 2,372. In the event, a total of 1,216 interviews were achieved and the response rate was 57%, allowing for invalid addresses. Further information on survey administration is provided in Annex B.

2.12 To assess the extent of any bias in the sample, it is important to consider whether people who did not take part in the survey differed in socio-demographic terms from those who did. At addresses where respondents refused to participate in the survey, interviewers recorded reasons for refusals, as well as a limited amount of socio-demographic characteristics of 'refusers'.

2.13 This data shows that, of those who refused to take part, 30% said they were too busy or they were always busy, while 16% said they never do surveys, 12% said they were put off by the subject matter and 4% said they were just not interested. These findings are consistent with outcome data for the previous waves of the survey.

2.14 Looking at the profile of non-responders, 31% were observed as being from elderly adult households, 18% were from families with children and 28% were classified as belonging to some 'other' type of household. For the remaining refusers, no profile information was gathered by interviewers. Ipsos MORI consistently finds that elderly households are among those most likely to decline to participate, usually because they feel that surveys are not relevant to them or that they are not well enough to take part.

2.15 The tendency for refusals to be higher among some groups than others (and we can only assume that this is the case based on the limited amount of profile data recorded by the interviewers) does not appear to have introduced any notable bias into the sample. There are two main points to note in this respect. Firstly, in spite of the relatively high level of refusals among elderly households, the survey in fact over represents elderly age groups and under-represents younger groups. The data was weighted to reflect this and to bring the achieved sample profile more into line with that of the general population. Secondly, and as the table below illustrates, the weighted profile of the sample is in line with the weighted profiles of other large scale national surveys conducted by Ipsos MORI over recent years, including the 2005 Scottish Household Survey.

Table 2.1: Sample profile

Source: Ipsos MORI

Missing data

2.16 All respondents in the survey were given the opportunity to complete the General Health Questionnaire ( GHQ12) and WEMWBS module. However, due to an error in the survey script a significant proportion of respondents were presented with only the first two questions of the GHQ12. More specifically, of the 973 respondents who agreed to complete the self completion modules, only 460 were presented with the full GHQ12 section, thus resulting in a significant amount of missing data. The survey Research Advisory Group and Ipsos MORI met to discuss the way forward in light of the omission. After reviewing a number of different options it was decided to recommend to relevant heads of division that no ameliorative action should be taken to address the omission. This recommendation was accepted by the Scottish Executive's Deputy Director, Mental Health Division, and Deputy Director, Health Finance Directorate Analytical Services Division. The GHQ12 omission and associated methodological issues are discussed in more detail in Annex H.

Analysis

2.17 Prior to the analysis stage, the data had to be weighted to account for the fact that only one person was interviewed per household. This meant that adults in multi-adult households had a lower chance of participating in the survey than adults in single person households. The data was also weighted by NHS Board, age and gender using 2001 census data.

2.18 For the purposes of analysis, computer tables were prepared to a specification agreed with the Scottish Executive. In the tables, responses to each survey question were analysed against a number of key variables, namely:

• Sex
• Age (8 groups - 16 to 24 years through to 75 years and over)
• Age interlocked with sex (6 groups - males aged 16 to 34 years, 35 to 54 years and 55 years and over; and females aged 16 to 34 years, 35 to 54 years and 55 years and over)
• Ethnicity (2 groups - White and non-White)
• Working status (3 groups - Full time, part time, not working)
• Household income (5 groups - ranging from less than £5,200 per annum to £36,400 or more)
• Ease of managing on income (3 groups - easy, manageable, difficult)
• Area Deprivation (5 bands - most to least deprived based on the Scottish Index of Multiple Deprivation ( SIMD) ⁶ which uses data from a range of sources including, the 2001 census, the Department of Work and Pensions, the Police, the Scottish Executive, local authorities and ISD Scotland)
• Qualifications (4 groups ranging from no qualifications to professional qualifications)
• Urban/rural split (6 groups ranging from most urban to most rural)
• NHS Board Areas (5 groups - Borders and South, Central Belt West, Highlands and Islands, Lothian and Fife, North East)
• Long-standing illness/disability/infirmity (3 groups - Limiting condition, non-limiting, none)
• Experienced a mental health problem (3 groups - A problem of their own, someone close with a problem, no contact at all)
• General Health (2 groups - good and poor)
• Psychiatric morbidity (2 groups - low and high mental ill-health)
• Mental wellbeing (3 groups - poor, average, and good mental wellbeing)

2.19 In addition to these basic cross-tabulations, multivariate analyses were undertaken to explore the strength of relationships between variables. The main types of analysis that were undertaken were regression analysis, segmentation analysis, correlation analysis and the aggregation of responses across different variables. A full description of each technique is presented in Annex F. It is important to note that cross-sectional data generated in this survey can be used to establish correlations between variables but not definitively identify causation.

2.20 Several questions in the survey were asked of only a sub-section of respondents. For example, the questions about recovery from mental ill-health were asked only of those who said they had experienced a mental health problem, which amounted to 384 people. Similarly each of the six vignettes were presented to only around 200 respondents, one sixth of the total sample. There is a limit to the amount of reliable sub-group analysis than can be undertaken on such small samples. To address this issue and maximise the explanatory potential of the survey, data from the 2002, 2004 and 2006 surveys were aggregated, thus tripling the effective sample sizes for questions which have run in all three waves of the survey.

Interpretation of the data

2.21 As noted in the technical appendices to the report (see Annex G) survey respondents represent only a sample of the total population. All survey results are subject to sampling variability which means that observed differences between sub-groups may not always be statistically significant i.e. they may have occurred by chance. Throughout the report, differences between sub-groups are commented upon only where these are statistically significant - i.e. where we can be 95% confident that such a difference has not occurred by chance (p<0.05). The formula used for calculating significant differences and a guide to statistical reliability is appended in Annex G.

2.22 Where percentages do not sum to 100%, this may be due to computer rounding or multiple answers. Throughout the report, an asterisk (*) denotes any value of less than half a percent.

2.23 It is important to note that the findings presented throughout this report are based on what people say about their attitudes towards, and experiences of, mental health problems and related issues. It may be that some respondents have chosen not to reveal particular information, for example, that they have experienced a specific condition or that they hold negative attitudes towards mental ill-health. This point should be borne in mind when interpreting the data.