Scottish Government

ANNEX H: OMISSION OF GHQ12 QUESTIONS FROM SURVEY SCRIPT FOR PART OF THE FIELDWORK PERIOD

Background to the omission

H.1 The survey questionnaire includes a self completion module comprising two batteries of questions. The first of these is the Warwick-Edinburgh Mental Well-being Scale ( WEMWBS), a measure of positive mental health. The second is the GHQ12 which screens for possible psychiatric morbidity. The WEMWBS battery is a fairly new measure and was included in the survey for the first time in 2006. The GHQ12 was included in the 2004 and 2006 surveys.

H.2 To minimise potential question order effects, the decision was taken to randomise the order in which the two batteries were presented to respondents and include an administrative variable in the survey script which would record the order for each case.

H.3 However, shortly after the start of fieldwork, it came to Ipsos MORI's attention that the administrative variable had been omitted from the CAPI script. To rectify this, the script was withdrawn and a new version created which included the missing variable. As part of this process a test version of the new script was created in which all but the first two items in the GHQ12 battery were removed to speed up the testing process - there was no need to go through all of the GHQ12 questions to establish that the new administrative variable was working properly. Unfortunately, the deleted questions were not then added back into the new script and the incomplete version was released to interviewers.

H.4 Consequently, respondents who were interviewed using the new version of the script were presented with only the first two questions from the GHQ12 component. In terms of sample sizes, this meant that of the 973 respondents who agreed to complete the self-completion module (as usual, a proportion refused this exercise), only 460 were presented with the full GHQ12 section, while 513 people were presented with the first two questions only.

Implications

H.5 In several respects the loss of the GHQ12 data does not affect the extent to which the survey can meet the objectives set for it - for example, to track awareness and understanding of mental health and wellbeing, and attitudes towards people with mental health problems. In other ways, however, the loss is significant. In the 2004 survey the GHQ12 was found to be an important discriminator in many aspects of the study. While results for the individual GHQ12 items were included in the 2004 report, the battery's primary use was in the form of a binary variable constructed from respondents' responses to each of the component items. Depending on their responses to each of the GHQ12 questions, respondents were given a GHQ12 score ranging from zero to 12. These scores were then recoded into a binary variable, 0-3 and 4+. Scores of 0-3 indicate no/few signs of possible mental health problems ('low mental ill-health score'), and scores of 4+ were labelled as indicating possible mental health problems ('high mental ill-health score').

H.6 The number of cases rated as 'high mental ill-health score' in the 2006 survey is 81 which is considerably lower than it would have been had the omission not occurred. The small sample size restrained Ipsos MORI's ability to undertake more detailed analysis of this group.

H.7 The omission also meant that there was a reduced number of cases for an analysis of the way people responded to the two self-completion modules, and the relative predictive powers of the two. However, it remains true that there were sufficient numbers to draw a conclusion about this relationship.

Options for the way forward

H.8 The survey Advisory Group, together with Ipsos MORI, met to discuss the way forward in light of the GHQ12 omission. Four main options were considered. Three of these options focused on ameliorative action, namely, recontacting the respondents concerned, imputation, and conducting the whole survey afresh using a new sample. The fourth option was to take no ameliorative action.

Option 1: Recontacting respondents

H.9 This approach would have involved going back to those respondents who were not presented with the full GHQ12 module, with a view to gathering the missing data and thus plugging the gap. During the assessment of this option, a number of issues were discussed.

H.10 The first issue related to the likely reach of the exercise. As part of the main survey interview, Ipsos MORI obtained permission to return to conduct further research with most of the sample - a requirement under MRS guidelines - and typically around 75% agree to this. Of these people, it was anticipated that successful re-interviews could be achieved with around 60%, depending on the method used and the period devoted to the exercise (see below). This suggested that it would be possible to 'plug the gap' in around 230 cases. Members of the Advisory Group felt that the addition of 230 cases to the existing 460 would not lead to a significant improvement in statistical reliability for any resulting analyses, particularly since the main significance of the GHQ module is its ability to identify a relatively small subset of the sample who exhibit particular characteristics (discussed further below).

H.11 A second key consideration was that the GHQ12 module asks people to report on their feelings 'recently', and this raised issues about the viability of gathering this information over 8 weeks after the main attitudinal data have been gathered. The user manual for the GHQ12 module explicitly says that in 'two stage studies' (which this would effectively become) the GHQ data should be gathered as soon as possible after the rest of the interview "since the GHQ is designed to detect relatively acute changes in state, many of which are short-lived, the expected effect of delay is that a greater proportion of high scoring respondents will be rated as non-cases on subsequent interview, that is, will be regarded as false positives" ²⁴.

H.12 In terms of possible methods of recontacting respondents, 3 options were identified: a self completion, telephone or face to face approach. The main issues discussed with regard to each were as follows:

• Self-completion - contact by post - In this approach, a short booklet would be sent to respondents, incorporating the WEMWBS and GHQ12 questions (with some kind of explanation). These would have had to be individually numbered in order that they could be tied back to the original dataset. Although the questionnaire could be addressed to the relevant member of the household, it would be impossible to ensure that the same person completed it. The response rate would be likely to be fairly low, perhaps 45%-50%, and we might expect differential response rates from the sample, perhaps related to their mental wellbeing, among other factors. Ipsos MORI reckoned that approximately 4 weeks would be required to undertake this work
• Telephone -Ipsos MORI obtained telephone numbers for around 76% of the sample and, anticipated that it could achieve a response rate of around 60% using repeated calls. However, it was anticipated that there would be some difference in the way that people responded to the questions over the telephone (in part because it would no longer be self-completion), compared with other modes. Ipsos MORI estimated that approximately 3 weeks would be required to undertake the exercise by telephone.
• Face-to-face - this method would have provided the closest match to the original study. A short self-completion module could be compiled and multiple calls at sample members' homes could have resulted in the achieval of interviews with around 70% of the original respondents who indicated a willingness to be re-contacted. Ipsos MORI estimated that it would take approximately 6 weeks to prepare for this work and to undertake the fieldwork.

H.13 Of these three methods, the face-to-face approach was the preferred option among the Advisory Group because it was likely to achieve the largest sample and to be the closest match to the original study. However, the Group recognised that the time required to undertake the face to face fieldwork would cause a significant delay in the study timetable.

Option 2: Imputation

H.14 It is not uncommon in complex datasets to have some variables incomplete (perhaps because of item refusal) and to need to 'impute' responses on the basis of other responses in the survey and the way in which other similar respondents have answered questions. Ipsos MORI proposed this as a possible option. However, members of the Advisory Group expressed concern that there was lack of appropriate variables upon which to base the imputation, and suggested the number of missing variables was too large to make imputation feasible.

Option 3: Starting the survey afresh

H.15 The third option considered by the Advisory Group was to start the survey afresh using a new sample. Although this was undoubtedly the 'purest' of the 4 options considered, it would have resulted in a considerable delay to the study timetable.

Option 4: Taking no ameliorative action

H.16 Key to the decision as to whether taking no ameliorative action was viable, was an assessment of the quality of the data that had been collected using the second (incomplete) version of the script. If, upon analysis, the second sample appeared in some way different to the first sample - either in attitudinal or behavioural terms - it would have had to be assumed that there was a non-response bias in the sample which undermines the reliability of the data. To test this, Ipsos MORI conducted a comparative analysis aimed at identifying any significant differences between the key sub-samples in the data. The analysis compared the responses of the two samples on several key variables selected by the Advisory Group, namely:

• WEMWBS (mean scores)
• QB4 and QB5- first two items in the GHQ12 which were presented to all respondents:
  • QB4 asks: Have you recently been able to concentrate on whatever you are doing?
  • QB5 asks: Have you recently lost much sleep over worry?
• QC1 - 9 attitudinal statements (presented after the self-completion section)
• QC3 - Knowing someone who has experienced a mental health problems
• QC5 - Personal experience of mental health problems

H.17 The results showed that there were no significant differences between the samples on the following measures:

• WEMWBS
• QB4 and QB5 - the first two items in the GHQ12
• QC3 Knowing someone who has experienced a mental health problem
• QC5 Personal experience of mental health problems

H.18 And of the 9 attitudinal items ( QC1) there were no significant differences for the following 6 statements:

• The public should be better protected from people with mental health problems
• Anyone can suffer from mental health problems
• People are generally caring and sympathetic to people with mental health problems
• People with mental health problems should have the same rights as anyone else
• People with mental health problems are largely to blame for their own condition
• People with mental health problems are often dangerous

H.19 However, statistically significant differences were evident for the following 3 statements (reference to "positive" and "negative" refers to absence or presence of stigmatising attitudes):

• If I had a mental health problem, I wouldn't want people knowing about it - Those who were not offered the full GHQ12 module are more likely to agree with this statement
• I would find it hard to talk to someone with mental health problems - Those who were not offered the full GHQ12 are more likely to disagree
• The majority of people with mental health problems recover - Those who were not offered the full GHQ12 module are more likely to agree

H.20 As there were no statistically significant differences between the two samples on the bulk of these measures - and that the significant differences that did exist on the three attitude statements were not internally consistent - both Ipsos MORI and the Advisory Group members concluded that the second sample did not display any systematic bias.

H.21 Another key question regarding the quality of the existing data was whether the number of respondents for whom full GHQ12 data has been collected was sufficient to allow for meaningful analyses. As already noted, only 460 people were presented with the full GHQ12 module. While this had some implications for the level of sub-group analysis that can be conducted on the data, the number of cases at the aggregate level was sufficient to compare the views of those with 'high' and 'low mental ill-health score' and for a comparative analysis of responses to the GHQ12 and WEMWBS.

Decision on the way forward and summary of rationale

H.22 On the basis of the various considerations set out above, it was decided to recommend to relevant Deputy Directors at the Scottish Executive that no ameliorative action be taken to address the omission. The over-arching rationale for this was that each of the alternative options was felt to carry significant disadvantages which would outweigh the problem of the missing data. Additionally, the existing data was felt by both the Advisory Group and Ipsos MORI to be of sufficiently high quality as to provide reliable GHQ12 data, amenable to aggregate level analysis. The recommendation was accepted.