3. Methods
This section outlines the research methods used in this project and highlights any limitations on the data resulting from these methods. Overall, 88 cognitive interviews were conducted with a broad range of respondents from different ethnic backgrounds, religions, age, sex and living in different parts of Scotland.
3.1 Cognitive Interviewing
Cognitive interviewing is a widely used approach to critically evaluate survey questionnaires. It allows an investigation into the way target audiences understand, mentally process and respond to survey materials. For example, when a questionnaire is designed it is possible that the author may intend one interpretation of a question but find that respondents presented with the question adopt an alternate understanding. If cognitive interviewing is used successfully in testing questions, leading to appropriate findings or insights, survey materials can then be modified to enhance clarity 8.
There are several different techniques that can be used in a cognitive interview. A technique called "retrospective probing" was deemed to be most appropriate for this study. Retrospective probing involves the interviewer presenting a question to be answered, the respondent answering it and the interviewer following up by probing for specific information relevant to the question or to the specific answer given (e.g. What does this question mean in your own words?). This probing can be done immediately after an individual question is asked or after the respondent has completed all the questions. As four reasonably brief questions were being tested in this study, we used retrospective probing after a respondent had completed all the questions.
3.2 The sample
The sample of respondents was designed to reflect the range of ethnic groups in Scotland, to cover a range of ages and to cover both men and women. It was not intended to be statistically representative of the Scottish population.
The sample was designed to include people from the main groups that we anticipated might:
- have trouble working out the meaning or intention of the question
- have difficulty locating a response option that they feel accurately describes them.
We interviewed:
- 30 African/Caribbean/Black respondents. This included 15 African/African Scottish/African British respondents, 13 Caribbean/Caribbean Scottish/Caribbean British respondents and 2 "other Black" respondents;
- 20 Asian/Asian Scottish/Asian British respondents. This included 9 Pakistani respondents, 7 Indians respondents, 1 Mixed Bangladeshi/English respondent 1 Chinese respondent and 2 "other Asian" respondents;
- 10 White Scottish/ British respondents. This included 5 White Scottish respondents and 5 "other White" respondents;
- 7 White African respondents;
- 7 Sikh respondents; and
- 14 Eastern European respondents. This included 8 Polish respondents and 6 respondents from other Eastern European countries: 2 Albanian respondents, 2 Czech respondents, 1 Estonian respondent and 1 Latvian respondent.
3.3 Respondent Selection
Most respondents were recruited from respondents to the Scottish Household Survey ( SHS) 9 who had agreed to the re-contacted for follow-up research for the Scottish Government.
The SHS dataset allowed us to sample respondents on the basis of their ethnicity (as classified by the SHS), religion, age, sex and location. Opt-out letters were sent to all potential respondents, asking them to contact Ipsos MORI if they did not want to be approached to take part in the survey. In the event, none did opt-out and they were then telephoned to arrange a convenient time for interview.
In cases where there were not enough respondents on the SHS database (Africa/Caribbean/Black respondents and Sikh respondents) or where the SHS did not collect the necessary demographic information (e.g. Eastern European respondents and White African respondents) additional respondents were recruited via on street recruitment (in Edinburgh and Glasgow) and 'snowballing' from contacts 10. We did not use community groups for recruitment because we were conscious of the politicised nature of some of the debate around ethnicity labels and wanted, as far as possible, to avoid recruiting individuals who had already been involved in the debate or who might feel they should 'represent' the views of their community group. None of the respondents were personally known to the research team and we were careful to ensure that no more than two respondents were 'snowballed' from the same source.
3.4 Fieldwork
Cognitive testing is a method used to assess the effectiveness of survey instruments in eliciting the required information from respondents. During the cognitive interviews we used a combination of interviewer observation (for example, noting points at which the respondent looked puzzled or confused, where they hesitated, where they seemed to be taking care or where they seemed to skim over or ignore instructions or response categories) and retrospective verbal probing (asking a series of questions after the respondent has completed the survey instrument).
The questions to be tested were provided by the Scottish Government. The questions covered:
- country of birth;
- current religion;
- national identity; and
- ethnic group.
Six slightly different versions of the form were used over the testing period (by 'form' we mean a sub-section of the full Census form with the relevant questions). These are all attached at Annex 1. The difference between the 1 st and 2 nd versions of the form was the addition of a 'Polish' response option in Q7 (ethnic group). All of the interviews with Polish and other Eastern European respondents were conducted after the addition of this response option. Versions 3 to 5 had increasing indentation of the 'other' write-in boxes, in an attempt to solve the problem of respondents thinking that the write-in boxes signalled the end of a question (this issue is discussed in detail in the main findings). The final version tested (Version 6) added letters A to E next to the headings in Q7, in a further attempt to clarify that it was all one question.
Interviews were conducted using a topic guide 11 designed by Ipsos MORI in partnership with the Scottish Government (attached in Annex 2). Most interviews were conducted in respondents' homes. For the convenience of the respondent, several others were conducted in a quiet room in the Ipsos MORI office in central Edinburgh, in a private corner of a café or restaurant, and one was conducted in a private interview room in a Scottish Government office in Glasgow. Interviews lasted around 45 minutes on average. The discussions were digitally recorded and transcribed. Respondents were given £20 in cash to cover any expenses, to acknowledge that they had given up time to take part, and to encourage participation from a wider range of people.
All interviews were conducted in English. This was on the basis that those with little or no English language ability would tend to delegate the completion of the Census form to someone within the household able to complete the form in English, or take it to a community group, spiritual leader or some other proxy able to provide substantial help with the interpretation and comprehension of the form in English.
3.5 A note on the terminology used in the report
One of main aims of the research was to explore perceptions, preferences and definitions in relation to terms used to describe ethnic groups. Inevitably, in writing this report we have had to choose certain terms over others in describing respondents. Our aim in each case has been to be as clear as possible in the context and we have not attempted to be completely consistent. Other than when we are explicitly discussing the findings and implications from the research, our use of any particular term should not be interpreted as a recommendation on terminology. The wording is that of the authors of this report and not the Scottish Government or the General Register Office for Scotland. If our choice of any terms causes offence to readers or to the respondents who helped us with the research, we sincerely apologise.