Scottish Government

CHAPTER TWO: METHODOLOGY

INTRODUCTION

2.1 The choice of methodology for this research was driven by the need to establish a set of priorities for service improvement which communities themselves consider to be important. As a starting point a literature review was conducted to identify existing information on community needs and views with respect to service provision in the Rural Services Priority Areas. This review was designed both to provide information to be used alongside the primary research findings in informing the discussions between the Scottish Executive and Community Planning Partners, as well as to assist in the direction and shape of the primary research. The primary research was intended to complement the literature review work, by, amongst other things, exploring the meaning of the terms 'accessibility' and 'quality' in relation to service use and provision in more depth. Qualitative focus group discussions were selected as the most useful approach as they would allow for both the expression of a range of individual views, as well as for discussion of, and where possible agreement on, the key issues and priorities for the community. The researchers sought guidance from Community Planning Partnerships on practical elements of the literature review and the conduct of the groups. This chapter describes the rationale behind the approach adopted and the process of conducting the project.

LITERATURE REVIEW

Aims and objectives

2.2 The roles of the literature review were as follows:

1. To review the findings of previous research and consultation exercises which had addressed issues of service accessibility, quality or priorities, as carried out within the 22 Rural Services Priority Areas and their associated Community Planning Partnership areas

2. To provide a summary of the findings of this review - including the different methodologies adopted in the previous consultation and research exercises - for use in informing the discussions between the Scottish Executive and Community Planning Partners on target setting in the RSPAs.

3. To support the group discussions through:

• informing the design and content of the topic guide
• identifying any gaps in existing knowledge which needed to be addressed
• providing supporting material (in the form of showcards) to be used, if necessary, to help generate or further discussion on a range of topics. These included a comprehensive list of possible services for consideration and examples of innovative services introduced elsewhere to help prompt ideas on innovative services which could be introduced to improve priority services in their areas (it should be noted that there proved to be little or no need for these showcards within the groups, as the respondents were able to discuss the issues in question without the need for prompting).

Methods

2.3 An internet search was conducted during April, May and June 2005 of local service delivery agents and other relevant private, public and voluntary organisations in the selected Rural Services Priority Areas for any research and publications relating to service priority, accessibility and quality. This yielded a substantial amount of material on completed and current consultation and research exercises, including the methods used and the results, although there was much variation between the different areas in terms of the quantity and depth of information available. The Internet search also provided details of the range of mechanisms through which agencies and residents/communities communicate with each other, as well as examples of innovative projects relevant to service provision.

2.4 The internet search was followed up by letters to all 8 CPPs requesting 'information on all completed and current research and/or consultation exercises carried out in the area - with all age groups from youth to elderly - that were relevant to the issue of service priority, accessibility and quality', including public, private and voluntary services. They were asked to provide as much information as possible on 'the nature of the consultation and particularly the precise methods used to gather the information' and 'the results of the consultation exercises'. They were also asked to provide examples of 'good practice /innovative initiatives in relation to service provision'.

2.5 The letter was then followed up with direct contact with all the CPPs by telephone and/or email, and in one case through a visit (Orkney). The purpose of this follow-up contact was either to gather information where none had been sent in response to the letter, or to get further details on information received or accessed via the web.

2.6 The final element of the literature review comprised a web-based and bibliographic search for general literature dealing with service provision in rural areas, including academic journal articles and books, policy documents and government reports, and publications of charitable organisations.

GROUP DISCUSSION

2.7 This section of the chapter describes all aspects of the primary research - that of the group discussion - element of the project, including:

• geographic coverage, location and timing of groups
• target audience
• recruitment
• incentives
• group discussion methodology

• pre group task
• workshops
• conducting groups in Gaelic
• analysis

Number, geographic coverage, location and timing of groups

2.8 Thirteen group discussions were held, covering 20 of the 22 Rural Services Priority Areas. The workshop locations and the RSPAs they covered are listed in table 2.1 below.

2.9 The geographic coverage and location of the workshops was decided in consultation with representatives of the relevant Community Planning Partnerships. Originally the workshops were to cover all 22 RSPAs, but discussions with the Highland Community Planning Partnership suggested that sufficient research and consultation had already been conducted in Tongue & Farr and Brora.

2.10 Coverage of the 20 RSPAs by the 13 workshops was achieved on the basis of logical geographic groupings. The venues chosen for the workshops were either hotels or community centres, which, as far as possible, were located in a central location within the RSPAs, unless a suitable location was not available, or to facilitate travel.

2.11 On the advice of the Community Planning Partnership representative for Eilean Siar, the planned single workshop covering the geographically disconnected Harris East, Harris West and Uig RSPAs was split into two mini groups within each of the respective communities. This advice was given on the basis that the two communities were likely to have very different priorities. It also overcame the need for very long journeys and overnight accommodation for some if a single group was conducted. The second focus group workshop in Argyll and Bute was held in a video conferencing facility in Bowmore on Islay. This was to facilitate 2 attendees from Colonsay who participated in the workshop via a video link. This was found to be the most effective means of enabling participation from this remote island community.

2.12 The discussions were held in the evening to ensure most people in employment had the opportunity to participate.

2.13 Table 2.1 below shows workshop dates and venues.

Table 2.1 Workshop catchment areas, dates held and locations

Target audience

2.14 The Scottish Executive's Mainstreaming Equalities policy was particularly relevant in this research and the project used it as a guide to choosing the target audience and finding ways of reaching them.

2.15 The target audience consisted of a cross-section of the population of the RSPAs, including 'excluded' and 'hard-to-reach' groups. The workshops were populated based on profiling each recruitment catchment area using 2001 Census data provided by the Scottish Executive. The focus groups could not, of course, due to their size, include a representative sample of the local communities. The aim was, though, to ensure, as far as possible, that each workshop had representation from the key demographic and socio-economic groups in each workshop catchment area. These were:

• age
• gender
• socio economic grouping
• household structure (lifestage)
• disabilities
• ethnicity
• car access
• Gaelic speaking
• local resident vs. incomer

2.16 There were some limitations to achieving an exact profile in each group, these being:

• the number of participants in a group discussion: it is standard for a maximum of 10-12 attendees to be recruited for a group discussion with the aim of 6-8 attending (although it was hoped that the level of interest in this study might result in higher than average attendance, which proved to be the case in most instances); this is in order to ensure that the group can be easily managed by the moderator and that there is sufficient time for all attendees to be able to contribute fully to the discussion. As will be evident, it is not possible in a group of 12 or less to achieve the exact proportional representation of the 10 socio-demographic indicators used to profile each of the research catchment areas, rather the best fit possible.
• the postal recruitment approach adopted: this recruitment approach had the key advantage of ensuring that every resident in the designated areas was invited to take part in the research (an essential requirement of this project). However, it meant that attendees had to be selected from amongst those that responded to the invitation expressing an interest in participating. In some instances the available pool of respondents from which to draw the attendees was relatively low.
• the above was exacerbated by actual attendance: as noted previously, it is usual for final attendance at group discussions to be in the region of 6-8 attendees, as some allowance for drop out in a voluntary exercise such as this is always necessary (due to unexpected commitments arising for the attendees or simply a change of heart about attending). Recruits are always asked to inform the research company if they will be unable to attend, and where this occurred, others who had expressed an interest in attending were invited to replace them; however, it was not always possible to replace like with like so sometimes the replacement was the closest fit rather than identical match. In some instances however (albeit fewer than usual in this research, i.e. attendance levels were much higher than average), recruits simply failed to attend on the night, so that it was not possible to make any attempt to replace them, thereby resulting in a gap in the 'ideal' profile.

2.17 The representation achieved at the groups is summarised in table 2.2 below. There was under-representation of 18 to 24 year olds, which is traditionally a very hard group to recruit in social/market research. However, when considering the final profile achieved in each group, it is important to note that the research approach adopted was designed to anticipate and compensate for the profiling limitations described above. One of the moderators' tasks in the workshops was to ensure the views of all sections of the community were aired, regardless of the level of representation. In addition, group attendees were invited to bring the views of other members of their family or the community along with them for discussion, as part of a pre-group task (see paragraph 2.33 below). The group findings demonstrate the success of this approach, as many service priorities discussed are ones of importance to sectors of the populations who were either poorly represented or not represented at all within the actual group itself.

Recruitment

2.18 12 attendees were recruited for each workshop with the aim of getting 8 to 10 participants on the evening of the event (as noted above, average attendance at group discussions is 6-8 attendees, but it was hoped that high levels of interest might result in higher than average attendance). For the Uig and Tarbet groups 8 people were recruited with the aim of getting at least 6 participants. As table 2.2 shows, the actual attendance for most groups was high, ranging from 7 to 12 participants, but in most cases being 10 or more (or 7 for each of the two mini groups).

Table 2.2 Socio-Demographic Profile of Workshop Attendees

2.19 As the recruitment needed to be as unbiased and inclusive as possible, participants were recruited by post. This ensured that all households were targeted rather than only those with a telephone or where the respondent is at home at the scheduled recruitment time. A disadvantage of this approach was the potential for a very low response, or complete non-response, from hard to reach groups such as the younger 18-24 year old age group. However, this was outweighed by the benefits of the approach with respect to this specific study, particularly given that measures were put in place to compensate for this, as described in paragraph 2.16.

2.20 An invitation was posted to every possible household in the RSPAs using addresses sourced from the Royal Mail's Postal Address File. This is the UK's most comprehensive source of address information. The Scottish Executive supplied the exact postcodes of the RSPAs to allow targeting of the mailshot.

2.21 As it was not possible to personally address each letter, which potentially may have reduced the chances of recipients reading them, the outside of the envelope included strong messages to raise interest in opening the invitation. These were:

"This is a genuine opportunity for you to voice your opinions on services in your local community"

and

"Important information from the Scottish Executive"

2.22 The letter of invitation covered:

• the name of the research agency
• the purpose of the research
• the methodology
• the venue for the group
• a pre-group exercise
• incentives available
• encouragement for the recipient to pass on the information to other members of the household if they are not interested or able to participate

Those interested were asked to complete and return a short questionnaire to enable the researchers to ensure that the group profile was as diverse and inclusive as possible. The questions sought information to match respondents against the census profiles of each workshop catchment area.

2.23 As the perceptions of current service priority, quality and access may be affected by time lived in the area and indeed by previous residence and experience of service provision elsewhere, the questionnaire also included a question to enable a representation of a mix of long term and more recent residents in the groups. A working definition of residing for under or over 5 years was chosen.

2.24 As the research also sought to obtain the views of people less motivated to participate in research, the recruitment questionnaire was designed to ensure people with a strong interest in the issues were not over represented in the workshops. The questionnaire sought to identify these individuals by asking whether the respondent was a councillor or a community councillor and to name any campaigning organisations that they might be a member of. Respondents fitting these criteria were not excluded if they also fitted other recruitment criteria where the pool of respondents was too small to ensure representation otherwise.

2.25 The questionnaire also stated that all information provided would be treated in confidence in keeping with Market Research Society guidelines.

2.26 The respondents were asked to send the questionnaire back to the researchers in a pre-paid envelope, along with their contact details. The respondents chosen to be included in the research were then contacted by phone to invite them to the group and to confirm travel, accommodation, any special needs and dietary requirements.

2.27 Everyone who expressed an interest in attending the groups was sent a reply, irrespective of whether or not they were invited to take part in the research. Those who were invited were thanked for their response and given joining instructions, travel tickets and hotel arrangements if needed. They were also sent a pre-group exercise to be undertaken in the week leading up to the workshop (see pre group exercise in paragraph 2.33 below). If they were not selected they were thanked for their response and the letter explained that, due to the level of interest in the research, they would not be invited to attend on that occasion.

2.28 The research was also publicised on the Rural Community Gateway website. The researchers also wrote to Community Planning Partnership contacts provided by the Scottish Executive asking them to publicise the research through their network of contacts.

Recruitment incentives

2.29 To compensate for the heavy investment in time and travel that would be encountered by many of these attendees, potential participants were offered a financial incentive and reimbursement for travel expenses, and, where necessary, overnight accommodation and breakfast for those who had to travel particularly long distances.

Group research methodology

2.30 In order to inform the setting of local service access and quality improvement targets, the methodology sought to establish the following:

• a realistic set of private, public and voluntary services that are the priority for residents
• a realistic level of preferred access to these services
• a realistic level of quality expected of those services

2.31 It was clearly important that the articulated needs were realistic, so that the broad targets set for local services are achievable. Consequently the methodology sought to focus attention on realistic service needs and improvements in a number of ways, in particular:

• asking participants to undertake a pre-group exercise involving the recording of their actual experiences of service in the week prior to the group, for discussion within the group.
• use of the pyramid projective technique in which respondents are asked to state what would simply "meet expectations" as well as what would "exceed expectations" or "delight" them.

Pre-group exercise

2.32 Along with the confirmation letter, participants were sent a diary to complete for the 7-day period leading up to the workshop. In this they were asked to record their experiences of using or wanting to use services. This was to ensure the subsequent workshop exercises were based on real events and needs. The diary sought information on the following:

• the services used that week
• views on how easy or hard it was to access the service(s)
• what would have made access easier
• level of satisfaction with the quality of the service(s) and why
• improvements they would like to see to the service used
• need for services not used because they were not available
• reasons why the service was not available

2.33 In order to broaden the range of issues potentially discussed at the workshop, and to compensate for any potential under-representation of certain sections of the population at the groups, each participant was sent three extra copies of the diary and asked to give them to other members of their family, or other people that they knew. The participant was asked to ask these people to complete the diary and give it back to them before the workshop. The participants were asked to read them and, if relevant ( i.e. where the views, experiences or needs differed from their own), to present the views of others in the group discussion ( e.g. a mother representing the experiences of her teenage son). However, importantly, any third party views expressed in the groups were presented as such. An example of the diary is included as Annex 2. The diaries were used purely as a prompt to remind participants of any pertinent issues, they were not formally analysed as part of the research. The diaries were collected and are held by the researchers.

Workshops

2.34 The scale of the task of the research required to produce usable results was quite large and consequently 2 hour workshops were undertaken. The workshops were designed to explore the range of individual views, as well as seek consensus on the three key core elements of the study. They were also sensitive to identifying any variation in views by different segments of the groups (eg by age, sex, occupation or location).

2.35 In addition, as participants were asked to consider the views of the wider community in addition to their own, (in part by considering the diaries they distributed to friends and family), the methodology sought to obtain the views and experiences of as wide a range of RSPA residents as possible.

2.36 The workshops were facilitated by skilled moderators and supported with a topic guide (Annex 3). The show material consisted of an extensive list of services and examples of innovative service delivery mechanisms used elsewhere that were derived from the literature review. Their function was to act as a prompt in the event that the group members had difficulty in recalling services or to encourage discussion on innovative ideas (see Annex 4). In the event, these were not required for these purposes, as group participants discussed a very broad range of services and desired delivery approaches unaided.

2.37 The format of the workshop consisted of three stages which were as follows:

Stage 1 - Agreeing a Range of Priority Services for Improvement

2.38 The first task was to establish an agreed range of private, public and voluntary services that the group believed that it was most important to improve for their community. This was achieved through the following process:

• the starting point was to encourage participants to consider as many services as possible, including private, public and voluntary services. This involved drawing up a comprehensive list of services that participants felt were important for them and their community, including both those that were and were not currently available.
• the group was then asked to prioritise the 10 services that they most wanted to see improved in access and/or quality terms. The list could include the provision of new services or improvements to the provision of existing ones. This is key to the approach of this research, which encouraged participants to decide amongst themselves the services that were important to them, and not be limited to choosing from a pre-determined list. The value of this approach was demonstrated by the way that some of the services identified are not amongst those usually included in the list of services that views are sought on in government surveys ( e.g. local employment opportunities, pest control, fuel prices etc). Agreeing priorities represented a significant task for the groups given the range of services people used, or might wish to use. Despite this, generally the groups encountered little difficulty in reaching agreement, even where the service in question was only used by one attendee ( e.g. the pier at North Ronaldsay). This is a further demonstration of the success of the approach in exploring community, rather than solely individual, needs. However, one notable exception was in the Daliburgh workshop. The group consisted of residents of Barra and South Uist and it was clear that the community needs differed with respect to the kinds of leisure services that were a priority and the need for a youth drop in centre. The outcome was that agreement was reached on 9 priorities and not 10 as in the other workshops.

Stage 2 - Considering Access and quality

2.39 Perceptions of the accessibility and quality of the services identified and agreed were discussed in relation to a satisfaction pyramid as shown in figure 2.1 below. A prior exercise was conducted to encourage the group to think about what access and quality meant to them. This was to ensure the pyramid exercise was conducted in the context of as full a range as possible access and quality scenarios.

Figure 2.1: Access and quality satisfaction pyramid

2.40 Participants were asked to assess the access and quality of their top 10 priority services for improvement in relation to the 4 levels:

• the lowest level was 'below expectations', which would mean that the current access or quality was unacceptable in their eyes
• the next level up was 'meets expectation', which would mean that access and quality was acceptable
• the next level up was 'exceeds expectations', which would mean that individuals were more than satisfied with the service provision
• the top level was 'delight', which would mean that access and quality were beyond what they would expect and would result in particularly high levels of satisfaction

2.41 Participants were asked to use this model to assess current access and quality, where applicable ( i.e. where the priority service improvement under discussion was for an existing service rather than a desired new service), to state what would be required to "meet expectations" for new services and then - for both existing and new services - what improvements would be required to reach higher levels of satisfaction. This exercise not only helped to identify what improvements were desired, but by distinguishing between those that would "meet", "exceed" or "delight", helped focus participants on "realistic" expectations as well as on more aspirational ones.

2.42 If necessary (which rarely proved to be the case), the exercise was prompted by considering different delivery mechanisms, as identified in the literature review, which might enhance access and quality. This exercise also involved an exploration of participants' own ideas on innovative or alternative ways of improving access, as well as their views on co-location of services as a solution to problems of access.

2.43 The workshop attendees were asked to give specific consideration to co-location of services as a solution to access.

Stage 3 - Channels of communication

2.44 The aim of this part of the workshop was to explore participants' awareness of existing channels of communication with respect to the provision and delivery of services in their community (with knowledge of Community Planning Partnerships, Local Authority representatives and individual service providers explored), attitudes towards them and how they felt these could be improved. It also explored what their ideal methods of communication would be and how they felt this would differ for different segments of the community.

CONDUCTING THE RESEARCH IN GAELIC

2.45 Again, being sensitive to equality issues, Gaelic speakers were available in all four of the groups which were held within Gaelic speaking communities (Uig, Tarbet, Dunvegan and Daliburgh) to assist, if necessary, with any language issues.

2.46 The final report has also been produced in Gaelic as well as in English.

ANALYSIS

2.47 All workshop findings were summarised by each of the moderators for use by the lead qualitative researcher in the final analysis. Eleven of the group discussions were also audiotaped and transcribed to assist in the final analysis of findings. In the remaining two groups (those held in Uig and Tarbert), there was a tape recorder failure so the findings were recorded as "live notes" by the college student assisting the moderator in each case, and these were used in the final analysis in place of transcriptions.

2.48 The qualitative findings were analysed by the lead qualitative executive. The analysis involved the following processes:

• carrying out a content analysis of the tapes and notes made during each interview or group
• combining the information from each workshop and cross-checking the responses across the groups in each area
• discussing the results with the project manager and director
• exploring and identifying patterns within or between workshop and social groups
• synthesising the data ( i.e. bringing it all together) for the final presentation and report.