Scottish Government

CHAPTER TWO METHODOLOGY FOR THE EVALUATION

Rationale for the methods employed

2.1. The evaluation of the Rural Community Gateway ( RCG) utilises 3 main data collection approaches: a desk study, an online survey, and a series of workshops. These were selected as the most appropriate ways in which to gather the information required to address the objectives of the evaluation, since it enabled findings to be gathered from frequent users, as well as new and infrequent users of the RCG. This mixed-method approach is now described in more detail.

The desk study

2.2. The objectives of this were to enable the evaluator to contextualise the focus of the primary data collection, and to locate the RCG within the wider 'community web environment' - both at that particular point in time and looking forward to the next 10-15 years. This was to enable the evaluator to gauge the extent to which there was still a rationale for the RCG's existence, given parallel developments in the world of the web.

2.3. The elements of the desk study were shaped directly by the scope described in the Tender Specification. These related closely to 3 of the 4 evaluation objectives, namely:

a. To evaluate the RCG against set objectives;
b. To locate the RCG in the prevailing community web environment;
c. To explore the future of the RCG in terms of direction and sustainability.

2.4 These objectives were partly addressed in the desk study by focussing on the following themes:

Table 2.1. Relating Desk Study findings to Evaluation Objectives

The online survey

2.5. This survey comprised a questionnaire targeted at users of the RCG, in order to gather information about the users themselves, plus their assessment of the RCG. The types of questions and issues used in the survey were designed specifically to address the 4 RCG evaluation objectives:

a) To assess the extent to which the RCG has met its high-level objectives;
b) To locate the RCG in the prevailing community web environment;
c) To explore possible further developments/improvements to the RCG with regular users and irregular users; and
d) To explore the future of the RCG in terms of direction and sustainability.

2.6. The participants for the questionnaire were self-selecting (in that it was a voluntary option to be involved in the survey). This may have lead to bias towards those who were either particularly computer literate, and/or who wished to reflect their ideas, concerns and/or support for the RCG. This inherent bias was recognised, and was addressed as much as possible through the complementary face-to-face element of the primary research (see below).

Survey questionnaire

2.7. The content of the online questionnaire was developed by the proposer (see Annex 1), and then the formation and technical/uploading aspects were carried out by working jointly with staff from Sift and with the Content Manager. The questions comprised a mixture of open and closed-ended format, to allow for (i) rapid collation of quantitative data through Sift-supported survey export tool, and for (ii) generation of qualitative comments relating to specific sub-themes under the 4 objectives.

2.8. The pilot phase was tested through individually targeted emails leading to a part of the Gateway site for this purpose. The 9 individuals selected for the pilot comprised those who were regular contributors to the Gateway, and thus it was felt that they would have an interest in supporting the development and testing of the online survey tool. The findings from this brief pilot were quickly integrated within the revised questionnaire which went 'live' and was advertised on the front page of the Gateway on Tuesday 10 ^th January 2006.

2.9. The survey ran on the Gateway until Monday February 27 ^th. Reminder emails were sent out twice specifically for the survey. The evaluation and survey were also mentioned in one of the fortnightly bulletins issued from the Gateway to users. A total of 241 responses were received.

Sampling

2.10. Sift sent a weekly cumulative output in Excel spread sheet to the Content Manager, in order for any targeting decisions to be made. As a result, on one occasion, the Content Manager targeted users in specific geographic areas which had been under-represented in the first tranche of survey responses. This 'quota sampling' approach had been anticipated as a possible targeting tool, although in fact was not extensively required. It must be noted that a statistically representative sample was not the key objective of the online survey; rather, the sampling was expected to be 'illustrative' of the spread of member-types. Additionally, non-members were also able to fill in the questionnaire.

The users workshops

Rationale

2.11. The focus of this part of the data collection was upon those who had experience of using the web. It comprised those who were frequent users of the RCG, and those who were infrequent users (first time users, one-off/non-returners). The aim was to gather more in-depth information than was possible from the online survey, particularly in the areas of how to improve the RCG, what its core mission should be, and how it should be sustained.

Sampling, recruitment and levels of attendance

2.12. Accessing frequent users: It was envisaged that those who were frequent users of the RCG would be more willing to be involved in workshops than those who were not, since they were likely to see the value of spending time addressing issues of its future direction and sustainability. It was therefore proposed to hold a single workshop in Inverness, and to offer travel and subsistence remuneration, so that participants from different parts of Scotland would not be discouraged from taking part on the grounds of cost. There were 5 participants in total from 5 different areas of mainland Scotland. Potential participants were selected by the Content Manager, using figures related to the highest 50 users during the 6 months from August to December 2005. Excluded from this selection were those who would be classed as "regular contributors", since they had already participated in a Contributors' Event in early November 2005, and would thus potentially be 'over-surveyed'. In addition, the Scottish Executive was keen to differentiate between 'contributors' and 'users', and thus the focus of the workshop remained firmly on frequent users.

2.13. Accessing infrequent users: By definition, this 'client group' seemed a more difficult one to attract, due potentially to low level of interest, or at least not seeing the RCG as a priority for discussion. It was therefore felt that individuals would be unwilling to travel any great distance to participate in a workshop. Hence, a small number of locally/regionally based events around Scotland were proposed. The areas initially proposed comprised: (i) Scottish Borders; (ii) Western Isles; (iii) South Lanarkshire; (iv) the Angus Glens; and (v) Caithness.

2.14. Coverage of user-types: The range of workshops allowed for a range of potential user-types to be accessed on their home territory. These included the voluntary sector, higher education, rural community learning and development, private enterprise in rural areas, other online communities, community web-designers, and rural development partnerships. In addition, the selected areas represented different types of rurality in Scotland, from remote through to accessible rural.

2.15. Methods of recruitment: The overall success of these location-based events was dependent on the strong links the researcher had with individuals in these specific areas, with some initial ground-work being carried out to ensure that local 'buy-in' existed. Specific workshop dates were then arranged, and the average number of participants was between 8-10 per workshop ³. They were typically held in a meeting room arranged by the local contact, and refreshments were provided for the participants. The discussions were recorded, with anonymity assured for participants in that only the researcher would have access to the primary data/recording.

2.16. Levels of interest and attendance: In 2 instances, there was a low level of response from 2 locations and so a decision was taken to switch one of the workshops to the Isle of Skye, which had the added benefit of enabling the issue of Gaelic language on the RCG to be discussed. It is important to note that when one is trying to research a group of "infrequent users" of any resource, generating a sufficient level of interest in the evaluation is a challenge, and thus a reliance on personal networks at local level remains critical.

Focus of the workshops

Issues discussed at the Participatory Workshops:

2.17. The workshops - for both frequent and infrequent users - were structured in a focus group manner around the 4 objectives of the evaluation, with specific prompts being used to ensure that the themes were interpreted in a similar way across the 6 workshops (see Annex 2). In addition to shaping the discussion around these 3 major themes, participants were also asked to address the following specific questions:

Frequent users of the RCG: possible developments and improvements in the Gateway and their reasons; future directions of the Gateway, again explaining their reasoning. Which ways should the Gateway NOT develop? Finally, how should/could the Gateway be marketed? Were there particular approaches that could be used to improve its relevance to a wider population? Could it link with other publicly funded sites?

Infrequent users of the RCG: why had one-time users stopped using the Gateway? Were there specific characteristics of the Gateway which pushed them away, such as: lack of relevance, lack of interest in its content, difficulty with navigation, use of improper language? How could their experience of the Gateway be improved? Was the Gateway marketed to them? Were there other sites which they used which gave them all the information they needed, e.g. at a local/regional level, or with specific relevance to their tasks or institution? What other sources do they rate particularly highly and why?

2.18. The Evaluator provided the continuity across workshops by facilitating each of the discussions. A thematic analysis of the findings from all 6 workshops was then carried out, which aligned the findings under: the 4 objectives, the specific questions put to frequent and infrequent users, and under emerging sub-themes which came from the participants themselves.

The "mission" workshops

Rationale

2.19. It had been planned, during the proposal stages, to facilitate a discussion with key staff involved in the content management of RCG at SCVO. In addition, the Evaluator was invited to informal discussions in Sift (to combine with the scheduled training session for SCVO staff). Further, the Evaluator was invited to run a workshop with Research and Policy staff at SEERAD, since they had been instrumental in shaping key decisions concerning the development of the RCG over its 3-year lifetime. The discussions with Sift concerned the challenges and strategies related to online community development (which are reported in the desk study elements relating to Best Practice in design). This "triangulation" approach links with the recommendations of Kim (2000), who states that the perspectives of the designers/hosts/commissioning organisations should also be taken into account ( i.e. as well as the views, ideas and concerns of the users) when seeking to evaluate next steps and options for development.

Themes

2.20. Specific themes related to the future development and 'mission' of the RCG were discussed with SCVO. At the workshop with SEERAD, definitions of rural community, the evolution of the RCG's aims over time, and future options for the Gateway, were discussed (see Annex 2). These findings comprise a valuable element of the evaluation, in that they illustrate the management and strategic perspectives of the RCG, thus complementing the users' data from the five workshops and the online survey.

Introduction to reporting of the findings

2.21. It is rare in evaluations of initiatives using Information and Communication Technologies ( ICTs) to be able to see how they have evolved over their lifetime. Typically, ICT projects are evaluated in their 'pilot phase', and as such, everything is ex ante, before the event. Informed speculations are then made as to how the initiative may or may not develop. However, with the evaluation of RCG we had an opportunity to look back to its inception in 2001 (as a recommendation from the SNRP) and pilot phase, then the 3 years of its full operating phase (2003-2006), then to look forward, at least to the next 18 months, and finally beyond. This was invaluable, since we were able to gain feedback from users (both frequent and infrequent) who had experienced the RCG for up to 3 years. In addition, we could gather views and perceptions from those variously involved in its management and delivery - again, from its inception through to the time of writing (May 2006). Within this overall structure, sub-themes focus on key elements of the RCG, including: its functions/roles, delivery, marketing, and degree of contribution from members and users. This report concludes by drawing together the evaluation findings into conclusions and recommendations for the next phase of the RCG, with specific tasks and targets being outlined.

2.22. The findings will therefore be structured around the lifetime of the RCG. This can be elaborated as: Phase 1 (Background: Scoping and Pilot), Phase 2 (Inception, Establishment and Consolidation), and Phase 3 (Business Planning and Next Steps). By reporting in this way, we will be able to see the initial rationale for the RCG, how the RCG has changed over time, the extent to which it was being used and valued, and how it must improve and develop if it were to be sustained beyond its initial three years of operation. The following figure illustrates this structure:

Figure 2.1. A Timeline approach to the Evaluation Report Structure, showing the Phases of the RCG, and the sources of the findings.

2.23. The findings for Phase 1 have been outlined above in Chapter 1. The findings for Phases 2 and 3 are now presented.