Scottish Government

CHAPTER 6 RESEARCH TRAINING AND SUPPORT

6.1 This chapter reviews the support mechanisms put in place to enable groups to carry out agreed research activities, and to manage and monitor the delivery of these. It examines the arrangements for research: training; planning; mentoring; networking; contract management; monitoring; dissemination; and aftercare.

6.2 The analysis presented draws on documentation and data contained in Scottish Executive programme files, supplemented by information provided in one-to-one interviews with leaders from 20 of the 21 completed Rural Voices Projects, and with representatives from the programme Secretariat and Advisory Group.

Arrangements for research training, support and grant administration

6.3 The design of Rural Voices explicitly recognised the need for appropriate measures to equip groups with the confidence and skills to implement research plans by themselves.

6.4 The arrangements put in place appear to have acknowledged the prior experience and capabilities of participating groups. This is confirmed by one Advisory Group member who commented that arrangements were designed on the basis of an "assumption of competence" on the part of participants, which seems at odds with the proposed focus on "novice communities".

6.5 The principal form of training and support was offered through 2 residential "training workshops" in each Round. Four workshops were held in total:

• Round One, Workshop One: Boat of Garten (31 March - 01 April 2003)
• Round One, Workshop Two: Oban (31 October - 2 November 2003)
• Round Two, Workshop One: Nairn (13 May - 14 May 2004)
• Round Two, Workshop Two: Boat of Garten (02 December - 04 December 2004)

6.6 Workshop activity was supplemented by informal "facilitation" from Advisory Group members, who provided a source of ongoing guidance and support.

6.7 Training and support arrangements covered the period from grant approval to the completion of agreed research contracts.

6.8 The research activities undertaken by participating groups were managed and monitored by the programme Secretariat against an agreed research contract.

6.9 In the following sections the above arrangements for research training, support, and grant administration are examined in more detail.

Research training

6.10 The application guidance notes for Rural Voices stated that successful applicants would meet at a "training conference" where they would be "provided with training on action research". Attendance at these workshops was a compulsory part of participation in the programme, and was supported by overnight accommodation, meals and travel expenses.

6.11 The formal training input to these 2 and 3 day residential workshops was extremely limited. In both Rounds One and Two of the initiative, the formal "training" input encompassed only 5 short presentations, which were delivered during a 1.5-hour slot on the first day of the event. These presentations provided only a short general introduction to various research techniques. An additional short presentation on "Budgeting for Your Project" was also incorporated into the workshops during both rounds.

6.12 Presentations and event facilitation was undertaken primarily by Advisory Group and Secretariat members in Round One, and exclusively by them in Round Two. This indicates a strong willingness on the part of voluntary Advisory Group members to give of their time and skills.

6.13 It appears that this input, combined with dialogue with facilitators, was helpful in introducing participants to research techniques. When asked "Was your group given information on the possible research/consultation techniques that might be of use in your project?" all of the 20 project leaders interviewed were content that this was the case. Indeed, interviewees commented positively on the helpful "guidance and examples of what could be used" and highlighted information received on "all sorts of techniques".

6.14 However, when asked "Did your group receive any training on how to carry out your research/consultation?", only 11 of the 20 interviewees reported that they did. These referred to training in the use of SNAP survey software, which was introduced in Round Two, and which did not cover how to carry out research. During discussions, 7 project leaders interviewed reported positively that this software had been used to assist them to analyse the quantitative aspects of their research. However, given delays in organising the training it appears that this was delivered late in the process. In this respect, one project leader commented that "it would have been more useful early on".

6.15 The introduction to research methods and techniques appears to have whetted the appetite of groups. One Secretariat member commented, "A lot of groups expressed an interest in receiving more training". A further training session on research techniques and data analysis was designed and was to be delivered during Round Two. Reportedly, however, this did not go ahead as groups could not commit to training because of the time and travel involved.

6.16 From consultations with project leaders there is evidence that only one group accessed further external training to equip participating volunteer researchers to undertake activities. This included a 3-day residential weekend that provided training for support staff and participating young people in Participatory Appraisal techniques. This was accessed through the local Council, and is described positively in the final report of the project as having provided a highly effective platform to deliver a successful action research project.

6.17 Overall, the measures introduced within Rural Voices appear to fall some way short of thorough research training that might be required to equip groups and their volunteers to use appropriate research techniques. Moreover, there is no evidence that groups were given encouragement to ensure that training was organised outside of the scheduled workshops.

Research planning

6.18 The first training workshop in each round provided the opportunity for successful applicants to revise and strengthen research proposals, with facilitation support available from Advisory Group and Secretariat members through surgeries and one-to-one discussions.

6.19 This iterative process of planning and reflection is consistent with the ethos of action research and represents a forward thinking and "engaged" process of grant making by the Scottish Executive. As one Advisory Group member put it, this process offered a key opportunity aimed at "turning applications into viable research proposals".

6.20 Project leaders regarded as positive this process of revising proposals. In particular the input of facilitators was welcomed, as acknowledged by one project leader who commented that they "provided a useful sounding board for ideas". Advisory Group members confirmed this, one of whom commented "most groups seemed to get a lot from the process".

6.21 In all cases groups came to the workshop with a pre-determined idea of the research/consultation techniques to be used, which was described in project applications. The comment of one Secretariat staff member that groups "all wanted to do surveys", reflects a more general concern that groups were proposing to use a narrow range of traditional and familiar research methods. The relatively fixed nature of proposals was commented on by a number of project leaders, one of whom noted "we had already decided to do a survey and got a lot of support from [the] facilitator on ideas of how to do this".

6.22 Final decisions on the choice of research/consultation techniques were made shortly after the workshop, drawing on the advice provided, and applications resubmitted as appropriate. One Secretariat staff member commented "facilitators pointed out options and guided them, but it was the groups who made the final decision".

6.23 Despite a general introduction to various techniques and dialogue with facilitators, the evidence points to refinement rather than wholesale change as a result of this process. In only 2 instances have project leaders reported a significant shift in approach as a result of planning and discussion at the workshop. In one case, changes in techniques were introduced and in the other a changing of focus and broadening of the research is reported.

Research mentoring and support

6.24 The Outline Proposal for Rural Voices did not set out explicitly whether or how any mentoring or facilitation support was to be provided and arrangements for doing so do not appear to have been explicitly incorporated into the initiative.

6.25 In the absence of alternative arrangements, the Advisory Group assumed the role of voluntary research-enablers or facilitators for participating groups, acting as sounding boards, advisors, and friendly critics. They fulfilled this role primarily through project visits but also through infrequent contact.

6.26 Advisory Group and Secretariat members suggest that groups were allocated a facilitator on the basis of: the match between the research method chosen and relevant expertise of the participating Advisory Group member; and the affinity that Advisory Group members developed with projects during the first workshop. This appears to have offered the basis of a strong mentoring relationship. However, concerns have been raised regarding the consistency and continuity of the mentoring arrangements. In reality, it seems that access to facilitators was limited by the availability and proximity of the small pool of Advisory Group members.

6.27 There is also concern regarding consistency of access to external facilitators. One Secretariat member reports that only "some projects were allowed to use them", and that in one case "being banned from using external facilitators strengthened their project as they decided to do it themselves".

6.28 While drawing on an inevitably limited pool of time and expertise, the Advisory Group facilitators were skilled in a range of professions. However, not all of the facilitators appear to have been equally experienced in offering research advice and support to groups. As one Advisory Group member acknowledges "I felt I offered wrong advice on a couple of occasions". Another commented however, that they "developed skills over time to advise groups, if didn't have the skills already".

6.29 The principal form of facilitation and support occurred through project visits. These appear to have been organised in an ad hoc manner, with limited clarity over role or purpose. One Advisory Group member describes these as "nothing more than an informal blether", while project leaders variously describe these as being about "checking progress", offering "support and guidance" and "addressing queries". There is no evidence that facilitation visits were distinguishable from monitoring visits nor that they adhered to, or were framed by, any explicit guidance.

6.30 Participating groups, however, generally regard the mentoring/support visits by Advisory Group facilitators and Secretariat staff as positive. This is evidenced by a range of comments from project leaders who highlight the supportive role played:

"You knew you could ask if you had any problems and help was there if needed."

"…showed a genuine interest in our work."

"Very encouraging."

"…offered access to expertise."

"…it provided practical help."

"…reinforced that we were doing ok and rubber stamped our progress."

"…provided little bits of advice on what we had done and what we were planning to do."

"…provided help throughout all stages."

"…clarified our position, reassured us even though we knew we would not deliver on time."

"…kept channels of communication open."

6.31 Only a small number of criticisms have arisen regarding the value of these visits to projects. For example, one project leader commented that it "was more for their benefit than ours", while another noted that "we needed someone who understood the project".

6.32 Overall, while welcomed by participating groups, the above evidence suggests that mentoring and facilitation arrangements could have been strengthened in several respects. They could have been: more clearly based around group requirements, rather than facilitator availability; more structured, and more clearly guided; and could have drawn on a wider pool of expertise and skills. These are arrangements that should have been formally built into the programme from the outset.

Research networking

6.33 As part of the Rural Voices experience, participating groups got the opportunity to come together and share experience during the 2 residential workshops. This represents a significant feature of the support arrangements, potentially important in enabling small isolated rural communities to tap into wider networks.

6.34 While the first workshop was designed primarily to equip groups to embark on their research activities, and the second to progress learning and action, both offered participants the scope to network, to share experiences, and to receive support from peers. Opportunities were specifically built in to each workshop for participants to present to others their plans and to share their challenges.

6.35 Based on a review of written post-event feedback, the workshops proved to be positive and enjoyable experiences that built relationships and reinforced the learning from the Rural Voices practice. This is widely confirmed by Secretariat and Advisory Group members, illustrated by one Secretariat staff member who commented on the "very positive atmosphere and experience for everyone".

6.36 On the whole, the feedback provided in post-event evaluation forms completed by participating groups points to a highly satisfactory, valued and effective series of events. Among the positive aspects of the workshops reported by participants were, the:

• efficient organisation and effective structure of the events
• opportunity to network and share experiences with other groups
• value of facilitated sessions in clarifying and resolving issues
• quality of conference and hotel facilities used
• effectiveness of the study tours in demonstrating research in practice

6.37 The popularity of the workshops was such that participants were supportive of the need for a third workshop to conclude their participation in the programme. This was requested by 17 of the 20 participants in the second workshop in Round Two, on prompting.

6.38 Despite a highly satisfactory experience, a small number of operational weaknesses were noted by participants, including:

• insufficient time to take in all scheduled surgery sessions
• the inaccessibility of the venues to some participants
• the lengthy duration of some sessions
• inattention to participant requirements (e.g. dietary)
• a limited time available for networking and informal discussion
• a full itinerary which made it difficult to absorb the information provided

6.39 The feedback on workshops attended in each round is similar in content. This reflects the common structure in both rounds and a consistently high level of satisfaction.

6.40 As gatherings and networking experiences, the events appear to have been of a high quality and provided the basis for effective networking between participants.

Research contract management and payment

6.41 A research contract was agreed between the Scottish Executive and each of the assisted groups based on approved applications. This provided a framework within which activities would be managed and payments made.

6.42 Advisory Group and Secretariat members have reported that contracts provided an appropriate mechanism to manage and monitor the delivery of agreed research activities. One Secretariat member commented that "it was a good idea to treat projects as equals, as capable researchers by giving them a contract".

6.43 Likewise participating groups report satisfaction with the contract management process. When asked "was the research contract/agreement that was put in place to guide your work clear and appropriate?" 19 of 20 project leaders interviewed indicated that it was. Only one project representative suggested that this was not the case, commenting on the inappropriateness of the "contractor status" of participating groups.

6.44 During the course of this evaluation, however, particular concerns have emerged regarding the robustness of contract management and grant administration procedures.

6.45 Concerns were raised over the professionalism of financial administration procedures. Four groups reported particular difficulties in receiving funds awarded: delays in receiving funds (up to 3 months in one case); cash flow difficulties arising from delayed or retrospective payments; payment into the wrong bank account. Secretariat staff have also confirmed that unnecessary requirements were placed on groups regarding the submission of receipts.

6.46 Concerns have also been raised regarding the consistency of payment procedures. Interviews with Secretariat members have confirmed that payments were often not made in line with agreed payment schedules (some staged payments were only paid in part or all together at once) and that payments were, on occasions, made before the completion of agreed research activities.

6.47 Additionally, issues have been raised regarding the robustness and consistency of contract design and management procedures. This issue was picked up by one Secretariat member who commented:

"The contracts were drawn up inconsistently and had little rhyme or reasons to their content. This was a major problem in managing projects."

6.48 Worries have been raised regarding the stringency with which the financial conditions of grants were applied. There is no evidence that unused grant funding was systematically returned to the Scottish Executive. Indeed, as one project commented in relation to an amount of unspent monies "the Scottish Executive didn't ask about this so we assumed that there was no problem with this".

6.49 Issues have also been raised regarding the extent to which contract and financial variations have been documented. From a review of project files it appears that contract variations, modifications to payment arrangements, and the virement of funds between budget headings were not formally and systematically documented as should be the case with research, or indeed any, contracts.

6.50 While some degree of flexibility is to be expected to ensure that contracts reflect the cyclical or evolving nature of the community action research process, it is vital that any flexibility is applied consistently, and that changes are adequately documented.

Research project monitoring

6.51 The monitoring of approved research activities, at a project and programme level, was principally the responsibility of the programme Secretariat and appears to have incorporated a number of main strands.

6.52 Projects were required to submit one progress report during implementation and a final report on completion. No consistent format was provided for the progress reports, and there is no evidence that these were systematically gathered or documented. A final report was produced to an agreed template by all completed projects, which provided the projects with an opportunity to reflect on the experience of Rural Voices and outline research findings.

6.53 The Secretariat maintained ongoing contact with assisted projects, by telephone and email. Secretariat members have commented positively on the willingness of groups to comply with requests for information and describe this as a "positive relationship, not adversarial". Sixteen of the 20 project leaders interviewed have reported that this contact with programme representatives occurred on at least a monthly basis. All project leaders reported this contact as helpful, with one commenting positively that "we never encountered this level of co-operation and support from a funder before". It should be noted, however, that this monitoring contact is poorly documented in project files, which hold only limited correspondence between the Secretariat and assisted groups.

6.54 Both Secretariat and Advisory Group members carried out project visits. Seventeen of the 20 project leaders interviewed were able to recall visits having taken place, with up to 3 visits noted in cases. There is no evidence that the role or purpose of the monitoring visits were set down in writing. Moreover, while project visits may have provided a useful first hand understanding of the project and its support needs, there is no evidence that the issues or lessons arising from visits was systematically or consistently recorded.

6.55 An independent evaluation was commissioned by the former Rural and Environment Research Team in March 2003 to provide a parallel, overarching evaluation framework for Rural Voices. Undertaken by Edinburgh Development Consultants this study reached draft final reporting stage, but was not completed. This evaluation process appears to have been problematic although it is not clear what the reasons for termination were. However, this appears to have been a vital missed opportunity, to independently assess the performance of the initiative on an ongoing basis.

6.56 Overall, the evidence available raises concerns regarding the clarity and robustness of the monitoring framework for the initiative. While a number of worthwhile actions have been undertaken, or attempted, the learning from these has not been systematically documented. There is no evidence that effective arrangements were in place to measure the progress and/or growth in capacity against a defined baseline at project level, or to systematically track and review performance of the Rural Voices programme against its defined objectives. This is a serious shortcoming for a pilot programme that was designed to test an innovative approach and inform the roll out of continuation action.

Research dissemination

6.57 Rural Voices was conceived with dissemination firmly to the fore. The expectation was that learning would be systematically captured and shared, and used to inform wider policy and practice.

6.58 The evidence indicates that no robust structure or strategy was put in place to ensure that the learning from Rural Voices was systematically used to inform wider policy and practice. The lack of an agreed dissemination strategy was widely acknowledged by Advisory Group and Secretariat members participating in this evaluation as a key limitation within the programme, with one Advisory Group member going as far as describing this as the "Achilles heel" of the initiative.

6.59 Some useful elements of dissemination activity did, however, occur. Three newsletters (October 2003, November 2004, May 2006) represented the main form of dissemination, and presented vivid case study illustrations of the experiences of projects. Advisory Group members report that information was also posted on the Rural Community Gateway website (from Round Two), although the scope or frequency of postings has not been documented.

6.60 However, key strands of the intended dissemination strategy set out in the Outline Proposal for the initiative did not occur. The reporting outputs from the projects were not, as intended, disseminated as part of the SNRP New Ideas in Rural Development Series or shared through an SNRP Rural Voices conference at the end of the first round. Indeed, until now, no reports or accounts of the programme, other than newsletters, have been released into the public domain. This might be attributed to the demise of the SNRP, but does not explain why alternative mechanisms to achieve dissemination objectives were not found.

6.61 Potentially, the expectation was that participating groups themselves would collate and share the research findings and lessons with others. However, the direct support offered to projects to achieve this was not substantial. When asked "Did your group receive advice from a Rural Voices representative on how to go about writing up and sharing your findings?", only 6 of the 19 project leaders interviewed confirmed that this was the case. This support was provided through one-to-one contact with facilitators at the second workshop and included advice on reporting research findings (2 cases) and preparing presentations or press releases (4 cases). In all 6 cases this support was regarded as helpful.

6.62 Requests for further support were noted in 6 cases during interviews with project leaders. Comments in this respect included:

"More visits from advisors."

"What do you do with this information now that you have it all?"

"Advice on what funding is most suited to their project."

"Another meeting with the report finished would have been useful."

"It would have been nice had they carried on and asked us periodically how we are progressing but of course they closed the programme."

"Support for putting the findings on the internet."

"… need [to be] re-motivated to carry you over the finishing line."

6.63 Overall, it is clear that dissemination arrangements within Rural Voices were weak, and varied significantly from expectations. This is concerning given the intended impact of the initiative in informing wider policy and practice.

Research aftercare

6.64 The evidence suggests that there was no structured or systematic support put in place to assist participating communities to translate research findings into subsequent project influence and action.

6.65 Some informal support appears to have been provided to assist groups to move the research forward after time limited funding came to an end, but this was ad hoc and informal. Late in Round Two a directory of funding options was sent to projects, and a presentation on fundraising provided at the second scheduled workshop. However, it appears that groups were not systematically guided towards potential future funding or support options.

6.66 Projects themselves recognise the limitations of support in this respect. When asked "Did your group receive any support from representatives of Rural Voices in taking forward its findings?" only 2 of 19 project leaders suggested that this was the case.

6.67 The Advisory Group and Secretariat did explicitly acknowledge the importance of aftercare arrangements. One Advisory Group member commented on the need for ongoing support and guidance to enable groups to fulfil their ultimate ambitions, and commented that "groups are only one-third of the way down the process". This comment recognises that action research, done well, should be an ongoing, cyclical and supported process.

6.68 During the course of interviews with Advisory Group and Secretariat members, 4 factors emerged as significant in explaining why specific aftercare arrangements were not put in place as part of the initiative:

• The administrative burden on staff was such that it was not possible to spend time supporting the previous cohort of projects during Round Two
• It was regarded as outside the Scottish Executive's primary role as a public funder to provide ongoing support after agreed contracts were successfully completed
• It was considered important to ensure that a dependence on facilitator support did not arise as part of a prolonged support arrangement that could not be sustained
• The growth in capacity and competence of assisted groups was such that further support was not deemed necessary

6.69 Despite the absence of support through the programme this does not fully explain why systematic signposting or linkage did not occur with other forms of capacity building support. One Advisory Group member commented that "it would have been better to link into local support arrangements". Another noted that a proposal put forward for ongoing network support to be offered as part of the CADISPA programme (University of Strathclyde) was rejected due to the pilot status of Rural Voices.