Scottish Government

APPENDIX A: METHODOLOGY

Background Principles for Evaluation

1 This formative evaluation has been guided by a number of principles developed from previous experiences and initiatives in building research capacity in the HE sector. These background principles are based on three elements that have emerged as key in understanding and, therefore, realising the development and enhancement of research expertise:

• Building research capacity (Box A.1)
• Developmental pathway for building research capacity (Box A.2)
• Governance and collaboration (Box A.3)

2. The guiding principles for building research capacity are based on the original aims and objectives, the organisation of delivery, the mode of delivery (in terms of the approach to professional learning adopted), and the evaluation or expected outcomes. These building research capacity guidelines are based on the previous work and experiences of the authors (see Rees et al (2007) for more information).

3. The illustrative developmental pathway for research capacity building initiatives of this kind was developed from an evaluation of other Scottish Funding Council ( SFC) Research Development Grants ( SFC 2006). The authors of the evaluation developed this pathway based upon the previous experiences of other research capacity building initiatives in Scotland to provide a generalised model or tool for leaders of future investments to evaluate their own progress.

Box A.2 Illustrative development pathway for research capacity building investments (with indicative timeframe)

Source: SFC (2006 ) Research Development Grant Evaluation, SFC/33/2006, Edinburgh: Scottish Funding Council (Annex A - Highlights, p. 20)

4. The guiding principles for governance and collaboration, based on Newman (2001), is useful in better understanding the relationship between governance (and management) and collaboration, and how different models of governance affect the form and characteristics of collaboration possible. It also highlights the tension between the aims and objectives of building research capacity in the HE sector.

Box A.3. Guiding principles for governance and collaboration

Source: Newman, J. (2001) Modernising Governance (p.114)

Methodology for Formative Evaluation

5. The formative evaluation is an empirical study comprising three parts:

• Part One - Documentary Analysis
• Part Two - interviews with AERS participants
• Part Three - interviews with Key Stakeholders

6. Part One of the evaluation involved the collation and analysis of a number of formal AERS documents, including: annual reports, bi-annual progress reports, participants' feedback, and the emerging outputs from all nine research projects and the AERSRCBN (including the AERS website and the Virtual Research Environment ( VRE)). A review of this documentation (Prior 2003) has enabled us to consider the 'official' development of AERS as set out against its aims and objectives, and to begin to consider the impact that AERS is currently having.

7. This part of the evaluation also attempted to establish robust estimates of the participation of individuals in AERS and its networks and projects. This proved a complex task and is discussed separately in Appendix B. The initial quantitative analysis of participation had two purposes. Firstly, to identify and review the coverage of AERS within Scotland through the participation of researchers from consortium HEIs and non-consortium HEIs. Secondly, to draw a sample of participants in AERS for interview.

8. Part Two of the evaluation involved collecting data directly from AERS participants on their experience of participation and their views on the benefits which they have derived through semi-structured interviews. A third source of data on AERS participation was through a short questionnaire administered via email to all members of LLT network participating through the AERS Virtual Research Environment ( VRE). A fourth key element to this part of the evaluation was to interview members of the core AERS team. Furthermore, an invitation was sent to all AERS participants (via networks) and via the AERS website to contact the evaluation team if they wanted to share their experiences and perceptions of AERS with us.

9. AERS had already developed a common framework for participant membership based on three different levels of engagement with the networks. This was used in our sampling strategy.

• Level 1: Entry Level - Interested 'users', observers or potential participants
• Level 2: Active membership - researchers and other stakeholders who take an active role
• Level 3: Close involvement (or core membership) - membership of the project teams

10. The sample for collating the experiences and views of AERS participants is summarised in Table A.1. For the purposes of sampling Level 2 and 3 participants were combined. There is some overlap in the membership of each source of participant identified (for example the Core AERS staff are also Level 3 participants). However, for the purposes of reporting, the final sample do not contain duplicates. Therefore, we obtained or received 64 unique views and experiences of AERS participants. ( NB As far as we can tell the email responses from VRE members did not come from participants we interviewed but we cannot be 100% confident that this was the case).

Table A.1 Summary of sample for network participation (Part Two)

11. The Level 1, 2 and 3 participants were sampled randomly for a number of reasons: (i) to try and reduce any biases in the views and perceptions of our sample; (ii) to ensure we have some representative coverage of AERS participants (e.g. by home institution, gender, academic research experience, career stage); (iii) to encourage participation in our evaluation by eliminating concerns by participants in their selection for interview.

12. The sampling process for Level 1, 2 and 3 participants was not straightforward. First, there was no single database of membership for AERS. Participant data had to be gathered from each Network and in different formats. It is also the case that there is duplication in membership, both across different networks and at different levels of participation. Where we identified individuals who were Level 1 participants in one Network and Level 2/3 participants in another Network then they were sampled only for the Level 2/3 interviews. A detailed breakdown of our original sample by Network is presented in Figures A.1 and A.2.

13. Of the original 30 Level 1 participants randomly selected we were only able to interview 7 from the original stratified random sample. Twelve of the original sample had to be replaced because we had missing contact information, because they had retired from their academic positions, and because the contact information we had was incorrect. Of the 19 total Level 1 participants we were able to contact, 3 reported that they had no knowledge of AERS and therefore refused to talk about AERS. Of the other 16 participants, 6 were from non-academic backgrounds (two professional teachers, two local authority participants, two other stakeholder organisations). The remaining 10 interviews were with academic participants (7 from consortium HEIs and 3 from non-consortium HEIs). It should also be noted that it transpired that four of the non-academic participants had no experience or real knowledge about AERS.

14. All Level 1 participants were interviewed by telephone.

15. Of the 16 Level 2/3 participants randomly selected, 3 were from non-academic backgrounds (one local authority participant and two professional teachers). The remaining 14 participants were academic staff in Scottish HEIs. A number of these were Fellows of AERS. It is important to note that all the non-academic Level 2/3 participants we had sampled refused to talk about AERS and their experiences of AERS specifically.

Figure A.1 Level 1 AERS participant sample

Figure A.2 Level 2/3 AERS participant sample

16. The majority of Level 2/3 participants were interviewed by telephone, although we were able to interview a number of AERS Fellows face-to-face.

17. It was decided to contact members of the AERS Virtual Research Environment ( VRE) separately from the other participants above because the only contact details available for them were email addresses. The evaluation team decided, therefore, that they would contact this group of AERS participants separately, via email, with the main aim of discussing the VRE. The dataset of VRE participants was provided by the LLT Network. We were able to send a short email questionnaire to 76 members from this dataset. We received 18 responses by the end of the evaluation period. 11 were from academic backgrounds (2 from non-Scottish HEIs, 5 from consortium HEIs, 4 from non-consortium Scottish HEIs), and 7 were from non-academic backgrounds (including 3 professional teachers). As before, it should be noted that two teacher-respondents reported having had limited or no involvement with the AERSVRE.

18. The sample of core AERS staff included the following:

• Chair of the AERS Management and Executive Committees
• AERS Coordinator
• AERS Research and Knowledge Transfer Fellow
• Coordinators for each AERS Network (4)*
• Employed research staff from all AERS Network (6)1

* This includes the former coordinator of the RCB Network. Their name was taken from the AERS website. We were only made aware that there was a new coordinator during the course of the evaluation.

19. Nearly all core AERS staff were interviewed face-to-face.

20. Part Three of the evaluation involved gathering the experiences and/or the perceptions of AERS from a range of key 'stakeholders' through an interview. A number of these stakeholders were also members of the AERS Management Committee, so perhaps had a much greater knowledge of AERS than their colleagues within the same organisations. These stakeholders included representatives of the following organisations:

• Deans of Education in Scottish Higher Education Institutions
• Scottish Educational Research Association ( SERA)
• Scottish Executive
• Association of Directors of Education in Scotland
• Convention of Scottish Local Authorities ( CoSLA)
• General Teaching Council for Scotland
• Learning and Teaching Scotland
• HM Inspectorate of Education (Scotland)
• Educational Institute of Scotland ( EIS)
• ESRC Teaching and Learning Research Programme

21. We were unable to identify anyone from the Educational Institute of Scotland to interview about AERS. Additionally permission was only received from someone from the Association of Directors of Education in Scotland for an interview after the evaluation period had ended. But even in this case they indicated having had little knowledge or experience of AERS.

22. The majority of stakeholders were interviewed face-to-face. A couple of stakeholders had to be interviewed by telephone.

23. Each interview was tailored to the interviewee, their level of involvement in AERS, and the organisation they were from. The following provides an indication of the topics that were covered in the interviews:

• Brief details of the respondent's professional background
• Nature of the respondent's engagement with AERS
• How AERS has developed
• What respondents perceive to have been highlights of AERS
• What respondents perceive to be the disappointments of AERS
• What obstacles have been encountered
• Respondent's experience of organisation of AERS
• Respondent's evaluation of professional learning made possible by participation in AERS
• Other benefits derived from participation in AERS
• Evidence of changed activities/behaviour resulting from participation in AERS

• What kind of impact AERS has had outside Higher Education sector
• The sustainability and future of AERS

24. Much of the analysis has been to represent and highlight the operation, processes and impact of AERS to date. This was based on combining the data from documentary evidence, and the interviews with network participants and other key stakeholders.

25. The evaluation team has also provided some commentary on the quality of research through our own consideration of AERS outputs. Additionally, we draw upon the interviews to gauge the impact/relevance or potential impact/relevance of research from AERS projects as a further indicator of quality.

26. We have also critically reviewed the structure and organisation of AERS - including its collaborative working, networking and dissemination strategies - based upon the interviews and the documentary analysis.

27. Many of the interviews were recorded and transcribed to aid our analysis and reflections. However, due to the sensitive nature of much of the material we gathered, combined with the small-scale nature of the sampling, we have decided to limit the amount of data we use in our reporting.

28. We also attempt to make little distinction in the relative progress of each AERS project and Network. Our evaluation focuses, therefore, on the Scheme as a whole. Clearly there will be some exceptions then within AERS to the general picture that we portray, but it is not our objective to produce a summative or complete evaluation on all aspects of AERS. We wish to identify the general achievements to date, highlighting areas of strength and areas that perhaps need greater attention over the remaining period of AERS funding.