Scottish Government

ANNEX C THE CASE STUDIES

Case study A
Knowledge transfer case study:
Evidence in lifelong learning network (ellnet)

Summary

• The Evidence in Lifelong Learning Network ( ELLNet) was established by the Scottish Executive Enterprise and Lifelong Learning Department in 2001.
• After some initial success in influencing the thinking of policy makers it was felt in ETLLD that the Network had become too academic, and too distant from the matters of interest to the Department.
• In March 2005, ELLNet was re-launched with a revised model of operation. The new Network was more policy-focussed. Contributions from external experts were stronger when the Network's agenda was linked to policy issues.
• The format was changed to the delivery of half-day events comprising 3 related presentations. This encouraged attendance and participation from a wider range of policy makers.
• The focus was on existing work rather than the commissioning of new work. The new format also emphasised the value of research undertaken within the Analytical Services Division and thus improved internal knowledge transfer.
• Originally, the Scottish Executive researcher role was as 'knowledge broker' and this was key to the success of ELLNet. However, the development of ELLNet required significant investment in researcher time to ensure the continued momentum of knowledge transfer activity. This level of resource input was difficult to sustain in the longer-term. It was also suggested that reliance on one or two individuals potentially restricted the wider engagement of other analysts in this form of knowledge transfer.
• The sustainability of the Network remained a key consideration. Members of the Network needed to benefit from their engagement if they were to remain actively involved over time.
• This case confirmed the need to design/manage events that were not over-dependent on individuals or hugely resource intensive for senior staff. This ensured that events were easier to sustain and run on a regular basis, particularly if staff resources were reduced and/or where there was staff movement.

Introduction

1. The Skills and Learning Research Network was established by the Scottish Executive Enterprise and Lifelong Learning Department in 2001. The aim of the Network was 'to be more imaginative in locating the ideas and perspectives emerging in research which can generate policy thinking in this field' and specifically to feed into the Review of Vocational Education and Training and the development of the Lifelong Learning Strategy.

2. The Network was re-branded, and put on a more permanent footing, as the Evidence in Lifelong Learning Network ( ELLNet) in 2002 to more accurately reflect its aims and membership. At that time, the main aim of ELLNet was to develop the role of research and evidence in policy making and practice in the longer-term by:

1. Facilitating dialogue between the policy and research communities;
2. Identifying key issues for policy development;
3. Maximising use of the existing knowledge base in relation to those issues;
4. Developing priorities for strategic research; and
5. Improving the coherence of relevant research across Scotland.

3. Members of ELLNet comprised a range of people with different perspectives, expertise and resources. Members were drawn from the research community (including academics, research organisations, and think tanks), the main policy makers in this area, practitioners and funders of research.

4. Following early successful engagement with policy makers, it was felt in ETLLD that the network had moved away from its original purpose. In March 2005, ELLNet was re-launched with a revised model of operation. The event focused on the development of an employability framework and included presentations on a number of ongoing research projects that were providing evidence to support the development of this policy area.

Assessment of ELLNet as an example of good practice in Knowledge Transfer

5. Reflection on the first year of operation of ELLNet by ETLLD Researchers in 2002 identified a number of demonstrable benefits including:

• Asking different types of questions - the focus had been on challenging fundamental assumptions (e.g. what were we trying to achieve?) rather than reviewing the operation of existing policy mechanisms (e.g. was Skillseekers working?).
• Taking a different approach - a strong analytical, evidence-based approach had been employed in formulating policy and justifying policy recommendations to Ministers and presentation of policies to Parliament and the public.
• Impact - this approach provided answers which were not anticipated e.g. the need to significantly redesign interventions for intermediate level work-related skills.
• Developing a shared evidence base - policy makers and practitioners worked together with the research community to develop a shared understanding of the implications of the evidence-base. For practitioners, this was beneficial in understanding the rationale behind policies.
• Developing an understanding of the policy process - the Network provided the opportunity not only for providing critique, but also the challenge of seeking solutions.

6. It was reported, that in its first year of operation the Network improved the evidence base and increased the credibility of the ASD within the Department. In providing advice to policy colleagues, Social Researchers found it helpful to refer back to discussions held at the Network with leading academics. However, within both ASD and policy divisions a doubt was expressed as to whether after that first year the Network had ever really fed into either policy or the forward research programme. The anticipated benefits to either the Department or external researchers never materialised. Following the ELLNet meeting held in Spring 2004, it was felt by Scottish Executive members of ELLNet that the Network had moved away from its original objectives to more of an academic forum and its usefulness for policy making within the SE was questionable. In particular, the following problems were identified:

Perceived problems with the former ELLNet model

• No clear objective - without the clear objective of feeding into the Reviews, the Network lacked focus and drifted from its original objectives;
• Too academic - it was felt that papers were too academic in nature (e.g. the 'take home' messages of policy relevance were not always clear or useful);
• Lack of policy focus - the Network had become more of an academic networking event rather than being specifically policy focussed;
• Too long - a whole day meeting was felt to be too long, with only a handful of people left at the end of the day for useful debate;
• Too formal;
• Administrative burden - keeping the Network active and arranging meetings was a huge administrative task for one Social Researcher to manage;
• Topic coverage - The wide range of topics covered in each meeting meant it was often difficult to have a meaningful debate and target the relevant audience;
• External location - policy makers in particular often could not attend a meeting for the whole day and were also unable to just 'drop-in' if the meetings were hosted externally in academic institutions.

7. In order to address these problems, Scottish Executive analysts and policy makers from ETLLD re-designed the format of ELLNet and re-launched the Network in March 2005.

Format of new ELLNet model

• More informal;
• Delivery of half-day event(s);
• Hosted in Scottish Executive buildings;
• Met 3-4 times per year;
• The agenda was driven by policy concerns;
• Meetings were to be around a theme with introduction from the policy lead and chaired by a policy maker from a different, but related, division to act as a 'critical friend' to stimulate debate;
• Used to 'showcase' internal analytical work in addition to external expertise;
• Were less of a commitment for members;
• 3 short papers per meetings, one of which was produced by ASD ;
• Only invited interested parties to individual events not the whole Network. There was a 'core' group of members who attended all meetings;
• Reduced administration involving policy divisions;
• A more multidisciplinary approach - making more use of statistics to provide context for instance
• No notice board/secure website.

Preparation and planning

8. The origin of the ELLNet was to feed academic evidence into the Review of Vocational Education and Training. A series of seminars were held and there was a very clear objective for academics to feed into this process. The work of the Network also contributed to a shift in the Minister's views and informed priorities in the Department. A more comprehensive Review was undertaken as a result of the input from external members of the Network, who generated different questions/policy concerns than those of the policy divisions. In addition, in the initial period, the Network generated a lot of ideas for more strategic research (e.g. 'bigger picture' research rather than evaluations of individual programmes).

9. The composition of the Network was originally selected in an attempt to move towards more solutions-based thinking. For example, it was suggested that academics tended to be strong on critiquing policies, but not so good at providing solutions (e.g. what would work better and where should resources be allocated?) whilst policy-makers tended to see research in a rather narrow mindset of programme evaluation. Funders of research in this field (e.g. Scottish Executive, Scottish Enterprise, Higher and Further Education Funding Councils) were also represented on the Network to try to facilitate more joined-up research funding and to consider the possibility of joint-funding. Whether this has happened in practice was not evident at the time of writing this report.

Resourcing

10. The substantial costs of running the original Network were met by the Scottish Executive with the main contributions sought from members being their time and thought. Members offered in-kind support (for example, venue for a meeting, chairing meetings or presenting current work).

11. The Scottish Executive provided core funding for ELLNet to support up to 4 full day meetings per year. The core funding supported:

• Venue and catering requirements;
• Travelling expenses and hotel costs for participants, where necessary;
• Fees for the preparation of up to 8 papers to be presented to the Network. Papers were commissioned by invitation from the Scottish Executive to ELLNet members or others. Papers were to be in an established format (5000 words comprising half review of evidence base, quarter implications for policy or practice and quarter implications for the research agenda). Facilitation of up to 4 workshop sessions could be commissioned by invitation from the Scottish Executive.

12. The sustainability of the Network was a key consideration. It was suggested that there needed to be mutual benefit for all members of the Network and this was essential if they were to remain actively involved over time (e.g. what's in it for them?). In the initial stages, Scottish Executive members of the Network were very conscious of the Network not being seen as the Executive seeking "advice on the cheap". Thus external members of the Network were encouraged, for example, to set the agenda for meetings and to present their research findings so that the Network was seen as a partnership resource rather than being Executive-led. However, in retrospect this contributed to the Network lacking clear purpose.

Medium/delivery method

13. In its first year of operation, the main value of ELLNet was the face-to-face meetings that it generated. In addition to the main forum, informal meetings with academics were held in-between meetings. 3 papers were commissioned and prepared by Network members. However, it was felt by some Scottish Executive Network members that these tended to be too academic and not of obvious relevance to immediate policy concerns.

14. It was originally planned to establish a secure ELLNet website/discussion forum with passwords issued only to members, but the technology was such that this did not operate as anticipated. This was not to say that such a discussion forum could not operate effectively, but ETLLD did not see a role for this in the future.

15. Under the new model of operation, only short papers were presented at each meeting. The aim was to showcase ongoing, internal work (e.g. present some statistical work, internal literature reviews, outputs from commissioned projects in addition to external expertise). The aim was to make the Network more policy focused and it was believed that by having an agenda more closely linked to thinking on policy issues, contributions from academics could be stronger.

Interactions and relationship management

16. Following the last meeting of the Network in 2004, it was suggested that there were some difficulties in attracting 'active' members willing to be continually involved. It was suggested that many academics or external researchers were prepared to give one-off presentations on a particular subject, but anything more than this was seen to become overly burdensome. The consequence of this was that meetings tended to be dominated by those from the Scottish Executive and a few 'core' academics from Glasgow, Strathclyde and Caledonian Universities. It was suggested that these academics were 'policy attuned' (i.e. they knew how to approach Government, and would have done so anyway in the absence of the Network). Furthermore, it was suggested that reliance on individuals (both internal and external) potentially restricted wider engagement.

17. The new ELLNet model aimed to reduce the burden on Network members. For example, members were not expected to attend every meeting, only those of most relevance to them. In addition, only short discussion papers were prepared for each meeting and the emphasis was on using existing analytical work rather than commissioning new work.

18. Within the Analytical Services Division, ( ASD) it was felt that internal knowledge was not being recognised and the format tended to give greater value to knowledge generated in academic institutions. The new model placed academic work alongside internal analytical work (which can be better tailored to policy audiences) and also provided an opportunity to share internal analytical work with an academic audience. As such, the new model for ELLNet facilitated internal knowledge transfer as well as knowledge transfer into the Executive from academic institutions.

19. Analysts from all disciplines in ETLLD continued to take a joined-up approach to providing policy briefings. Policy makers knew how to use the ASD and expected this mixed approach. However, the external members of the Network tended to be dominated by (educational) sociologists. Some economists were invited to specific events by special invitation. However, the Scottish Executive was plugged into relevant economists through SCOTENT (Scottish Economic Policy Network http://www.scotecon.net/text-site/index.htm) and more informally (via the ASD Head who is from a recent academic background). There was also a Scottish Labour Market Statistics Group to discuss the development of statistical work in this field.

The X factor

20. The literature suggested that personalities were key to successful knowledge transfer activity. In this example, initially, a large proportion of time (estimated at between one-third to one-half of a senior research officer's time) was devoted to managing the Network and, importantly, building relationships, commenting on papers and briefings. The initial meetings were 'staged' in that a lot of time and effort was put into making them work. For example, presenters were 'paired' so that a more traditional academic presenter was perhaps matched with a presenter from a 'think-tank' to add different dimensions to the discussion. In the beginning, the role of Scottish Executive researchers as 'knowledge brokers' was, it was reported, fundamental to the success of ELLNetthough it was also argued that reliance on key individuals (both internal and external) perhaps got in the way of wider engagement of other analysts. However, subsequent changes in personnel meant that the Network became more akin to a project to be managed (e.g. one of many) and latterly sufficient resources were not being devoted to making it work as originally envisaged.

21. The new model scaled down the resources needed and acknowledged that on some topics there was a level of expertise within the ASD. Therefore, it was not always necessary to look outwards. On the other hand, Scottish Executive analysts were not necessarily experts in their policy field since they were mainly recruited through general boards (rather than for specific posts) and there has tended to be significant movement between posts. Striking the right balance between internal and external expertise was therefore important.

22. People in the ASD reported that the analysts within ETLLD tended to have a multi-disciplinary approach and policy makers within the Department expected this. Policy makers were receptive to research and were keen to see the development of, for example, more forums and lunch-time seminars. This was not necessarily the case in every Department.

Conclusion

23. A new model of operation for the ELLNetwas introduced in March 2005 with the aim of being more closely linked to policy issues. The revised format included a re-balancing between internal and external knowledge with a greater role for the sharing of internal analytical work. The role of the Network in developing a strategic framework for lifelong learning research needed careful consideration. It certainly appeared to be the case that this process, originally envisaged for the Network, never materialised and the external members of the Network had little, if any, impact on the forward research programme. Therefore, they received little benefit in joining the Network. The sustainability of the Network remained a key consideration and ways to encourage continued involvement of Network members needed further discussion. It was envisaged that by having an agenda more closely linked to thinking on policy issues, contributions from external experts could be strengthened.

Case study B
Knowledge transfer case study: mainstreaming equalities website

Summary

• The Mainstreaming Equalities Website ( MEWS) was set up by Development Department's Analytical Services Division ( DD: ASD) Equalities Research team and was launched in December 2003.
• The site was developed as a proactive response to a request for support from the Equality Unit on the range of research available.
• MEWS site content was organised around equalities groupings and Scottish Executive topic areas. This allowed for cross-referencing between topics and groupings, and considered issues of multiple discrimination.
• The extent to which the site was used within the Executive was unclear, although there existed informal reporting of use by Social Researchers and some policy colleagues (particularly in the Equality Unit).
• Outside the Executive, some use was reported, including the use of the site as a support tool for new European Union member states.
• The site was cited as an example of good practice and, at the time of this report, it appeared to be the only resource of its kind in Europe.
• The future direction and development of the site was also being reviewed.
• Social Research teams across the Executive needed to take ownership of the site. More work on this will be required from the team. However, the site was already cited in the Procurement Guidelines.
• Training courses on using the site were available to Social Researchers, but not to policy colleagues. This was seen as a gap and there were questions about the extent of policy awareness and use of the site.
• Further development of the site was expected to take place from Spring 2005.

Background

1. The Mainstreaming Equalities Web-Site ( MEWS) was formally launched in December 2003 after a year of development. The site was developed by the DD: ASD Equalities Research team as a response to generic requests from policy for better understandings of what research was available.

2. MEWS described itself as:

A collection of equality related (web-based) resources, designed to help policy makers and researchers understand the broad range of equality issues that have relevancy across policy areas. This is not a research data-base, but, rather, an accessible set of resources that can enable both policy makers and researchers to gain some insight into how the needs of different equality groups should be considered when embarking on a research or policy project. The reports referenced on this web site will give you a sense of how an equality issue can impact on a particular policy area. As such, if you are embarking on project work relevant to, for example, housing you will find it helpful to look at how equality issues have impacted on this policy area in different ways.

3. Although this seemed clear, there was some uncertainty about the site's identity, which is discussed below.

Main aims

4. The 3 main aims of MEWS were:

a) To promote the mainstreaming of equality in research and policy work by providing a single access point to introduce policy makers and researchers to relevant issues and further information;

b) To make better use of existing equalities research by making it more easily accessible for policy makers and researchers; and

c) To promote the mainstreaming of equalities in research by providing access to relevant tools (i.e. the researchers' toolkit).

5. These aims were still in place at the time of this report, although these may change following further developments to the site from Spring 2005 onwards.

Planning and preparation

6. Initial planning and further development of the site was taken forward by the Equalities Research team and involved liaison with the Equality Unit, Social Researchers across the Executive, and other analysts. The site took approximately 1 year to prepare - from the start of discussions to its launch. The site took longer to put together than expected, with the major challenges being the amount of preparatory work required and the time and effort spent agreeing the topic areas to be used on the site.

Quality of the source and of the research presented

7. The research available through the site was not quality controlled in any formal sense, although many reports were known to (or were actually commissioned by ¹²) the Social Research team. The site was not a systematic review service, nor was it a guarantor of best quality research. However, there was some concern that this might be seen to be the case. Some initial checks were done on the information included, but these were by relatively informal assessments. More rigorous assessments would have been impossible in the timeframes provided.

Medium/delivery method

8. Set up and maintained by the DD: ASD Equalities Research team, MEWS provided a number of key website-based services:

• Easily accessible research reports, research findings, fact-sheets, and policy papers from a range of agencies and organisations, not just the Scottish Executive. These were available in full from direct links on the site;
• Reports and links were searchable by research topic and equalities group and could be cross-referenced;
• An Equalities ToolKit to help researchers and policy colleagues consider equalities issues throughout the research process;
• Information about mainstreaming equalities issues into policy;
• Links to a range of equalities organisations.

9. The site provided links to research and other information-related publications, making it possible to put together a list of reports on a specific topic cross-referenced by an equality grouping - for example, minority ethnic groups and schools - and click on the links to access the reports. Alternatively, searchers could also narrow down their search by a further equalities grouping to consider whether previous research has considered multiple discrimination. This service can help staff write better-informed project specifications that do not serve (albeit inadvertently) to make the equalities gap wider. It also helped to avoid commissioning work unnecessarily, where other evidence already exists. The site also indicated gaps in the available research.

10. MEWS helpfully backed up this services with short and longer introductions to key issues for each equality group. These provided an interesting starting and thinking point for research and policy staff to take on board.

11. The site also featured a 'toolkit' for researchers looking to consider equalities issues within the research process. The structure for this was the development, tendering and management of a research project, asking researchers to consider the diversity of issues that may be relevant to the task and whether a mainstreaming approach would positively enhance the outcomes and applicability of that work. An example from the ToolKit is given below:

The Research Idea

12. You have identified your research question(s) and aim(s) and how will you build in equality considerations?

Use of resource

13. At the time of our discussions, the team had little clear idea of how MEWS was being used, how often it was being used and by whom. They had the sense, however, that some people were using it a lot, for example Equality Unit colleagues, while across the Executive, it was perceived that it was not being used on a regular basis, except by some researchers.

14. In a sense, this was one of the difficulties of evaluating the use of websites or databases as mechanisms for effective knowledge transfer. Assessing the amount and extent of use, and the difference it makes in qualitative terms, can be problematic. Our literature review stressed the importance of one-to-one contact in effective KT, and of course, IT-based resources did not provide this, however something additional needed to be slotted in. This was not to say that MEWS and similar resources lacked value; just that the extent to which they were truly knowledge transfer activities as defined in the main body of the report needed to be considered with some care.

15. During 2005/2006, further development of a 'mainstreaming' post within the Equalities Research Team was anticipated. It was suggested that this post will work closely with policy colleagues in the newly-established Mainstreaming Project Team (based within the Equality Unit) on a range of mainstreaming issues and will include responsibility for the review and development of MEWS and providing training around it.

16. The website was not updated significantly since its launch and further work on this was required. More ownership from Social Researchers across the Scottish Executive would facilitate this, in that researchers in particular divisions could take responsibility for passing on new reports / information in their area to the Equalities Team when possible. In support of this, MEWS was cited in the Social Research procurement guidelines.

17. However, informal feedback from a contact in the European Union suggested that the site was being promoted, particularly to new member states, as an example of excellence and a unique resource within Europe:

"… it remains a unique resource, as far as I can see from consulting various websites across Europe, and it is really impressive. Going back to the theme of lesson learning from other administrations and countries, the SE also has something to promote!"

Events to date

18. A number of training sessions have been held for Social Researchers on using the MEWS site, with research staff and Equality Unit colleagues, taking in mainstreaming issues. Feedback from the training events on the site was generally positive. MEWS was seen as a useful resource and a number of suggestions for improvement were made, e.g. highlighting where strategies to mainstreaming equalities were successful, rationalising the number of categories, incorporating a search facility, and regular updates. Feedback also suggested that a greater policy input (more time for questions and answers and a policy official present throughout the session) would have been helpful. It was suggested that the training could be repeated regularly, could be compulsory for researchers and could be extended to policy makers. One of the key roles of the policy-based Mainstreaming Project Team was to develop mainstreaming training and tools for policy makers across the Scottish Executive over the coming 2 years. This work will build upon existing approaches to mainstreaming and so the focus and contribution of MEWS will be reviewed as part of this work. It is therefore likely that both the content of, and training related to, MEWS will undergo substantial changes in the future.

The X factor

19. The X factor in this case was the determination of the research team to get the website up and running. This determination extended not only to managing the practical problems, but also organising wider consultations across social research and policy.

Lessons learned

20. There was some uncertainty about the site's identity. The site lacked a clear focus in terms of future work. MEWS was not considered to be a 'research database', but there was uncertainty about what else the site could be, whether it was meeting its full potential as a tool for policy and research. The site was also problematised by a disparity between the mainstreaming principle and a centralised Equality Unit driving forward equalities issues that were not mainstreamed across the office. This did not help the team to promote MEWS outside its core constituency. The team also felt, in retrospect, that the profile of the site suffered because policy work that was supposed to accompany it had been delayed. However, in early 2005 a Mainstreaming Project Team was finally established within the Equality Unit with the objective of establishing clear structures, mechanisms and tools through which mainstreaming equalities could be put into practice across the Executive. This work included a review of MEWS content and structure and a discussion of the issues concerning the site's identity.

21. The team thought that getting the site up and running was an achievement in itself. However, there were some useful lessons for the future:

• There was an underestimate of the amount of time required to plan and prepare the site. This was a much bigger task than was anticipated.
• The Scottish Executive topic areas on the site did not fit equalities issues particularly well and may need revising.
• The positioning of the site (within the Social Research part of the Executive's website) may need changing. The website would be more strategically placed on the policy gateway. This option may be explored as part of any site review to inform the work of the Mainstreaming Project Team.
• Much effort will be required to get policy makers across the Executive to use the site on a regular basis. One of the key outputs of the Mainstreaming Project Team will be to establish commitment and ownership across the Executive for the principles and processes of mainstreaming equalities. This will facilitate the promotion of the site within this context.
• Social Research branches need to be encouraged to develop a good relationship with the website and its aims.
• There was perhaps too much information currently available on the site and its content may need pruning.
• The site needs ongoing investment in terms of time so that it can be updated regularly.
• The issue of quality control needs further discussion and debate.

Future plans

22. The development of a portfolio of work within the team, which focuses on mainstreaming issues, will give the site fresh impetus. There was a sense that the site needs to move on, in addition to being kept updated and expanded, and that the relationships of social researchers within the Executive to the site will need to be developed (see above) as a first step to engaging more with policy colleagues across the Executive.

23. MEWS can be accessed at the following web address:

http://www.scotland.gov.uk/mainstreaming/?pageid=403

Case study C
Knowledge transfer case study: Economic and Social Research Council programme on devolution and constitutional change

Summary

• The Economic and Social Research Council ( ESRC) Programme on Devolution and Constitutional Change involved a series of seminars, organised through the Office of the Chief Researcher ( OCR), hosted by the Scottish Executive and targeted at both analysts and policy officials in the Executive.
• At the time of writing, 2 seminars had been held. The first was on The Scottish Economy and the second was on Public Attitudes in Scotland. A further seminar on Europe was also being considered.
• The ESRC is well-recognised and respected as a quality, academic organisation that meets all the necessary criteria for quality research.
• The planning and preparation of the seminars was carried out in close consultation with the ESRC and subject areas were chosen in relation to the targeted audience's interests.
• Of the 2 seminars delivered, the first was more successful in respect of knowledge exchange and interactivity due to the size of the venue. The second seminar was held in a large 'theatre-style' conference room and was not conducive to interactivity.
• The first seminar's audience comprised mainly economists, whereas the second seminar's audience consisted largely of policy officials from a number of departments in the SE.
• Policy officials, OCR and the ESRC's Programme Director all considered that the seminars were a useful and valuable means by which to impart information, exchange views and knowledge and learn more about each others' work.
• Lessons learned from the seminars included:

Background

1. The Economic and Social Research Council ( ESRC) organised a research programme on Devolution and Constitutional Change. The Office of the Chief Researcher considered that many of the research projects in the programme were of relevance to different areas in the Scottish Executive, particularly the Policy Unit, the Public Service Delivery Group and European Affairs.

2. Contact was made with the ESRC Programme Director, who was very proactive in developing the research policy interface between academia and analysts and policymakers in the Scottish Executive. It was agreed that a series of seminars, hosted by the Scottish Executive ( SE) and presented by the ESRC would provide the opportunity to present information relevant to policy and provide a forum for an exchange of knowledge and ideas between academics, analysts and policymakers. A Senior Principal Researcher ( SPRO) took on the role of co-ordinating and organising the seminars and their content, and was the main Scottish Executive contact for the seminar programme.

3. A seminar programme was therefore set up, that would provide new and relevant information on current issues relating to Scottish Executive policy, and where contact and networking could take place with the potential for follow-on contact and knowledge transfer between the ESRC and the appropriate Executive staff.

Main aim

4. The main aim of establishing the ESRC seminar programme was to provide new and useful information relevant to analysts and policymakers in the Scottish Executive, and to facilitate knowledge transfer and exchange of ideas between the participating groups.

Events to date

5. At the time of writing this report, 2 seminars had been held (the economics seminar and public attitudes seminar) and a further seminar on Europe was planned.

a) Public Policy Seminar on The Scottish Economy

6. The seminar programme was launched with a seminar on The Scottish Economy, and policymakers and economists from relevant topic areas were invited to attend. An average sized conference room was booked, but more people wished to attend than was anticipated. However, most of the attendees were SE economists. The seminar provoked a good discussion on the issues, but no feedback from policy was received on how useful it was to their requirements. However, a couple of senior policymakers who participated asked several pertinent questions, and interacted with the presenters.

b) Public Policy Seminar on Public Attitudes in Scotland

7. Because of the experience of the first seminar, a larger conference room was booked for the Public Attitudes Seminar, and many policy colleagues were invited who had a strong interest in the area. In the event, the majority of policymakers who said they would attend did not, due to other priorities, and most attendees were social researchers. There were few senior policymakers, and most were band C1 and under. The presentation went well, though the discussion following the seminar was disappointing and there was little interaction or exchange of ideas. However, feedback from policymakers who had attended indicated that the seminar had been very useful in providing valuable new information that met their requirements.

Planning and preparation

8. The planning and preparation of the seminar programme involved liaison and contact between the SPRO and the ESRC Programme Director. Discussions took place on:

• The desired format of the seminar (not to be academically focused, but rather directed at the target groups of Scottish Executive practitioners);
• The relevance of the research projects and findings to current high profile issues in the Scottish Executive (e.g. devolution);
• The timeliness of the seminars to particular areas (e.g. Public Attitudes and developments in the Public Service Delivery Group and the Cabinet Delivery Report);
• How the seminars would be promoted (highlighting the relevance of the research to policy issues).

9. Preparation was not formal, but was on an informal basis and events agreed by way of interaction and discussion.

Quality of the source and of the research presented

10. In this particular case, the quality of the source was not in question, and similarly the credibility of the research and methodology used was not an issue, due to the ESRC's strong and established reputation for being a quality, academic organisation that meets all the necessary criteria for quality research. There was not an issue of the audience (analysts and policymakers) 'trusting' or respecting the presenting organisation, as it was accepted that the presenters were experts in their respective fields.

11. The SPRO was acutely aware of the importance of research issues and findings being presented in a clear and focused manner, so that the knowledge transfer was 'user-friendly', would promote understanding and facilitate discussion. Although the importance of this was discussed with presenters in both seminars, the first seminar was perceived to be 'academic' in nature. However, because it was mainly attended by Scottish Executive economists, the measure of understanding was great, and much discussion was provoked and continued during and after the seminar. The second seminar on Public Attitudes was much more focused and made relevant to policymakers, but provoked little discussion. This may perhaps be because the seminar was so successful in getting the 'take home message' across, that the audience felt little need for questions or discussion on the issues.

Medium/delivery method

12. Interactive seminars, followed by discussion at lunch, were considered the most appropriate methods to disseminate the knowledge acquired by the ESRC in their research, and for knowledge transfer to occur in an environment where people were respectful of each other's knowledge in their respective areas. Relevant 'end users' (i.e. analysts and policymakers in the Executive) were advised of the seminars and invited to attend and the seminars were tailored to the audience. Following the first seminar, the SPRO did a paper to Management Group on the key messages arising from Devolution. However, no feedback was obtained (nor sought) from Management Group on this approach. The fact that seminars were followed by lunch, provided the opportunity for discussion to take place on a more 'one-to-one' basis, and this proved a useful and informal means of transferring knowledge in these areas, and proved to be popular with those who attended.

13. The planned programme of seminars was, and continued to be, timely in that they were being held in the context of the Strategy and Delivery Group having a strong interest in public attitudes following Devolution, and the Cabinet Delivery Report. The third seminar, yet to take place, was expected to be on Europe and Scotland's changing relationship within Europe since Devolution, and also in the context of the Meeting of European Heads in Edinburgh in December 2004.

14. Costs of holding the seminars were minimal. The ESRC did not charge for attending and presenting, and the only costs incurred to the Executive were for catering. The seminars, therefore, constituted a good 'value for money' option to transfer and exchange knowledge in these policy fields.

Interactions and relationship management

15. A strategic approach was taken to the planning of this seminar programme. It occurred in the wider context of current political issues pertinent to the Scottish Executive. Because it was a planned programme of different seminars (although all relating to relevant contemporary political issues), the programme was seen to be more effective than a 'one-off' activity, and it was hoped that relationships will continue to develop as the programme continues, and knowledge transfer will be a consequent valuable result. The seminars provided a useful forum for academics to learn about the role of Scottish Executive practitioners and vice versa, and enhance the academic/policy interface.

16. However, it was not clear how much follow-up occurred between Executive attendees at the seminars, and the academics involved in presenting the research. The SPRO and organisers identified nominated contacts in Social Research to keep in touch with relevant ESRC projects, but no follow-up was carried out to ascertain whether this actually occurred. There was also no mechanism in place to establish whether other analysts and policymakers at the seminars engaged with the academics on a regular basis, or at all, following the seminars. Although most indicated that they had found the seminars very useful, no attempt was made to monitor the outcomes in terms of continuing knowledge transfer between Scottish Executive attendees and ESRC researchers.

The X factor

17. The success of the seminars that have occurred thus far was partially due to the relationship between the SPRO involved in organising the seminars, and the ESRC Programme Director who had a good understanding of what was required, who the audience was, and how academics can effectively get their message across. The seminars involved a great deal of work and planning to organise, and required considerable inputs of time. It was therefore essential that both the Scottish Executive organiser (in the Office of Chief Researcher) and the Programme Director were 'in tune' about what was required, were dynamic in the presentation of results and enthusiastic about the concept and value of knowledge transfer. In this case, these factors were present, contributing to the seminar programme's success.

From the policy attendee's perspective

18. Interviews were conducted with 2 attendees of the seminars. The first interview was with someone who attended the seminar on Public Attitudes, with a specific interest in public attitudes in Scotland to inform the Strategic Audit of Scotland, the paper on Confident Scotland, and the Horizon Scanning Sub Group.

19. The attendee found the seminar very interesting and did use one specific aspect (information on public attitudes) to inform their Confident Scotland and Horizon Scanning Sub Group work. They were particularly interested in information on attitudes of people living in, but born outside Scotland, regarding identity and confidence. Apart from this information being extremely valuable to their work at that time, they found the whole seminar interesting from a general policy perspective on political engagement and devolution.

20. When asked whether the seminar presented ample opportunity for interaction and the exchange of knowledge, the attendee indicated that although there were opportunities for questions at the end of the seminar, it was 'difficult to interact' due to the large size of the group. It was suggested that the format of the seminar could be changed to allow more opportunities for interaction and that the seminar could have been followed up by (or could have included, following presentations) smaller 'breakout' discussions on topics of particular interest, thus providing the opportunity for practitioners to ask questions, and for an exchange of knowledge to take place. The preferred option of this attendee was that smaller discussion groups could be built into the programme and could take place a few days later, once Executive staff had time to digest the information, reflect, and devise a list of questions and issues they would like to discuss with the researchers.

21. The attendee thought that a programme of seminars such as this, was a useful way to exchange knowledge between researchers and practitioners, and should be continued, with perhaps improvements to the format, including targeting of particular groups for smaller type seminars following the larger event. Although the large size of the venue was not conducive to interaction, which was one of the main purposes of the seminar, this should still be retained so that the largest number of people possible could participate. This attendee also considered that reducing the numbers to only targeted groups would be a mistake, as it would lose a potential audience for which the information would be really useful. By changing the format to a large seminar presenting information, and then followed by smaller focused group discussions, this would address the issue of non-interaction at the main event.

It was also suggested that it would be helpful to those who attended if they were sent some details on the background of the work undertaken by ESRC before the seminar took place, so that the audience was informed and prepared to ask questions at the event.

22. Interestingly, the second attendee interviewed had very similar views and suggestions as the first. The second attendee had attended the same seminar as Head of the Local Democracy Team in FCSD and had a particular interest in local government elections, public attitudes to local government and other institutions, and the extent of the public's involvement in political participation. This attendee was already familiar with 2 of the speakers at the seminar, having worked with them on various local democracy issues, and was therefore aware that the seminar would be of much relevance to the work of the Scottish Executive.

23. However, when asked about the promotion of the seminar, and whether it was targeted appropriately, the second attendee indicated that no publicity material about the seminar had been received, but that its details had been passed on by a colleague in another Department who was aware of the relevance of the topic. This individual therefore became aware of the seminar through a 'circuitous' route, when ideally the team should have been sent the details.

24. Indicating that the information presented in the seminar added to the sum of knowledge of local democracy issues, the attendee used this information to build on advice given to Ministers, particularly on issues of election turnout, standing as a political representative and the views of ethnic minorities on the political process. The seminar therefore contributed to cumulative knowledge on the subject area and assisted the attendee in their everyday policy work in local democracy. Although the attendee was already aware of academic work going on in this area, the seminar presented them with the opportunity of learning more about it, and provided a convenient and easily accessible venue in which to do so.

25. The attendee's views on the format and venue of the seminar not being conducive to knowledge exchange or interaction were very similar to the other attendee. It was felt that there was a lot of information to be considered in each of the presentations, and this attendee recalled that the seminar was rather 'rushed' at the end due to running out of time. The attendee also considered that the information could have been better presented and reflected upon, had the seminar been split into 3 separate seminars on each of the issues presented. However, it was also recognised that by putting on 3 separate seminars, Scottish Executive invitees would perhaps not be able/have the time to attend each one, and that holding the one seminar had the advantage of presenting different kinds of, but related, information at the one event. The second attendee also thought that the format of the seminar may have been improved by having shorter presentations which would have allowed more time for questions, or, like the other attendee, splitting off into specific workshops/group discussions either at the event or a few days later. The latter would allow time for reflection and preparation for discussion of the pertinent issues.

26. The attendee indicated that they would like to see more of these kind of events provided, but suggested that the same seminar in fact be held twice as this would enable the seminars to be smaller, more conducive to interaction, and would suit those employees who worked part-time, and could not always attend seminars held on particular days of the week. If such seminars were held twice, then this would allow not only part-timers, but employees in general to have more choice in a busy work routine, to attend at least one of the seminars held on a particular day.

From the ESRC Director of Programme's perspective

27. The Director of the ESRC Devolution and Constitutional Change Programme considered that the seminars were an appropriate means of transferring/exchanging knowledge between ESRC academics and Scottish Executive officials. Their view was that a 'private' forum was probably more congenial for officials, and when centrally organised through OCR, it made it easier for the ESRC to 'reach' a cross-section of officials across departments.

28. The Director was very much involved in the planning of the seminar programme, and the seminars were set in dialogue with OCR, with the matchmaking of subjects to audience carried out. However, the Director recognised that it was still important for academics to present in a way suitable for non-specialists and that careful briefing beforehand should address that issue.

29. Similar to the views of policy officials in the Executive, the Director considered that the latter seminar offered less scope for interactivity and knowledge exchange due to the format of the large conference room (banked lecture theatre), which did not encourage dialogue. More exchange took place at the former seminar, where the room was smaller and more conducive to interactivity. The Director would prefer, in future, that more of such seminars should be held, and in rooms which encourage dialogue, perhaps stretched over 3 hours with a coffee break midway. Often when a 2-hour seminar finishes with lunch, officials have to 'leave quickly' and the opportunity for networking is lost.

30. In conclusion, the Director considered that from an ESRC point of view, the seminars were successful, and that having a dialogue with officials in this way was an effective 'multiplier' of research findings and a way of building confidence in engagement with academic research. However, it was also noted that not all departments in the SE yet have 'buy in' and further work may need to be done in showing the credibility of academic work for policy audiences and in joining up work across the Executive. The Director also indicated that Scotland has a number of excellent universities which do applied policy work, and they should be 'used' by the Executive to provide seminars on their research and findings.

Lessons learned

31. Lessons have been learned as a consequence of hosting the first 2 seminars, and will be taken into account when the third is held:

• Disappointing attendance by policymakers in the SE was probably due to pressing priorities at the time. However, perhaps if the seminar had been 'sold' in a different way, with a particular 'policy focus', attendance may have been greater. The marketing of the seminars may have focussed too greatly on the academic side (e.g. mention of the ESRC) and not enough on the usefulness to the policy side, therefore potential attendees may not have viewed the seminar as essential to their work.
• The purpose of the seminar has to perhaps be made more clear, to both the ESRC and to those invited to attend. The purpose is not just about knowledge provision by the academics, but about knowledge exchange and transfer between the academics and attendees and should be of mutual benefit to both. If this is made more clear in the marketing of the seminars, it may lead to more interactivity and exchange taking place on the day.
• The venue of the seminar has to be considered carefully. The first seminar worked well regarding knowledge exchange, but the conference room was too small for the number of people attending, and resulted in people 'crammed' into the room. However, the second seminar was held in the largest of conference rooms, in a 'theatre style' presentation, and all agreed that the venue was too large and not conducive to interactivity. Although the feedback from policymakers attending the second seminar was very good, claiming they had found the information provided very useful, no exchange of knowledge took place, and the academics may not have benefited as much as they could have from it, by way of receiving knowledge and learning more about the Executive's role in these research fields. The next seminar is likely to be in the form of a 'round table' approach, targeting fewer, but more senior, policymakers to attend, and ensuring a more interactive exchange.
• However, it may be that by targeting fewer policymakers to attend, others who would be generally interested in attending may 'lose out'. The suggested format of a main event, followed by smaller discussion groups on particular topics, could be considered for future events.
• Hosting the same seminar twice within a short space of time may prove difficult to organise, particular for those giving up their time to present papers, but would enable SE analysts and policymakers to have a choice on which day to attend, and would have the additional benefit of cutting down the size of the seminar, facilitating the opportunity for interaction and knowledge exchange.
• Consideration should be given to holding the seminars over a longer period (for example, 3 hours rather than 2) with a coffee break midway for networking. No policy officials interviewed recalled lunch being provided, implying that they left at the end of the event and missed out on the opportunity of networking over lunch. If a coffee break was provided half-way through the event, this would provide an ideal opportunity for networking, interaction and informal knowledge exchange to take place.
• It would be helpful to those who were invited to attend, if brief papers on the background of the research taking place by the ESRC were provided before the seminar, so that attendees could come fully briefed and prepared to ask questions at the event.
• No papers, overheads of notes of the seminars were circulated to attendees following the seminar. After the second seminar, overheads were circulated only to those who requested them. It would therefore be better practice if notes and overheads could be circulated to all those who were invited (whether they attended or not), with details of the research findings and relevant contact details, so that any follow-up could be encouraged to take place.
• A mechanism (e.g. a brief questionnaire to those who attended) could be put in place where follow-up by OCR on knowledge exchange that takes place in these areas could be monitored and reported on, thus providing the ability to report on the outcomes of the seminars and the usefulness of the knowledge transfer that took place at the time and any resulting contributions to the ongoing work of the SE.

Future plans

32. Plans are in progress for the third seminar on Europe to take place, using the 'round table' approach. Meanwhile, the SPRO who was responsible for planning and organising the first 2 seminars has now moved post, and a new SPRO is in place. However, the previous SPRO agreed to organise the Europe seminar to maintain continuity, and will take account of lessons learned from the first 2 seminars, in planning and organising future seminars in the ESRC programme. Meanwhile, the new SPRO organised other, non- ESRC seminars for the Strategy and Delivery Unit, bringing in academics and practitioners to talk about their research and how it informs the Scottish Executive, building on the concept of knowledge exchange and transfer gained from the ESRC seminar programme.

Case study D
Knowledge transfer case study: working group for research into the legal services market in Scotland

Summary

• The Working Group for Research into the Legal Services Market in Scotland was an innovative, but pragmatic use of academic expertise in the enterprise of knowledge transfer.
• It was a model of knowledge transfer that drew on the role of academics as knowledge providers and practicing professional researchers.
• The Working Group also brought together external stakeholders, including professional and practitioner groups, with academic experts and the Scottish Executive in a way that sometimes hindered, but also enabled, the research effort.
• The development and success of the Working Group relied on and/or reflected, the existence of a research-attuned policy environment in this part of the Scottish Executive, but also an imaginative, innovative Social Research team.
• The involvement of high calibre, reputable academics was also key to its success and to the trust placed in it by Scottish Executive policy makers.
• The Working Group's research sub-group was based on a relationship between Scottish Executive researchers and academic experts/researchers, not just of knowledge providers and intermediaries or brokers respectively, but as partners.
• Although the consensus was that the Working Group was a success in knowledge transfer terms and that it surpassed expectations, there was a view amongst the researchers (both academic and government) that the research had taken a backseat to the stakeholder debates.
• The academic researchers and policy officials both felt that their involvement in the Working Group had encouraged a greater understanding of one another's perspectives, but some tensions had arisen over the timescales involved.
• Given the autonomy afforded to the research sub-group in designing and conducting the primary research for the Working Group, it was crucial that the researchers and policy officials communicated effectively to ensure that policy information needs were met effectively.

Introduction and background1. The Working Group for Research into the Legal Services Market in Scotland was imagined, developed and set up by the Civil Law Social Research team in the Scottish Executive. The Working Group met between March 2004 and March 2005. At the time of writing, it was understood that a key output of the Working Group would be a final research report to inform the preparation of Scottish legislation in response to European Union ( EU) legislation.2. The Working Group was established in response to the need to 'map' the Scottish legal services market, and in particular to identify any restrictive, anti-competitive practices that might contravene EU competition laws.

This exercise was recognised to be a massive under-taking, and one that could not feasibly be achieved, certainly not within the available timescales, via the usual route of commissioning external research or conducting in-house research. The key challenge was that it was impossible to identify at the outset which parts of the legal services market needed to be examined and thus to define manageable, self-contained case study research projects without spending considerable time - time that was not available - reviewing the extensive legal services literature.

It was therefore decided that the only viable way to complete it in the timescale was to exploit directly the existing knowledge and experience of 'academic experts', together with representatives of professional, practitioner and other relevant stakeholder bodies, via an iterative, step-by-step, process of documenting what was known and, in so doing, identifying what was not. The gaps in knowledge could then be filled by academic and Scottish Executive researchers working together and conducting primary research.

3. The full Working Group comprised 20 plus members (though this number fluctuated). This included 2 Scottish Executive Social Researchers, 4 academics, 3 Scottish Executive policy officials, together with representatives from a broad range of stakeholder professional and government bodies and voluntary and pressure groups (such as the Law Society, Faculty of Advocates, Office of Fair Trading, Consumer Council, Legal Aid and Citizens Advice Service). As such, the Working Group brought together policy makers, government researchers, academic researchers as well as practitioner and other stakeholder bodies and groups. The Chair of the Working Group was the Head of the Scottish Executive's Access to Justice and EU Act Division. The Working Group met every 2 months.4. The researchers on the Working Group - both academic and Scottish Executive - therefore made up only a small contingent, but were able to meet more regularly. They met as a sub-group after each full Working Group meeting and, the Scottish Executive researchers worked closely with the individual academics in their work for the Group.5. The Working Group was an innovative means of drawing on academic expertise, and one that overall appeared to be working well, although its success in this enterprise was yet to be formally evaluated. Assessment of the Research Working Group as an example of good practice in knowledge transfer6. While academic members formed only a small part of the Working Group, a key role also existed for the stakeholder representatives in contributing to the knowledge base and what follows in this paper is an assessment of the Working Group as a knowledge transfer case study drawing on 'academic expertise'. The findings are based on conversations with, and the views of, 2 social researchers, 1 policy official, and 2 of the academic experts involved in the Working Group.

7. Early findings suggested that the Working Group was successful in meeting its aims of drawing on academic subject knowledge and expertise and research experience in order to build up a useful evidence base on a large topic in a short period of time. In the policy official's view, "the Group has managed to open up the Scottish legal services market and shine a light on the various corners of it and I think that the report it will be able to produce will be an excellent evidence base."

8. The criteria for assessment employed here are some of those agreed by the Social Research Knowledge Transfer Group.

Preparation and planning

9. There can be little doubt that both the establishment and success of the Working Group reflected, amongst other things, a research attuned policy environment in this part of the Scottish Executive. One of the academic expert members of the Working Group talked about it being 'very receptive and welcoming of research' and much more so than other parts of the Executive. This provided the opportunity for an effective, innovative research team (and leader, in particular) to think imaginatively, but pragmatically, about the most effective and efficient way to secure the valuable input of academic experts. But more than this, the development and operation of the Working Group reflected a fair degree of trust in the Civil Law research team by policy officials, as well as in the Working Group, including its academic members, by the Scottish Executive members (research and policy).

10. The Legal Studies Research Programme was published in 2003 (for the 2003-2006 period). In this, academics and other researchers were invited to express interest in a number of key topic areas, including that of legal markets. Those who expressed interest, together with others who were known to have either done work or have a particular expertise in this field, were contacted and invited to tender for participation in the Working Group. 4 tenders were received, and all of these were accepted.

11. The invitation to tender for academic involvement made it clear that the Working Group was to be 'tripartite' - that it would not consist only or mainly of researchers. It was not - indeed, could not be - specific, however, in its description of the required input of the academic researchers (in terms of, for example, the topic areas to be covered and the timing of their work). 2 academics were contracted for 20 days work, and 2 more for 10 days each, though this was later extended to double the amount of contracted time from 60 to 120 hours, at a cost of up to £40,000. Payment covered all involvement with the Working Group (including attendance at meetings, writing papers and conducting primary research - see below). Although at this stage it was still not clear exactly what work would be required of the academic experts, it was anticipated that their roles would include both those of knowledge provider and practicing professional researcher.

Quality of the source and of the material12. Trust in the Working Group was based on the involvement of senior, experienced academic researchers of high calibre and good reputation. Tendering was via a detailed letter of application, with an extended CV, detailing relevant expertise and research experience.

13. This combination of qualities of the academic experts played a key role in the research work of the Working Group and its management (see below). Access to the kind of knowledge of the legal services market would not have been possible without senior academics who had extensive knowledge themselves, as well as contacts within the system, who would be prepared to share information, and in particular, to participate in the primary research conducted for the Working Group (other members of the Working Group were also able to provide such contacts for the primary research). The academics all had specialist fields of knowledge and expertise, all making a distinctive, but complementary, subject contribution to the Working group.

14. It was unusual for academics at this level to be conducting (rather than managing/overseeing) research themselves. The extensive practical research experience of one of the academics was particularly valued and it was drawn on to ensure the high standard of the research being conducted (though also to highlight its limitations).

15. A further key skill of at least some of the academic experts was their extensive policy research experience. One in particular had a long track record of contract research for the Scottish Executive (and Scottish Office before). Interestingly, this academic expert suggested that the invitation to participate in the Working Group was particularly attractive because of the possibilities that it could provide in yielding a greater understanding of the policy environment as well as offering insights into the policy making process. Moreover, following their engagement with the Working Group, these academic experts felt that they had achieved the above.

16. For the policy official on the Working Group this led to 'relevant input', which included verbal (at meetings) and electronic comment on drafted papers and debates, as well as the writing of literature reviews and research papers. Again, this particular group of academic experts were well valued and the importance of recruiting individuals of this calibre was recognised. This is outlined in the following quotation, "We've been lucky in the choice of academics and they've all had a good commonsense approach, understood what was needed and produced useable material. There's been no material from the researchers that we've simply side-lined. It's all been used and it's all been woven into the final report. So yes I think the researchers were well-chosen."

Interactions and relationship management17. As highlighted earlier, a key feature of the Working Group was that it brought together over 20 individuals from the fields of policy, practice and academia. Relationship management was crucial to the success of such a large and broad group of individuals and bodies, each with their own viewpoint and objectives, particularly since the main forum for discussion was face-to-face meetings. But what was clear was that the full Working Group and the research sub-group enjoyed a degree of autonomy, and the latter at least did not seem to feel that policy makers controlled the Group and its work. The objective of the work of the Group to proceed iteratively and flexibly appeared to have been met at least to some extent.

18. 2-hour meetings of the Working Group took place every 2 months. Before each meeting the Scottish Executive chairperson, policy officials and social researchers met to agree the agenda. Most of the papers for the meetings were also prepared by these individuals (particularly the key policy official). Administrative support for the Group such as the arrangement of meetings, circulating the papers, taking the minutes and so on, was also provided by the policy group). Although Scottish Executive members - both research and policy - set the agenda for meetings, the direction of the work of the Working Group was dynamic and driven by the Group itself.

19. Similarly, although the work was agreed by policy members of the Group, the programme of primary research was determined by the research sub-group, although, again in a way that reflected the direction of the Working Group discussions. This meant, in the words of one of the academic researchers, that while "the full Working Group meetings establish the broad parameters of the kinds of things to be explored,…the research sub-group within that has some leeway to think about how we will do this, how we will go about it, from what perspective… we have [also] carved out for ourselves, if you like, an area that I think was instigated by at least one [academic] member of the research group'.

20. The research sub-group met more often than the Working Group as a whole. This sub-group was managed by Scottish Executive researchers, but was based on a relationship of partnership and mutual respect. Indeed, the Scottish Executive and individual academic researchers worked together on some of the research projects. One of the academic experts found this to be one of the most enjoyable aspects of their involvement in the Working Group.

21. The full Working Group meetings, on the other hand, were sometimes tense, even explosive, affairs. This was not unexpected given that they brought together professional and practitioner groups with very different vested interests. Debates were occasionally heated and, in the words of one Scottish Executive researcher "horrendously stressful" to be present at and to manage. An unanticipated, but very welcome, role to have emerged for the academic experts on the Working Group, then, was one of arbitrator. This was based both on respect for their knowledge and reputation, and their position as being 'neutral' or at least independent of the Scottish Executive.

22. Debates were sometimes polarised between the stakeholder groups, but the academic researchers "helped the Group take a much wider view and much more evidence-based view." In some cases, their input even "helped to win round" the professional bodies to accepting that some of their rules and practices were anti-competitive and, therefore, to reform them voluntarily. The Working Group with its academic representation worked as a knowledge transfer activity, not just because it enabled academic knowledge transfer into the Scottish Executive for policy purposes, but also because it led to knowledge transfer between the stakeholder/practitioner and academic members (in both directions - the former were helpful in providing the researchers with contacts and privileged access to individuals and knowledge to aid the primary research).

23. The Working Group was particularly helpful for Scottish Executive policy officials who had, to some extent, avoided being cast in the role of 'the enemy' and having to take an unpopular line, "it means that the Executive doesn't have to argue themselves blue in the face or exert overt pressure that might be counterproductive. The professional bodies do accept a good argument and we've had good arguments, and pieces of analysis from the researchers."

Lessons learned: what has and has not worked well

24. The above discussion suggests that, in some respects, the Working Group for Research into the Legal Services Market in Scotland surpassed initial expectations - and hopes - of its Scottish Executive managers and members. There were a number of unanticipated, but positive and welcome, outcomes.

25. There were some tensions over timescales between the academic researchers and the policy officials, in particular between, "political deadlines and the research approach which was more measured. We move on different tempos." It was recognised from both sides that the final research report of the Working Group could be used in the preparation of legislation for Spring 2006, and that there was some urgency in getting the research completed for this. Some of the academic researchers found the compromise difficult and preferred to do a 'proper, thorough job' by conducting 'much more in-depth, comprehensive analysis' and by covering a wider range of topics/case studies. There was some understanding of the academic perspective on the part of the policy officials, but this was seen to be a legislative opportunity not to be missed, "I have a lot of sympathy with doing thorough, comprehensive work but we've got to prepare the bill by 2006." Moreover, the policy team was more than satisfied with the scope of the work and the quality of the research conducted.

26. Although the title of the Working Group (Working Group for Research into the Legal Services Market in Scotland [emphasis not in original]) suggested a central role for research in its work, the research members - both academic and Scottish Executive - felt that research was secondary to the stakeholder discussions, debates and 'battling for their position'. Indeed, one of the academic researchers suggested that, "the research is probably only just really starting to take off…..because research takes a long time to plan and do, while the debates don't." In Working Group meetings it was common that, "research needs and findings will only be turned to at the end of the meeting."

27. A further issue raised related to the need for better communication between the research sub-group and policy officials. Occasionally, research papers were 'not ideal in policy terms' or 'not quite the analysis we'd hoped for'. There was no suggestion, however, that there were barriers to closer working, rather a recognition by policy officials themselves that they might need to be clearer in communicating their policy needs.

Conclusion28. The Working Group for Research into the Legal Services Market in Scotland was an innovative, but pragmatic, knowledge transfer case study. It emphasised a wide ranging role for academic experts, encouraged closer working relationships between government and academic researchers, and brought together policy officials, stakeholders and practitioners, and academic and government researchers in a way that effectively and efficiently enabled the development of a (policy) 'useful' evidence base. This was not to say that the Working Group was without problems, but according to the policy makers and researchers involved it was a success.

Case study E
Knowledge transfer case study - what works for children?

Summary

Introduction

1. 'What Works for Children?' was part of the ESRC funded EvidenceNetwork and was a joint initiative between Barnardos City University and the University of York. Funding of £430,000 was allocated to City University for 4 years and the project ended in March 2005.

2. The stated purpose of the What Works? node was to promote positive outcomes for children and young people, in collaboration with practice, policy and academic colleagues in the UK and beyond. This was a group where there was already a strong central policy push for the greater use of evidence and where there was good research evidence of the difference that early intervention can make. The emphasis of the initiative was on the sharing and implementation of research evidence, rather than the generation of 'new' evidence.

3. The original objectives of What Works? were:

4. 'What Works for Children?' set out to operate predominantly at the implementation end of the evidence spectrum and focused on the following questions:

Following a development phase in year 1, the initiative worked with 6 Children's Fund programmes in Yorkshire to look at levers and barriers to implementing research evidence into service planning and practice. In the final year, the initiative worked more intensively with 2 of these programmes (Children's Fund projects were delivered through multi-disciplinary teams and boards and eventually all local authorities in England will have Children's Fund resources). What Works for Children? contributed from the earliest stage of project development through to implementation.

5. However What Works had a far wider reach than the Children's Fund. In addition to its comprehensive website, its outputs included Evidence Nuggets, research briefings, newsletters, peer reviewed articles, an evidence guide, seminars, practitioner workshops, and presentations. From 2002 until the project's conclusion, it provided an evidence request service. This was provided by the R&D team in Barnardos for their staff. The team also developed a research use assessment tool and a project planning and review tool to improve the transfer and translation of research evidence into practice. New training modules for creating and sustaining an evidence-based culture were also being developed.

6. In summary, What Works for Children? combined the roles of providing evidence for practice and developing mechanisms for, and facilitating the transfer of, that knowledge. At the time of this report, project funding was coming to an end and as part of a final report, ESRC team members will be reflecting on what has appeared to work more or less well in the course of the project, and possible models of 'What Works' developments for the future.

Assessment of What Works as an example of good practice in knowledge transfer

7. What Works met a number of the previously identified criteria for success in knowledge transfer.

Preparation and planning

8. As this initiative was funded by the ESRC, preparation and planning was part of the original application. However, the original design evolved during the development phase with intensive work on the ground proceeding alongside research evidence syntheses and website development. As What Works for Children looked for a "group to influence" the project worked very closely with the Children's Fund in Leeds (where the Development Officer was based.) It was anticipated that it would be easier to make an impact where new services were being developed rather than attempting to turn things around with established services. Most of the tools and resources currently provided by What Works have also developed over the course of the project.

9. Helen Roberts, formerly Head of R&D at Barnardos, and now Professor of Child Health at City University, together with the Barnardos R&D team set up the original What Works publication series as summaries of evidence for practitioners in children's services. These pre-dated the What Works for Children initiative and together with a report commissioned from Barnardos R&D team by the Joseph Rowntree Foundation on Making Connections: Linking Research and Practice provided some of the required background to the What Works for Children project. Barnardos itself has a research informed culture - research activity is integrated with policy and practice and, at the time of writing, over 30 research and development officers were employed by the organisation.

Quality of the source

10. Structurally the What Works initiative was one of the elements of the ESRC Evidence Network and it combined the practice-based skills of Barnardos with academic expertise in York and City Universities. The What Works for Children team had a strong relationship with Sandra Nutley and the Research Unit for Research Utilisation (a marriage of KT theory and practice).

11. The various products provided by What Works varied according to the type of need and were developed in consultation with users.

Evidence Nuggets were summaries of research evidence on specific interventions . The aim of these was to provide robust research which could help with decision-making. Some topics were identified by practitioners themselves. At the time of this review, 6 Evidence Nuggets had been prepared on mentoring, parenting, cognitive behavioural therapies, breakfast clubs, home visiting and traffic calming. Layout and readability were important features along with academic quality. These reviews were approximately 12 pages long and were structured for web exploration though they were available as pdf. files to download. The reviews were done systematically but they were not systematic reviews; they were peer-reviewed. Feedback, sought during distribution and at a separate seminar, was positive about the concept and respondents found the Nuggets useful and easy to read. The practitioners particularly liked having information about the costs of interventions, and having all the relevant information on an intervention in one place. The Nuggets were, however, time consuming and resource intensive to produce and this was an anticipated cost of this method. They were part of the R&D project and were expected to end with the project unless they were updated as new trials and systematic reviews were completed.

Research briefings were prepared in response to specific requests from practitioners. They were summaries of systematic database searches for research on interventions in education, offending and social care. At the time of this report, 5 briefings had been prepared.

Overviews: These were wider in scope and at the time of writing, 1 had been prepared. This described some of the context and content of interventions introduced to reduce inequalities in child and adolescent health. The Overview highlighted to practitioners what was known about effective (and ineffective) responses to address inequalities and provided guidance on the balance between targeted and universal services.

Other initiatives included an Evidence Request Log, a store of Q&As and quick searches for individual practitioners. This evolved as the What Works Development Officer formed working relationships with service planners and requests for research evidence and information started to come through to the research team. This was initially dealt with on an ad hoc basis, but when the number of requests grew it was decided that the service would be formalised and monitored. A log of questions and answers was developed along with a format for responses. The difficulties of getting research into practice cannot be underestimated even in a research attuned culture. When an evidence request service was advertised in the Barnardos Staff Bulletin, only a very small number of childcare staff submitted requests.

Other publications, conferences and presentations: What Works for Children? also produced a range of articles for peer reviewed journals and articles.

12. In addition to the development of research products, the presence of research and development staff on the ground as knowledge brokers was also important in building credibility and trust. They performed diverse roles, challenged decisions and claims to authority, and asked "why" questions to present the broader picture. In these debates, evidence on the costs of services was an important influence on evidence for decision-making.

Quality of the material

13. What Works for Children collated and reviewed existing work, using a peer review process for some of its products. This work was done by experienced academic researchers. Much attention was paid to presentation and fitness for purpose/relevance to the user (see above). The evidence service provided to the Children's Fund was targeted on 5-13 year olds, and on topics including educational attainment, crime reduction, victimisation and health inequalities. The role of evidence was both to provide research evidence on the effectiveness (or otherwise) of potential services and guidance on likely outcomes. In responding to the needs of practitioners and service managers The What Works team had, however, also become involved in training on evaluation.

Medium/delivery method

14. The What works for Children? website http://www.whatworksforchildren.org.uk was the key portal for practitioners and other users wanting to access products and services. It was clearly structured and easy to use. A recent survey of web users found that the main groups were children's services practitioners (32% of users), students (16%), academics (14%) and General Practitioners, GPs, (4%). Other users included managers, teachers and librarians. The site also had a range of international users.

15. In addition to research products mainly disseminated through the website, What Works developed a number of tools to support the interpretation and application of evidence. These included:

16. These products were developed in response to an analysis of the knowledge transfer 'problem' including the identification of potential barriers to using research evidence from the practitioners perspective viz. knowing where to look; being able to assess whether research evidence was good/useful or not; and understanding the relevance and potential application of research evidence. For example, it is frequently assumed that the internet has revolutionised access to research and evidence, but What Works' consultations with practitioners indicated that while the internet was a popular source of research, few of them had used (or heard of) online resources such as Pubmed and ERIC which could enable them to access relevant studies. (This problem may affect the What Works website itself - in a straw poll among analysts in the Children and Young People's team in the Executive only one in 5 was aware of it or had used it).

17. What Works for Children also provided:

• Training seminars on relevant subjects
• Training and support in evidence-based practice for service planners/management/practitioners, and
• Training events in critical appraisal skills, web searching skills, and support in asking relevant questions.

Interactions and relationship management

18. Through working closely with service planners and practitioners What Works for Children? gained a first-hand understanding of the many barriers to integrating research evidence into policy and practice. They found that service planners' understanding of the term 'research' was broad, and a range of activities were referred to in this category, including small-scale local evaluations and consultation exercises. A need for continued support in the areas of acquiring, assessing, adapting and applying research was highlighted.

19. Working alongside Children's Fund practitioners meant that the researchers "stay[ed] grounded in the real world". "People often get bogged down in processes and outputs" - and don't see the bigger picture; in this context the What Works team had a role to play in helping service planners make decisions. It should be recognised, however, that sometimes there was very little evidence available on effective interventions on complex problems.

What has and has not worked well

20. What Works for Children? made considerable progress because of its focus on the implementation end of knowledge transfer. It also provided fairly intensive support to practitioners in accessing and using evidence in the planning and rolling out of new services for children and young people. The project achieved all its objectives to differing degrees, but the design of the project was not one which will be capable of measuring the effectiveness of one approach against another. The direction taken by What Works? meant that it had perhaps done less on the interactive network of research, policy and practice idea; in particular relationships with policy e.g. DfES were sporadic. It was also unclear whether this initiative managed to develop replicable models of implementing evidence-based interventions due to much of the practical work being context specific.

21. The Development Officer post was to be funded beyond the end of ESRC funding in March 2005 to develop the role more widely and to influence the agenda of Barnardos and other agencies. The training modules will also live on.