Scottish Government

Main Findings

• Masterclass appears to have had a catalytic effect on the use of ICT, although there are other ICT initiatives and strategies in place that are also working to the same goal.
• The main impact to date has been an increase in the introduction and use of new ICT equipment and this is at least, in part, associated with Masterclass.
• Specific examples of good practice in ICT are already being provided by Masterclassers and these are likely to have increased by the second phase of the evaluation.
• The benefits of Masterclass to individual participants include improvements in ICT skills, opportunities for networking, sharing of good practice and championing ICT within schools or the local authority.
• Most support elements of Masterclass, including initial training and the SETT conference are well regarded. However, only around half have made use of the on-line facility and only around a quarter have used on-line facilitators.
• Co-ordinators actively support the activities of Masterclassers, although there are some resource constraints that need to be overcome.
• Many Masterclassers would like to see the involvement of more individuals across the education spectrum.
• There is a need to ensure that fresh impetus is given to Masterclass and there are a number of ways in which this can be created :
• Regular meetings
• Increased opportunities for networking
• Setting up interest groups both within and across local authorities
• Ongoing development of the website
• Further development of toolkit materials
• Wider dissemination of good practice

Introduction

The Masterclass Initiative was launched in September 2002 to support local authorities and schools in the development of their ICT integration strategies. LT Scotland is responsible for the delivery and management of the Masterclass programme. There are currently around 600 participants who are involved in the initiative and these have been selected across all Scottish local authorities. All are key staff with a role in the development of ICT nationally and / or locally. Participants in Masterclass include classroom based staff, centrally based staff and senior management in schools.

Each local authority has also appointed a co-ordinator to work alongside participants, the education authorities, teacher education institutes and the Scottish Library and Information Council.

The principal aims of Masterclass are to :

• Establish a shared vision of the potential and challenges of ICT in learning at all levels
• Influence, guide and support pedagogical change using ICT across Scotland
• Provide inspiration and encourage effective leadership in the embedded use of ICT in learning, teaching and management
• Develop and share expectations of good management of ICT at all levels
• Contribute to the ongoing development of a toolkit to support the effective use of ICT at all levels
• Create a community that is able to implement and sustain the vision
• Develop both national and local capability by encouraging community members to contribute to developments at both those levels
• Facilitate the dissemination of good practice in the use of ICT across Scotland.

Aims and Objectives of the Evaluation

The specific aims of the evaluation are to :

• Assess the effectiveness of the delivery of the Masterclass programme from the perspective of all the audiences currently involved in delivery, management, co-ordination and participation
• Assess the impact of the Masterclass programme on the participants themselves and on the settings in which they work
• Additionally, the evaluation aims to :
• Explore the ways in which Masterclass is likely to be used in the longer term
• Ascertain the fit between the Masterclass programme and other ICT initiatives that have also been introduced (such as Headstogether) and gain an understanding as to how ICT is used in the workplace and the benefits it can bring
• Understand the extent to which Masterclass participants are meeting the aim of becoming ICT champions for the future
• Identify any ways in which additional support should be provided by local authority co-ordinators or LT Scotland facilitators in order to maximise the effectiveness of Masterclass
• Identify examples of good practice for dissemination

Methodology

The research methodology for each phase of the evaluation comprised a number of key elements:

1. Qualitative research among Masterclass participants, non participants and co-ordinators within schools, Teacher Education Institutes and local authorities. A total of 52 in-depth face-to-face interviews were conducted across 24 organisations.
2. Qualitative research among Masterclass facilitators who assist in facilitating the website by using it regularly and developing materials and activities for it. A total of 5 in-depth interviews were conducted (four by telephone and one face-to-face).
3. Quantitative research among Masterclass participants and local authority co-ordinators. A total of 506 semi-structured interviews were conducted.

MAIN FINDINGS

Effectiveness of the Delivery of Masterclass

• There is a recognition that Masterclass is one of many ICT initiatives impacting on the education field at present.
• In the opinion of Masterclassers, Masterclass appears to have had a catalytic effect on the use of ICT (76% agree that this is so), but this must be tempered with the knowledge that there are other initiatives and strategies in place which are working at the same time as Masterclass towards the same goal.
• In terms of the specific elements of Masterclass that have been evaluated, most of the support elements are well regarded.
• The initial training is rated very positively in terms of quality (getting an average score of 7.9 out of 10 for this). Suggested improvements centre on some segregation between groups taking part to allow like-minded individuals to learn from each other - with 40% in favour of this, and a need for opportunities to consolidate what is learned (61% want more time for this). A minority (29%) found the initial training too intensive.
• Other training elements that have been used are well regarded, especially the digital video training and learning styles training (with scores of 8.7 and 8.2 respectively awarded out of 10) although relatively few have experienced this.
• Almost all Masterclassers (93%) have attended SETT. They also regard this as a very useful event (giving it a score of 7.9 out of 10).
• Only half (53%), however, have made use of the on-line facility with even fewer (25%) making use of the on-line facilitators. Levels of use of the various parts of the on-line community are restricted and none of the elements approach global awareness, even amongst users of this facility.
• Few participants had a clear idea of what Masterclass would deliver at the outset. In the absence of detail, assumptions were made and for the largest group of participants a key expected outcome is improvement in their own ICT skills (stated by 90%). Beyond this personal benefit, however, Masterclass participants hold expectations about the nature of the community that they are joining and articulate a belief that networking (92%) and shared learning opportunities (96%) will arise.

IMPACT OF MASTERCLASS

• In terms of impacts, first and foremost schools have noted an increase in the introduction of new ICT equipment, although this does not always derive from Masterclass. A lot of different types of equipment are identified, but digital technology in the form of projectors, cameras and video recorders emerge as likely to have been introduced as a result of Masterclass learning as have smartboards and whiteboards.
• In terms of other benefits, we have seen that Masterclassers hold expectations about improvements in their ICT skill levels, and the creation of a community of like-minded individuals that they will join. These expectations are largely realised. For example:
• Masterclass has helped improve ICT skills levels and confidence in these - with around 90% in the survey agreeing that this is an outcome.
• A large majority (90%) also believe that Masterclass provides opportunities for networking although only 76% of all participants claim they still make use of these opportunities. Local authority co-ordinators play a key role in facilitating networking opportunities and 79% of participants believe that they do help. But, there appears to be more need for networking opportunities at a cross local authority level.
• Some 88% of participants have had a chance through Masterclass to share good practice and opportunities for this are said to be increasing (by 65%).
• While loathe to regard themselves as ICT champions, Masterclassers are active in a wide range of champion activity albeit that this is more evident within schools than within their own local authority area or outwith the area. Masterclassers are most likely to say they have had an impact on the use of equipment within schools and to have improved knowledge about ICT but some are also influential on policy and developing best practice.

LEVEL OF ACTIVITY

• Overall, there is a feeling that fresh impetus needs to be given to Masterclass.
• Meetings for Masterclassers are held on average every two months, but this does seem to vary by area.
• Just 53% of those taking part in the survey have made use of the Masterclass on-line facility at all. Even amongst those who do use the facility, the frequency of use is low overall. Only 1 in 10 participants use the facility on a weekly basis, a further 10% every 2-3 weeks, but most (58%) use it less than once a month and half do not use it at all.
• Co-ordinators actively support the activities of Masterclassers in a number of ways which are essential to the development of networking opportunities, the sharing of good practice and the creation of ICT champions. However, co-ordinators face resource constraints and acknowledge that there are ways in which they would like to offer more assistance. Some way to overcome these barriers must be found.

BARRIERS AND SUGGESTED IMPROVEMENTS

• There is no widespread belief amongst Masterclassers that the selection process to find Masterclassers should be altered, but some believe it would be better if it is more open or that more information is made available so that volunteers can come forward.
• In terms of selecting candidates, one aspect to consider is the extent to which staff have time to disseminate their Masterclass learning to others. Those who have time ring fenced for ICT development activities within their current school role are more likely to be able to fulfil this aspect of activity.
• A majority of Masterclassers (83%) feel that they would like others to have a chance to do Masterclass.
• There is a need for more ongoing support with 48% saying it is hard to know how to put into practice what you learn and 71% favouring more meetings to help them.
• Meetings play a key role in allowing networking and sharing of good practice but occur with varying degrees of regularity.
• In terms of the on-line resource, there is a high level of non-use and either access needs to be improved or Masterclassers need to be assured that this is now better. Amongst users of the on-line community, there is a need for greater awareness of what is available within it, for it to be kept up-to-date and certain groups would benefit from better provision of information on specific topics.
• There is some demand for more opportunities to network (72% say this could be better) and 60% say they could share good practice more. The channels for doing this are similar in each instance and more meetings, a better on-line community and the provision of cover or time set aside to allow Masterclassers to do this will facilitate this activity.
• One third would like opportunities to meet others in their peer group.

RECOMMENDATIONS

The recommendations arising from this evaluation centre on ways to improve the role of the ICT champions, and improving the support developed in relation to Masterclass and from local authority co-ordinators and at a national level.

Improving the role of ICT champions

1. There is a tendency for champion activity to be dismissed in conversation, arising largely from a dislike of the champion term. A more appropriate term might be considered.
2. A beneficial form of support to champions is making key speakers available to help them. Masterclassers would welcome a planned programme of talks by speakers such as Alan November.
3. There is a need to ensure that the initial motivation and enthusiasm for Masterclass is maintained amongst Masterclassers. The on-line facility needs to be given fresh impetus and Masterclassers should be encouraged, in fairly rigorous terms, to access the facility on a regular basis.
4. Further support for the status of Masterclass should be provided at every opportunity, through the SETT conference and by publicity and write ups of good practice arising from Masterclass. This should be made available as widely as possible with attention drawn to the presence of Masterclassers in schools or the education department.
5. Learning from Masterclass should be made more easily accessible to others with an interest in ICT. Champions could be encouraged to involve others within their school in ICT development and local authority staff could encourage this by setting goals.

Improving support related to Masterclass

1. Some local authorities appear to be struggling with the impact of Masterclass commitments on the other duties of co-ordinators. Some indication of the support required from the local authority might ensure that this is better established and more forth coming.
2. The on-line facility needs to be kept up to date with fresh innovative thinking to ensure that it is regarded as a key facility to be used on a regular basis.
3. Any training of new Masterclass participants also needs to make sure that new Masterclassers realise the importance of the on-line community.
4. Undoubtedly, the maximum impact from Masterclass is found where Masterclassers have been able to make use of the learning provided. The development of some exercises to encourage Masterclassers to consolidate their learning from the training would be helpful.

Improving the support from local authorities

1. The role of local authority co-ordinators in impacting on the outcome of Masterclass is significant. The study indicates that this aspect needs to be adequately resourced in terms of :

• equipment for funding Masterclass driven initiatives
• integration of Masterclassers into strategic development and decision making within schools / local authorities
• provision of support for champion activity through regular Masterclass meetings and other opportunities for sharing such as conferences, exhibitions and joint training

2. Some guidance arising from this study might usefully be developed for local authority co-ordinators indicating the mix of very effective roles that they can provide in support of their Masterclassers.
3. Additionally, co-ordinators might consider ways of devolving responsibility for some support activities to others within their area.

Improving support at a national level

1. One of the main gaps at present appears to be in the cross-pollination of ideas, learning and knowledge between local authorities. LTS could consider ways in which it might provide a mechanism for ICT co-ordinators to exchange learning and ideas through the creation of local authority peer groups, for example.
2. Consideration might be given to ways of developing a pool of subject champions, to reduce the feelings of isolation experienced by some individuals.