Scottish Government

Executive Summary

The Assessment is for Learning Development Programme (AifL) was designed to bring together the various purposes of assessment into a single coherent framework which would answer questions of accountability, standards and the monitoring of progress and performance, but which also emphasised the role of assessment in supporting individual pupils' learning in the classroom.

The Assessment is for Learning Programme

The programme combined top-down and bottom-up development, using a project-based approach. Ten projects were identified which, together, addressed the aims of the programme, covering aspects of formative and summative assessment, recording and reporting. While the theme of each project was determined at national level, the precise nature of the work undertaken within individual schools or school clusters was determined by the staff in the schools working with local authority personnel. Most classroom-based projects followed an action research approach, with planning, reviewing, reflecting and modifying phases, culminating in a case study report. Schools that participated were given financial support, for example, to provide supply cover, materials and resources or staff development activities. Support was provided by local authority personnel, project development officers and members of education faculties (or their equivalent) in the universities.

The programme was designed to develop assessment practices that would better meet the needs of all concerned; the wide-ranging and multi-faceted nature of the initiatives required flexible and responsive management processes. The Assessment Action Group ( AAG) maintained strategic oversight of the programme and its evaluation and included representatives of education authorities, schools, university faculties of education, parent groups, professional associations, the Scottish Qualifications Authority ( SQA), Learning and Teaching Scotland ( LT Scotland) and the Scottish Executive Education Department ( SEED). The Assessment Programme Management Group ( APMG) was responsible for the 10 projects within the programme. Specific conferences, seminars and staff development events were organised nationally to provide opportunities for participants (practitioners and others) to meet, review and reflect on strategies and progress. These took place across the country in order to maximise access by authorities and schools.

The Projects

The Evaluation

An evaluation of the AifL Programme was commissioned by the Scottish Executive and carried out by the Quality in Education Centre, University of Strathclyde. The evaluation began in January 2003 and aimed to determine the effectiveness of the programme in respect of its processes, products and short-to-medium term impact. It was undertaken in two main phases. Phase 1 involved analysis of existing documentation and questionnaires to the four main groups of participants - teachers, development officers, local authority co-ordinators and higher education representatives. Interviews were conducted with various key individuals. A sample of schools participating in each project was identified for further study, providing case study data across a range of projects. Phase 2 focused on the use of the outcomes of the projects by practitioners, managers and policy makers. It included analyses of reports and project materials from schools, further interviews with case study schools and a survey of schools including a number of Associated Schools Groups. (The programme had moved on to working with clusters rather than individual schools and the evaluation reflected this.)

Survey Samples

Twenty-six schools were visited for the case studies.

While the evaluation of the AifL Programme had specific aims, it remained responsive to changes in the overall programme and the design was modified as appropriate, reflecting the philosophy behind the programme itself which gave considerable freedom to schools and teachers to develop practice within their own context at a pace and in a manner that reflected local needs.

As evidence on the implementation of the various projects became available, the management team responded to expressed needs and feedback from those involved. Thus the data from the different phases report what was the case at the time; many of the issues identified by participants were addressed as they came to light through the evaluation and other mechanisms.

Key Findings

1. The programme: an initial analysis (February to March 2003)

• While ten individual but linked projects formed the basis of the programme, the initial focus was on the development of Personal Learning Plans as a means of supporting a unified system of recording and reporting. The projects were devised as a means of making developments 'manageable', but all had a role in developing a coherent national system of assessment.
• All local authorities were involved in Project 1 and at least one other project. Over 170 schools and more than 500 teachers took part in the pilot developments. Each project had an associated project development officer and representative from higher education.
• While all participating schools were expected to undertake a baseline audit of assessment, not all did so, with some schools indicating that it was inappropriate and did not fit within the school development planning cycle.
• Milestones were set for schools to produce case study reports and examples of products and some schools found it difficult to meet these. However, as the Scottish Executive were concerned that schools grow and develop at a pace to suit their own needs and context, the timescales were treated flexibly. Because of this most projects at school level took a little longer than originally anticipated.
• Those involved in PLP projects varied in their understanding of the purposes of PLPs and the implication for developing ICT support. A common reporting framework was not produced as planned, with resultant consequences for the focus of Project 9, which was intended to build on that framework. Further development work on PLPs was undertaken, with a proposed framework being circulated for consultation in September 2004.
• From the documentary analysis conducted between February and June 2003, it appeared that involvement in Projects 1, 4 and 9 was focused exclusively on project-specific developments (viz: no mention of PLPs in relation to Project 1 or formative assessment in Project 4). On the other hand, some participants in Projects 2, 3 and 8 were recognising links with formative assessment.

2. Phase 1 (January to December 2003)

The key participants

• Just under two-thirds of the pilot schools participated in the evaluation surveys, with a higher proportion of primary than secondary schools responding. All projects were represented and all but one local authority.
• Most local authority co-ordinators held multiple roles, which caused tension in time management and availability. In addition, many had not appreciated the scale of the programme.
• Development Officers were committed to the programme but were holding varied and demanding remits, particularly in terms of liaison across stakeholders.
• Higher education representatives had varied roles in terms of demands on their time and the activities in which they became involved. This was the least clearly defined and understood of the three central roles. Some felt integrated into the project while others felt more isolated.

Impact on practice

• Headteachers/senior managers indicated relatively clear strategic objectives for the staff and pupils as a result of their involvement in the project; teachers' objectives were more at the operational, day-to-day level.
• Almost two-thirds of schools indicated that they were already trying to develop assessment practices along the lines of the project(s) and so were building on existing strengths.
• The projects involved, variously, pupils across the pre-5 to S6 stages, although there was evidence of greater involvement at P5 to P7. There was an emphasis on literacy and numeracy or combinations of curricular areas, although other individual subject areas were also identified.
• Views on the impact of involvement with the AifL Programme were, in the main, positive with regard to changing practice, pupil involvement and relationships with parents (relevant to the specific project).
• Those who engaged purposefully with the programme, particularly those projects concerned with formative assessment strategies, showed considerable commitment and enthusiasm as a result of their involvement.
• A range of benefits was identified by both headteachers and teachers, depending on the individual project, with raised awareness, increased knowledge and understanding of assessment and increased collaboration across groups reported. Local authority co-ordinators, HE representatives and project development officers all noted that a key benefit of the programme had been enhanced teacher motivation and enthusiasm in the participating schools. The sense of ownership engendered by the 'bottom-up' element of the programme contributed greatly to this.
• The main challenge was time. There are two aspects to this. The first is concerned with the constraints resulting from school timetables and schedules that limit flexibility and responsiveness to initiatives. The second, for many teachers, is finding the time to become familiar with and access sources of information ( e.g.LT Scotland) on assessment and implementing change. Other challenges included engaging all staff and maintaining enthusiasm.

Impact on meeting information needs

• Developments in assessment practices and procedures were beginning to meet information needs more effectively than previously for class teachers and pupils and, to a lesser extent, parents, with less progress towards meeting the needs of other teachers and the local authority. Many respondents indicated that it was too early to make judgements on the impact of the project in this area.
• Only 53 teachers from the sample had been working on PLPs, with the majority of them indicating that their use was demanding but manageable for both teachers and pupils. These teachers indicated that PLPs were meeting teachers' and pupils' information needs more than those of other groups. It was unclear how the use of PLPs in schools could meet the information needs of local authorities.

Convergence of assessment arrangements

• Lack of awareness/knowledge of projects other than those in which their authority was involved caused concern to LA co-ordinators with regard to future dissemination and implementation across authority schools.
• Reservations were expressed about the extent to which key groups and participants understood how the 10 projects linked together to form a coherent programme. Teachers knew very little about projects other than the one in which they were involved, although headteachers indicated they were more aware of the other projects.

AifL Programme issues

• Collaboration and community of practice: The three key support groups were beginning to work together and with policy makers, although each group identified limitations. Where working together occurred, it was reported as a positive and successful experience, albeit relatively novel. There was need for ongoing development of these relationships. Although teachers were benefiting from working more in collaboration with colleagues within their schools, many had not yet had the opportunity to liaise with colleagues in other schools or at conferences, and so were not yet participating in a wider community of practice. Face-to-face meetings (in groups or one-to-one) was the preferred method of communication in order 'to get work done'. The website and web-based forum had been used by a very small number of participants.
• Funding: The majority of schools had found the additional funding adequate for their purposes, with additional supply cover being the main resource purchased with it. Some schools had bought new information and communication technology equipment and various other resources and materials.
• Sustainability: Most schools considered that the initiatives would become part of school development planning and resourced from school budgets; only a small number considered that they would need extra funding.
• Planning: Similar themes were beginning to emerge across the groups of respondents. For example, local authority co-ordinators, headteachers and teachers, on reflection, felt that they would have benefited from a longer lead-in period and more information on the programme as a whole as well as on individual projects.

3. Phase 2 (January to December 2004)

Developments and progress during 2003 to 2004

• Many local authorities had appointed additional staff to take forward AifL developments: 14 had appointed development officers during 2003 to 2004, with a further 6 in 2004 to 2005. Roles varied, but most were responsible for taking forward operational aspects such as organising in-service and supporting schools, while assessment co-ordinators focused on strategic management. In some authorities these roles were shared. Ten had not appointed additional staff.
• All authorities had delivered a programme of staff development on assessment-related issues. Participation in staff development across authorities varied from all schools in some authorities to one-fifth of primary and one-tenth of secondary schools in others. In about one-third of authorities more than 40% of primary teachers were involved. However, in the majority of authorities, fewer than 20% of secondary and special needs teachers participated. The main focus was formative assessment, with PLPs mentioned by a few authorities.
• The majority of authorities had engaged clusters of schools in Associated Schools Group developments, with 30% of identified school clusters becoming involved. Again the main focus was formative assessment.
• The HE representatives had all engaged in assessment-related research and all identified ways in which AifL had been introduced into Initial Teacher Education and other teacher education provision.
• In response to the school survey, 16% of the targeted schools reported that no further developments had taken place for a variety of reasons and so they could not complete the questionnaire.
• Only 53% of the original pilot schools responded to the survey. In the majority expansion of the original work had occurred or was planned, mainly through involving more teachers, other year groups and areas of the curriculum. Seventy-four percent of these schools reported undertaking developments of aspects of AifL other than the work of their original projects.
• Schools involved originally in Project 1 (formative assessment) were more likely to be expanding their original work. For those involved originally in other projects, formative assessment was the most frequently mentioned new development, followed by PLPs.

The impact on policy and practice

• Two-thirds of headteachers thought that the AifL Programme had substantially influenced their development planning, though primary headteachers thought this more than secondary headteachers.
• With respect to National Priorities, headteachers were more likely to indicate that AifL had impact on Achievement and Attainment than on other priorities, although primary headteachers were more likely than secondary headteachers to see AifL complementing all aspects of the curriculum.
• Involvement in AifL was encouraging the development or revision of school assessment policies.
• Headteachers and teachers agreed that there was increased awareness of research related to teaching, learning and assessment, clearer understanding of assessment, changes to classroom practice, more varied approaches to assessment in use, improved feedback to pupils and more meaningful discussion with pupils about their learning. Generally, this agreement was stronger for those who had been involved in Project 1 from the beginning.
• The greatest challenges to introducing change were time and engaging all staff. Time was at a premium both for preparing materials and engaging in dialogue with colleagues, due to competing priorities and also, in some cases, lack of supply cover. Agreement that there was resistance to new developments and difficulty in changing practice had increased since the first survey. Maintaining enthusiasm and engaging new staff was more challenging as the programme progressed.

The impact on pupil motivation and attainment

• Some teachers thought it was still too early to comment on benefits to pupils but, for those who did, there was broad agreement that pupils had become more actively involved in their learning, were better equipped to assess their own learning, had shown increased confidence and self-esteem and were themselves positive about the changes. There was less confidence in the ability of pupils to set targets and engage in peer-assessment. Generally, there was stronger agreement on all of these issues from those involved originally in Project 1.
• About a quarter of teachers indicated that they had evidence of improved pupil attainment, but the majority thought it was too early for this. This had changed little from the 2003 survey.

Staff development

• As noted above, local authorities had been actively involved in promoting staff development in assessment during 2003 to 2004.
• The main types of staff development in which people had participated were peer-delivered 'in-house' events, joint events with cluster schools and local authority-delivered courses. From the local authority perspective, peer development and events involving national experts had the greatest impact, followed by local authority-delivered events.
• There was broad agreement across all participants that progress had been made in providing extensive staff development, though with the cautionary note that much work was still required to reach the wider teaching community.
• The style of staff development encouraged in AifL and, in particular, by Project 1, took the form of 'action research' which involved recall days, discussing with colleagues, reflection and writing case study reports. There was evidence that not all headteachers and teachers had attended national events or contributed to case studies.

PLPs and meeting the needs of pupils, parents, teachers and others

• The development of PLPs had been slow, with only 15 authorities indicating that they were in the authority improvement plan. There were different opinions as to the purpose, and hence content, of PLPs, with local authority, HE and school representatives all contesting their suitability for recording and reporting purposes. Rather they were a tool to support learning and therefore related more closely to developments in formative assessment.
• The specific focus on PLPs in meeting information needs was not addressed in the second survey, due to the lack of widespread PLP development. However, a general question was asked on how all aspects of AifL met these needs. The greatest contribution was in meeting the information needs of pupils and the class teacher, with teachers, in particular, agreeing that developments had improved these aspects. About 40% of the headteachers and about a third of teachers thought that substantial progress was being made with regard to meeting the information needs of parents. Some thought it was still too early to make judgements on how AifL contributed to meeting information needs.
• Benefits to parents in terms of increased contact with the school and teacher, and better understanding of how they can help their child, were project-specific, with participants in Projects 2, 3 and 9 recognising these benefits and others indicating that this was not a focus of the development. Some indicated that relationships had always been good and AifL had had no impact; others recognised that improvements had occurred but not necessarily because of AifL.

Convergence of assessment arrangements

• An important aspect of moving towards a coherent system of assessment is the drawing together of the contributions of the separate projects. At the beginning of session 2004-2005, both headteachers and teachers reported a lack of awareness of projects other than the one in which they had been involved. They knew more about formative assessment and PLPs but reported knowing little or nothing about Projects 3, 4, 5, 8 and 10; 70% of headteachers and 49% of teachers indicated that they had a clear understanding of how the projects linked together to form a coherent system.
• The issue of bringing together classroom assessment and more formal means of assessment such as National Assessments, AAP and 5-14 testing was less clear. At the time of the second phase of the evaluation there was a lack of clarity regarding the latter three, as the outcomes of the 'Consultation on Assessment and Testing: 3-14' had not been published. There was evidence of perceived conflict between formative assessment and the summative approaches of National Assessments and other tests and examinations.
• Local authority co-ordinators expressed concern about difficulties schools had experienced in the use of the online National Assessment Bank. Almost all headteachers thought that using the online National Assessment Bank meant additional workload and costs to schools, with only a quarter agreeing that it was easier to manage than the previous approach.

AifL Programme issues

• Collaboration and community of practice: AifL developments had brought different groups together to work in new relationships, though not all within the same frame of reference. Both LA co-ordinators and HE representatives identified the opportunity for networking as a major strength of the programme. However, while authorities were working with each other and sharing ideas and some of the HE representatives were working cross-institutionally, relationships between local authorities and HEIs were still developing. The main sources of support for teachers remained within their own schools (management and other teachers), though schools were working more with other schools. However, cross-sector and wider networking was less developed for the original pilot schools. The Associated Schools Groups were beginning to encourage greater links within clusters and across sectors.
• Practitioner-led developments: The central role of the classroom teacher in taking forward developments and engaging in practitioner research was seen as a major strength of the programme, contributing to successful outcomes in many schools. This had led to high levels of commitment and enthusiasm. The process of 'growing policy', that is, allowing it to emerge rather than imposing it from the top down, was appreciated.
• Funding: The provision of funding to the pilot schools was recognised as a major strength of the programme, with almost 50% of headteachers saying they would not have undertaken any of the developments without it. The main use was for the purchase of human resources - for supply cover or to pay teachers to work in their own time.
• Sustainability: About half of the authority representatives indicated that AifL would be 'embedded' into teaching and learning policies and plans and that authority funding would be allocated to ongoing developments. Others thought that additional funding was necessary to maintain the level of development and to continue with developments in PLPs, use of ICT and reporting. Headteachers were also divided between those who thought they could sustain developments from their existing school budgets and those who saw the need for ongoing additional funding.
• Monitoring progress: Around half of the local authority co-ordinators indicated that AifL issues were integrated into existing quality assurance procedures.
• Planning: A recurrent theme from local authorities and headteachers was the mismatch between improvement and development planning and the funding cycle.

4. Conclusions

A number of key points emerged from the evaluation, some related to the design of the programme, others concerned with the outcomes.

• The combination of 'bottom-up' and 'top-down' approaches was effective, although the degree of choice allowed in determining projects makes it more difficult to generalise across the programme.
• Several factors were identified as contributing to the success in some authorities/schools. These included:

• funding
• supportive networks
• staff development
• expert input, at appropriate points.

• Small-scale research projects engage teachers, giving them a sense of ownership and control over developments and the management of change.
• Small-scale projects can result in fragmentation if the parameters are too wide.
• Funding to 'kick-start' the programme was greatly appreciated and primarily used for staffing. This allowed schools to devote significant amounts of staff time in the early stages of the project.
• Significant change occurred in schools and authorities that embraced the opportunities offered by the AifL Programme.

Careful consideration should be given to the following issues in relation to the ongoing development of AifL:

• An important factor in effective development is the way in which local authorities take ownership and strategic leadership of the initiative, following the model used initially with the pilot schools, i.e. practitioner development supported through dialogue with colleagues, wider networks and communities of enquiry.
• The co-operation and collaboration between authorities and schools with representatives from the faculties of education, as relevant, is beginning to show signs of developing constructively and is to be encouraged.
• There is a need to resolve uncertainties around the purpose and content of PLPs. This might be best achieved through practitioner development and encouraging dissemination through relevant networks, as is beginning to happen.
• There is a need to continue to develop understanding as to how different initiatives, including AifL, contribute to national priorities and, in particular, to improving teaching and learning with a view to engaging learners and raising attainment. This needs to be supported nationally.
• Similarly, understanding of how classroom assessment can serve both formative and summative purposes remains patchy, particularly with regard to how they relate to externally set and designed assessment. There is a polarisation of the two purposes which is not helpful in understanding, for example, how external assessment might contribute to formative assessment within the classroom.
• If the Toolkit is to be developed so that it is more readily used at school and ASG level, some provision of resources would be useful to enable local authorities/schools to contextualise it for local use and development.
• This is just a beginning, albeit a positive one. However, it may be necessary for the Scottish Executive to maintain a high profile for AifL, through national and regional events and ongoing publicity.

5. Further research

Some issues that would benefit from further research were identified during the evaluation:

• Strategies to engage secondary schools more meaningfully in the programme
• The impact the programme has on pupils as more schools and teachers become involved
• The impact the programme has on teaching strategies and teaching cultures
• The impact of PLPs as a unifying concept to support the needs of pupils, parents and teachers
• The development of networks and communities of practice
• Staff development approaches most likely to lead to successful implementation of the programme in schools.

The above impacts of the AifL Programme should be assessed on a longitudinal basis, in an attempt to determine long-term changes in practice.