Scottish Government

4.0 CLASSROOM OBSERVATION

4.1 Aims

The aims of the Classroom Observation were:

i) To document and analyse teachers' drug education practice in terms of the approaches and resources in place over a wide range of drug education topics;

ii) To observe how pupils used the information provided in the drug education lessons in the following contexts:
- in clarifying or defining attitudes to drugs and drug use;
- in thinking through social and personal consequences associated with drug use;
- in developing skills and strategies to deal with potential situations in which drugs may to offered and used;

iii) Where specific packages were used, to assess the amount of variation in their use across schools and teachers, and to identify the key dimensions most likely to be changed by schools and teachers;

iv) To identify the impact of extraneous factors on drug education such as classroom layout, the teaching staff delivering the lessons and unplanned occurrences and disruptions to lessons;

v) To assess the extent to which the drug education content, teaching and learning strategies, approaches and resources actually employed by teachers reflect the features of drug education associated with effectiveness.

4.2 Methods

The classroom observation element of the evaluation obtained data through the use of three research instruments:

• An observation schedule ( Appendix A);
• A post-observation proforma to explore teacher response to the lesson (Appendix B);
• A contextual information proforma to cover additional points of interest about the school generally ( Appendix C).

The development and format of each of these research instruments is now described.

4.2.1 The Observation Schedule

Development

The development of the observation schedule was undertaken in a local authority not sampled for the evaluation's observations. A number of classroom observations were conducted to inform the researchers on the following issues which the schedule would be required to address:

• The levels of flexibility needed in the design of the schedule to ensure that both qualitative and quantitative data could be reported;
• The likely extent of the parameters of drug education lessons generally and particularly in relation to methodological differences between primary and secondary lessons;
• The extent of the range of content and topics covered in drug education across the primary and secondary age-ranges.

Three lessons in a primary school and three in a secondary school provided a basis for the development of the schedule. None of these lessons were video-recorded. However, video-recordings of drug education lessons in a Scottish authority made in the previous year ⁶ were used for piloting the observation schedule in its draft form.

Training was also provided to members of the observation research team in the use of the observation schedules. This was conducted over two days and also covered orientation to the project and general points on research-based observation techniques.

Structure of the Observation Schedule

In its final form, the schedule allowed the observer to record standard lesson descriptors such as class grouping, classroom layout, how long the lesson took and basic factors such as the size of the class. The schedule layout enabled notes to be made on the introductory processes prior to the activities, with space for the narrative of how this was done and for notes on pupil and teacher interactions of interest. The focus was on whether or not lesson objectives were explained and whether or not links to previous lessons or previous learning were made.

The main part of the schedule was designed to enable notes to be made on up to five activities (with additional pages to be used if necessary). As for the introductory section, this was also laid out to enable a narrative to be recorded, with observers invited to comment on:

• The content of each activity;
• What the activity was designed to address, e.g. whether its main focus was on information, skills or elements like problem solving;
• The type of activity selected e.g. was it a prioritising exercise or addressing attitudes clarification;
• Whether there were specific packages and resources used;
• How the young people were organised over the course of the activity.

Space was also provided for recording pupil and teacher interactions, particularly how questions were asked. Also of interest was what could be gleaned from the activities on broad approaches, including learning and teaching strategies in place across activities.

The schedule ended with a section on what happened at the end of the lesson. Of interest was whether or not the teacher had undertaken any reviewing activities such as reminding the class of what they had covered or what they had learned.

The schedule allowed the observer to record any unplanned and unexpected factors and/or incidents that could have influenced the activities. It was also structured to enable some post-observation reflection to be made by the observer on the following:

• The overarching concept behind the lesson as a whole and of its various activities;
• The extent to which the observer felt that the main messages were being understood;
• The extent to which the lesson impacted on the pupils' understanding of drugs;
• The proportion of pupils engaged during the lesson overall;
• The suitability of the classroom for the lesson.

4.2.2 The Proformas

The proformas were designed to provide additional contextual information to support the data obtained during the observations, which was designed to be the main instrument of the classroom-based data gathering element of the evaluation.

The focus of the proformas was as follows:

• The Post-observation Proforma covered teachers' views about the lessons which had been observed and enabled comments on relevant broader issues to be recorded. There was also space on this proforma for additional reflective comments made by the observer.
• The shorter Contextual Proforma was structured around broad headings on drugs, the school and the neighbourhood. This proforma was for use with respondents in school senior management posts.

4.3 Sample

4.3.1 Planned Sampling Frame

Selection of Local Authority Education Departments

Following the Survey (see above), five local authorities were initially selected to participate in the Classroom Observation. The sample was selected according to two main criteria:

• Geographic spread: representing both urban and rural locations and east, central and west regions of Scotland.
• Reported drug education approaches: including both local authorities which are seeking to encourage a standardised approach to drug education across all their schools (e.g. local authorities which have recommended the use of a specific curriculum or package), and those in which there is more diversity across schools.

Selection of Schools

Within each of the five local authorities, six schools (30 in total) were to be selected for observation, subject to the agreement of the headteacher and teachers involved in drug education. These six schools selected in each local authority were to comprise three primary schools, two secondary schools, and one other school, which could include independent schools or schools addressing special or additional needs. (See Table A4.1)

Table A4.1. Planned Sampling Frame

*Others to include independent schools both primary and secondary, and schools addressing special or additional needs

Schools were to be selected according to the following criteria:

• When drug education was to be delivered: The study's timing required that this part of the evaluation needed to be conducted over the later months of 2004 and as early as possible in the first months of 2005.
• Drug education delivery arrangements: the sampling was planned to include schools which delivered drug education through teachers with PHSE specialisms and those who had not received PHSE training or had built up experience in this area.
• Socio-economic status: it was anticipated that the schools used would cover a broad socio-economic span, based primarily on figures for free school meal entitlement.
• Ethnicity: this was not specifically used as a structured sampling factor, but it was anticipated that schools with different levels of ethnic minority representation would be part of the observation programme.

Selection of lessons

From the 30 schools, 100 drug education classes were to be sampled for observation. These lessons were to include lessons delivered across the range of both primary and secondary year-groups (Table A4.2).

Table A4.2. Planned Observations by Year Group

4.3.2 Achieved Sample

Factors Influencing School Selection

The schools were to be selected only from among those which had responded to the Survey, with the Survey analysis providing information on which selection could initially be undertaken. The local authorities to be used would be required to meet the following criteria:

• That 3 primary schools (statutory) could be identified which offered drug education during September - December 04 to any one year group within P1 - 3 and to any one year group within P4 - 5 and to any one year group within P6 - 7;

and

• That 3 secondary schools (statutory) could be identified which offered drug education during September - December 04 to S1, S2, S3 and S4;

and

• That one other school could be identified which offered drug education during September - December 04 and where three different lessons could be observed.

However, it became clear that none of the local authorities had enough schools responding to the Survey which met the above criteria. Because of this, the time-span for delivering drug education lessons was extended to March 05, which enabled four local authorities to be identified and a fifth broad area was added by combining two adjoining local authorities. Towards the end of the observation period, an additional local authority was also involved to ensure that the requisite number of observations could be undertaken.

Initial contact letters were sent to all the schools meeting the required criteria and follow-up telephone contact was set in place to check that the following conditions were in place:

• That the school was willing to support the observations;
• That the terms in which drug education was to be delivered were compatible with the evaluation timetable;
• That the teachers involved were informed about what was expected and that they had agreed to be observed.

Schools involved in the Observations

In all, 140 schools were contacted by letter inviting them to support the classroom observations. From these, 60 schools (44 per cent) indicated they were willing to take part. (This percentage varied between 33 and 59 per cent across local authority groupings.) However, some of these schools were delivering drug education to only some of the year groups of interest (as distinct from all the year groups of interest being available for observation in the same school) at the critical times for the observations. In the end, a total of 40 schools took part in the observations, selected from across five local authority groupings:

• Aberdeenshire, supplemented by Aberdeen City;
• Fife;
• Glasgow City, supplemented by Edinburgh City;
• Perth and Kinross;
• South Lanarkshire.

In order to observe the required 100 lessons across these local authority groupings within the time-scale of the evaluation, the number of schools in which observations took place was, at 40, ten more than had originally been planned.

Number of Observations and Year Groups Observed

The total number of observations undertaken was 100 (See Table A4.3 below.) Slightly fewer primary observations were done than was anticipated. However, a number of composite groupings in three primary schools enabled each of the seven primary years to be observed in two observations (as distinct from an anticipated three selected years per primary school). In other schools, smaller class groupings also provided a broader range across fewer observations.

Table A4.3. Observations by Local Authority Groupings

*Across three schools, 1 in Aberdeenshire and two in Fife, six observations were of composite classes, (Ps 1 - P3/4 and Ps4/5 - P7, giving two observations where three might have been expected.

Eighteen of the observations were of composite classes, all in the primary sector (Table A4.4).

Table A4.4. Year Group Combinations in Composite Classes Observed

Of the eighteen composite classes, the majority were a combination of two classes. No observations were undertaken in classes where more than four year groups were combined in the same classroom, as indicated below. Overall, there were:

• Ten observations where two year groups were combined;
• Four observations where three year groups were combined;
• Four observations where four year groups were combined.

The composite classes observed were distributed across local authority groupings as follows:

The number of observations of composite classes provided the evaluation with information on what occurred within slightly more year groups than observations of single year-group classes would have done. This is outlined in Table A4.5. Fifty five per cent of the observations included primary classes, with 41 per cent covering secondary years one to four. The remaining four per cent of year groups observed formed the "other" category, either from the independent sector or from schools addressing special needs. (One of the observations in the category of special or additional needs was undertaken in a unit within a mainstream school. This unit was designed to accommodate children from across the local authority as a whole.)

Table A4.5. Number of Year Groupings Included in the Observations

School Location

The selected local authority groupings provided a wide geographic spread of schools and representation of different types of location. Just under a third of the schools were based in towns, with a similar proportion based in cities, and a quarter in rural areas. (Table A4.6)

Table A4.6. School Location

Religious Affiliation

Six schools in which observations took place (15%) were Roman Catholic schools, four of which were in the secondary sector and the remaining two were primary schools (Table A4.7). A total of 15 lessons were observed in these six schools.

Table A4.7. School Religious Affiliation

Free School Meals Entitlement

Thirty one of the schools observed (just over three quarters) had a free school meals entitlement ( FME) of 30 per cent or less (Table A4.8). Five of the schools had FME for over half of their pupil roll, with this being represented more in the city areas.

The average FME percentage figure for the schools observed in each of the local authorities indicates a great deal of variation. In the more rural local authorities, the schools observed held FME for around one in 20 (at 4.6 per cent). In comparison, the combined cities authority grouping represented FME for around two in every five pupils.

Table A4.8. Free School Meals Entitlement

*for schools in which observation took place

4.3.3 Interviews with Teachers

A total of 78 teachers were interviewed using the Post-observation Proforma (Table A4.9). However, 18 of these interviews were incomplete and hurried for a number of reasons, most often because of other pressing classroom responsibilities. Interviews were well distributed across the local authority groupings and across the different age-groups observed. In addition, using the short Contextual Information Proforma, 33 of the schools provided background information about the school and community, sometimes through a number of respondents.

Table A4.9. Number of School-based Interviews

*Where there was not enough time to cover all the topics on the proforma.

4.4 Analysis

Data from the observation schedules provided a description of actual practice in drug education lessons and an overview of how pupils responded to these lessons. The range of topics and activities covered formed a major part of the analysis. Pupil engagement levels were also central to the analysis, as were the approaches informing the lesson structure and delivery.

Data from the post-observation interviews also helped to inform the data from the observations, both of which were used to triangulate and enhance the Survey findings.

The data has been analysed using the Statistical Package for the Social Sciences ( SPSS) facilitating analysis of both qualitative and quantitative data.