CHAPTER TWO: METHODOLOGY
2.1. Throughout the evaluation period, contact has been maintained with the PTS and YES, who have promptly responded to our many questions and provided other relevant information and reports. There are two main aspects to the evaluation: firstly the qualitative, which aims to build up a picture about how xlerate with xl works, and secondly, two quantitative surveys, which look at the students views of the programme, as well as providing measures of their enterprising attitudes and behaviours.
QUALITATIVE - SITE VISITS AND INTERVIEWS
2.2. This aimed to look at a wide range of views from those involved with xlerate with xl. Most of this was carried out in face-to-face meetings, but also some by telephone. For a number of the meetings, we also used short questionnaires to help create a focus and, where appropriate, we have followed up individual responses for further clarification.
1. Meeting with xlerate with xl the manager and coordinators (January 2005). This was to establish and agree the detailed parameters for the evaluation, and to provide us with outline details of the programme and key issues.
2. Site visits to the ten pilot clubs (April and May 2005) including, meetings with the advisers, as well as group and individual discussions with the students. Where necessary or appropriate follow-up telephone conversations with the advisers were carried out.
3. Xlerate (enterprise) training days (April and June 2005). This was carried out by YES and about 20 advisers and coordinators attended. They were held in Glasgow and Edinburgh, and again we did some follow-up telephone calls and email contacts with advisers.
4. Focus group meeting with the coordinators (June 2005) to discuss organisational aspects of the programme. Ongoing contact was maintained to enable clarification of any issues arising.
5. Two focus group meetings for advisers from all of the pilot clubs. These were held in two locations, Ayr and Perth (Sept 2005) with 11 attending. Besides the group meeting each adviser completed a short individual survey which was used to help set the scene and structure the meeting. Several advisers were contacted afterwards to clarify specific issues.
6. Telephone interviews with headteachers (seven) which were preceded by a short written questionnaire. Although these were not scripted they did focus on issues and concerns, in particular any perceived weaknesses of the programmes, or specific difficulties they had encountered.
7. Attendance at a number of events, such as the adviser conference, on an informal basis.
8. Overall, ongoing contact with PTS and YES was maintained throughout, including more formal meetings to review the findings and interim reports.
QUANTITATIVE - SURVEYS
2.3. Two surveys were undertaken during the course of the evaluation - one which looked at the enterprising qualities of the students, and the other which looked at their views and perceptions of the programme. These were both conducted on two occasions using the same questions, with the difference showing how the students' views have changed over the period as the students moved from S3 to S4:
1. Student Survey Round 1 (S3) February / March 2005
2. Enterprise Survey Round 1 (S3) February / March 2005
3. Student Survey Round 2 (S4) November / December 2005
4. Enterprise Survey Round 2 (S4) November / December 2005
2.4. The Student surveys sought the views of the students taking part in the programme and addressed five main areas:
1. How xlerate with xl had helped, or could help support them whilst at school.
2. What they thought of the sessions generally (and their advisers).
3. Which aspects of the programme were of particular benefit.
4. The bigger picture beyond xlerate with xl.
5. Plans for the future.
2.5. A total of 43 questions were asked, with the 5-point Likert scale being used for most of the responses - Definitely Yes, Mainly Yes, Neither, Mainly No, Definitely No (and also a non-response option). In terms of delivery, most clubs did an on-line version; however, due to difficulties accessing computers, a few clubs did a paper-based version. Because of possible variations in responses to the format of the questionnaire the same method was used for the second round - i.e. those that did it on-line the first time did it on-line the second time.
2.6. The enterprise surveys assessed the students Enterprising Attitudes and Behaviours - i.e. how enterprising they were at the time of the first survey when in S3, and again when in S4 after having taken part in the programme for about two terms. In addition, control groups of similar students in each of the host schools were surveyed, which it was hoped would give an indication of the relative progress made by the xlerate with xl group with respect to their peers. 9
2.7. In terms of the design, just as there is no simple standard definition of enterprise, so there is no standard off-the-shelf survey instrument. Therefore, it was necessary to design a suitable survey. The DtS definition 10 was used as the starting point. This gave a number of more discrete factors or descriptions of enterprising attitudes and behaviours, which we used as the basis to select, where possible, suitable existing behavioural and attitudinal scales. Where necessary, we augmented these with items from our own CEM indicator systems 11 and, in some instances, new questions were produced 12.
2.8. This process gave us a bank of around 120 potential survey questions which, it was felt, would encapsulate enterprise. These were then trialled with a large secondary school (S1, S3 and S5) 13, and from an analysis of the results, we produced a final survey with 50 questions. As with the Student survey a 5-point Likert scale was used, and most clubs were able to complete it on-line.
2.9. From the two surveys ( WSA and xlerate with xl) we received in excess of 1100 complete and useable responses from the first round. Further analysis allowed us to further refine the survey and reduce the number of questions that we would use to 37, which contributed to eight factors or dimensions of enterprise which we have identified as:
1. Self Confidence
2. Motivation
3. Curiosity
4. Organisational Confidence
5. Determination
6. Responsibility
7. Persistence
8. Vocational Confidence.
2.10. However, in doing this, we had to drop one of the original scales - Risk Taking, which, for reasons that are not entirely clear, produced inconsistent responses. It may be that the notion of Risk Taking per se is not necessarily consistent with enterprise, or simply that we asked the wrong questions; either way more research and trialling of the questions would be needed 14.
2.11. In terms of usefulness this final enterprise scale of 37 questions which is used in this report was found to have a high level of internal reliability (Cronbach Alpha 0.89) 15. This means that responses to the individual questions are quite consistent, and therefore the entire scale measures essentially the same qualities - i.e. enterprise.
2.12. In terms of the initial distribution, all those clubs with S3 groups were asked to take part in both surveys; however we were concerned about the possible response rates. The PTS has had difficulty in the past getting reasonable response rates, which is not uncommon with other educational surveys. Indeed, 20-30% appeared to be typical rates for other surveys connected with enterprise in education and this does very much limit the value of the results 16. Therefore, we agreed at an early stage that we would make sure we got a 100% response rate from the 10 pilot clubs, but would still invite the others to take part. It should also be noted that for the first round of surveys, all the Pilot clubs would be doing the new xlerate module, whereas the other clubs were mostly doing the old xl enterprise module.