Scottish Government

Evaluation of the Survey Process

Introduction

49. All respondents to the activity survey were asked to complete an evaluation form on the activity survey process. This feedback will be used by Learning Connections to develop and improve future data collection.

50. Results from the local authority and non-local authority evaluations have been summarised below. Quantitative results from the respondent evaluation forms are supplemented by qualitative information from the free text fields in the evaluation form.

51. The guidance issued to all respondents was split into: evaluation form guidance; data collection process; online return; and overall.

Evaluation results: local authority responses

Local authority response rate

52. All 32 local authorities responded to some or all of the evaluation questions.

Evaluation of the guidance by local authority respondents

53. Guidance for the data collection exercise was issued along with the online survey. Over three quarters (78%) of local authority respondents found the guidance in general either 'very helpful' or 'quite helpful'. Two (6%) found the guidance quite unhelpful. The rest said that they found the guidance 'helpful'.

54. Respondents were then asked how useful they found particular sections of the guidance and there were a range of views on this:

• Summary section - Eighty one percent of local authority respondents were either 'very satisfied' or 'satisfied' with the summary in the guidance and a further 9% were 'quite satisfied'. Only two respondents said they were 'quite unsatisfied' with this section.
• Clarity of the guidance - Eighty one percent of local authority respondents thought the summary was either 'very clear' or 'quite clear' and two respondents felt it was 'quite unclear'. The remainder said it was clear.
• On the 'Background' section in the guidance only two local authority respondents (6%) said they were 'not very satisfied' with the information provided. Seventy eight percent were either 'very satisfied' or 'quite satisfied' with the information provided in the background section. The remaining 9 percent were 'satisfied' with the background section.

55. Specific comments offered by local authority respondents, in relation to the guidance issued included:

• Guidance is still not clear for those for whom CLD is not a core part of their function or background.
• Whilst clear to CLD officers, it was reported that the guidance was too lengthy and may deter some potential contributors from participating. Several Lead Officers in local authorities had produced abridged versions of the guidance for their staff.
• Several positive comments were made regarding the inclusion of examples in the definitions this year and it is clear that this helped respondents to understand the definitions.
• There are still problems in communicating that returns should be made for the number of groups who were assisted during the reporting week rather than the number of people in those groups. It was suggested that this should be made clear on the form as well as in the guidance.

Evaluation of the data collection process by local authority respondents

56. Respondents were asked questions about the time and effort required to complete the survey. The average time taken by local authority respondents to complete survey in 2007 was 33 hours, slightly lower than in 2006, when the average was 36 hours. There was a huge variation in the amount of time taken by various organisations; the minimum being 3 hours and the maximum 200. Eight local authority areas did not respond to this question and some of those commented on the difficulty in recording the amount of time spent on the task.

57. There was a huge variation in the responses given about how many people in the organisation were involved in collecting, collating and entering the data. These ranged from 2 in one local authority to over 200 in another. This question may be open to different interpretation in different authorities, where some include everyone involved in the collection process and others count only those who collate the information.

58. When asked how easily accessible information was, the survey found in relation to local authority respondents that of those who answered the question:

Q1: National priority one: achievement through learning for adults

• For 25% of respondents information was easily and readily accessible;
• For 16% it had to be collected specifically for the exercise;
• For 50% the data existed already and was accessible with some effort.
• The remainder did not answer this question

Q2: National priority two: achievement through learning for young people

• For 19% of respondents information was easily and readily accessible;
• For 28% of respondents it had to be collected specifically for the exercise;
• For 44% data existed already and was accessible with some effort.
• The remainder did not answer this question.

Q3: National priority three: achievement through building community capacity.

• For 13% of respondents information was easily and readily accessible;
• For 34% of respondents it had to be collected specifically for this exercise;
• For 44% data existed already and was accessible with some effort.
• The remainder did not answer this question.

59. More local authorities reported that the data existed already and was accessible with some effort, and less reliance on collecting the data specifically for the exercise, than in 2006. This may indicate that Local Authorities are improving their own management information systems and staff are able to access data collected as a matter of routine to complete the CLD Activity Survey. Indeed, one Local Authority commented in the 2007 survey, that recording data is now a practice that all staff are required to do in the normal course of their work.

60. It is expected that the next national survey of CLD Activity will be carried out, depending on the results of the review, in 2009. Over the years, as staff become increasingly familiar with national data collection, management information systems will be able to adapt to easily obtain the information needed. This has come a long way since the pilot study in 2005, when for example, for building community capacity, 60% of respondents had to collect data specifically for this exercise (now 38%).

61. When asked how typical the levels of activity were in the data collection week, 72% of local authorities responded that they were either 'very' or 'quite' typical. However, a substantial number (19%) reported that it was either quite or very untypical. Many of these respondents did not give specific reasons for the week being untypical, however, those that did, mentioned either bad weather leading to many cancellations or that the reporting week coincided with special weeks in their programme which either increased or decreased the numbers of participants in comparison to a "typical" week. It was open to organisations to move the reporting week under special circumstances and this could have been applied, particularly where the weather played a significant part in the week being untypical.

62. Specific comments around the data collection process included:

• Difficulties in calculating the number of hours spent on the survey.
• Understanding of "what is CLD" is patchy.

Evaluation of the online return process by local authority respondents

63. 69% of local authority respondents said they were either 'very satisfied' or 'quite satisfied' with the online form as a way of reporting the information. 17% said they were 'satisfied' and four respondents (13.8%) said they were 'not very satisfied'. No respondents said that they were not at all satisfied with the online form.

64. The most common specific issue raised was that the electronic form had to be completed in one session and could not be saved or returned to later. Unfortunately, this was a restriction due to the survey software used for the on-line return. A copy of the survey form in Word format was sent out with the documentation for 2007. However, some respondents mistook this for the actual survey form and tried to return it on-line. The problems arising from this may have been the cause of increased dissatisfaction with on-line reporting, though it should be remembered that the vast majority of respondents in local authorities were satisfied with this type of submission.

65. The computer software used for the 2006 and 2007 Activity Survey has now been replaced by a system which will allow people to go in and out of the survey before submitting it. It may not, therefore be necessary to send out the word version of the survey for future collections.

Relevance of data collected to local authority respondents

66. Encouragingly, most of the respondents (83%) said they saw the relevance of the data being collected to their own work. Two said they did not think it was relevant and three were not sure. A large majority (26 respondents) said they would be willing to participate in the activity survey again; four said they did not know and two did not answer the question. No one said they would not participate next year.

67. When asked what other information should be collected in future activity surveys, local authority respondents gave a number of suggestions, including:

• More targeted questions on learners' experience,
• The numbers of people in the groups provided with community capacity building support.
• Analysis of male/ female split
• As with last year, there were requests for more detailed information regarding the depth of the learning experience eg staff time or a breakdown of the types of opportunities available.
• Educational guidance sessions should be recorded separately.
• Recording partnership work which takes place at the venue.
• It may be useful to stipulate whether activity is taking place in priority or non priority data zones.
• Data related to outcomes, as well as participation.

Local information systems

68. It is clear that many local authorities have shown commitment to developing and improving their information systems and as a result of this, collection of the basic information we ask for on the CLD Activity Survey is a much easier for many local authority staff.

Evaluation results: responses from non-local authority providers

Response Rate by non-local authority respondents

69. Of the 95 returns submitted by non-local authority partners, 70 (74%) included responses to the evaluation questions. The difference between the number of returns and the number of responses to the evaluation is largely explained by the fact that some organisations submitted more than one response because they operated across several local authority areas but only completed one evaluation. There were a small number of organisations which did not respond to the evaluation at all and organisations did not always respond to all questions.

70. The number of responses varies from question to question in the evaluation and it should be noted that percentages in the following section, for non-local authority organisations, will be calculated using the number of actual responses to each question. Therefore any percentages should be read as 'the percentage of those who responded to this question'.

Evaluation of the guidance by non-local authority respondents

71. Respondents were then asked how useful they found particular sections of the guidance and there were a range of views on this. Scoring was based on a 5 point graded scale where 1 was very helpful/ clear / satisfied and 5 was very unhelpful/ unclear/ dissatisfied. Data on responses by section are set out below.

• General guidance on data collection - Based on 69 responses, almost two thirds (61%) of respondents found the guidance issued for the data collection was either 'very helpful' or 'helpful'. Three respondents found the guidance 'unhelpful'.
• Clarity of guidance - Based on 68 responses, 57% of respondents found the guidance either 'clear' or 'very clear'. Over a third (34%), however, found the guidance neither clear nor unclear and six respondents found it 'quite unclear'.
• Summary - 62% of respondents were satisfied or very satisfied with the summary section of the guidance. A further 36% were neither satisfied nor dissatisfied with this section and only one respondent was quite dissatisfied.
• Background - Just over two thirds of respondents (67%) were either very satisfied or satisfied with the background information given in the guidance. A further 30% were neither satisfied nor dissatisfied and only 2 respondents (3%) said they were quite dissatisfied.
• Questions - 60% percent of respondents were either satisfied or very satisfied with the questions asked in the survey, the remainder were neither satisfied nor dissatisfied. They are the same questions which have been asked every year of the survey since it began.
• Definitions - 64% of respondents were either satisfied or very satisfied with the definitions. A further 34% were neither satisfied nor dissatisfied and only one respondent was not at all satisfied. Learning Connections responded to suggestions in 2006 feedback and included more practical examples in the guidance. This received positive feedback in the 2007 evaluation, saying that it was helpful in identifying appropriate activities to be included in the survey.

Evaluation of the data collection process by non-local authority respondents

72. The average total time taken to respond to the survey by non-local authority respondents was found to be 2 hours (this was significantly less than the time invested by local authorities). Factors affecting this may be differences in scale of the organisations concerned, and the fact that non-local authority partner organisations reported that information was considerably more accessible to them.

73. Non-local authority organisations only answered the questions on the survey that were relevant to their activity. The percentages below only relate to the organisations which answered the specific question.

74. When asked how easily accessible information was, the survey found that in respect of adult learning opportunities:

• For 52% of respondents, information was easily and readily accessible;
• For 37% of respondents, the information existed already and was available with some effort.
• For 11% it had to be collected specifically for the exercise.
• (Based on 65 responses)

75. When asked about getting the information on youth work opportunities:

• For 60% of respondents information was easily and readily accessible;
• For 34% of respondents, the information existed already and was available with some effort.
• For 6% of respondents it had to be collected specifically for the exercise.
• (Based on 35 responses)

76. And finally, in relation to community capacity building:

• For 60% of respondents information was easily and readily accessible;
• For 30% of respondents, the information existed already and was available with some effort.
• For 9% of respondents it had to be collected specifically.
• (Based on 43 responses)

77. As with the local authority organisations, these figures appear to indicate an improvement in the availability of existing data which can be accessed for the purposes of this survey. However, that is only true of the organisations who actually participated. The low response rate is a concern; it means that there is still little reliable information on the overall scale of provision by these sectors.

78. Care should be taken when comparing year on year for the non-local authority sector as the majority of organisations who responded are not the same for both years.

79. More than three-quarters (77%) of respondents felt the week selected (5 to 11 November 2007) was either 'typical' or 'very typical' in terms of levels of activity (compared to 21% who reported it was 'quite' or 'very untypical' (the remainder did not know). Reasons given for the week being untypical included other events such as in-service week and youth work week. Some areas were affected by bad weather and others simply reported the week was particularly quiet eg because of availability of volunteers or other workers.

Evaluation of the online return by non-local authority respondents

80. Almost half (49%) of respondents were 'very satisfied' with the online return form as a way of reporting the information, and 28% said they were 'satisfied'. Four were dissatisfied and another 5 were not at all satisfied; that is 8% percent who gave negative feedback on this aspect of the survey.

81. The vast majority of respondents were satisfied with completing the survey on-line and there were several positive comments from non-local authority respondents regarding the efficiency of responding on-line. Less positive comments generally related to difficulties in being able to access the online form, either because it was unclear how to do so or because of technical difficulties.

82. There were some suggestions for improving the existing on-line survey, relating to the software:

• The survey had to be completed in one sitting and this proved problematic. (New software is now available to be used in future surveys, which will enable people to go in and out of the returns before final submission).
• There were several comments suggesting that creating an option to print out responses for their records should be explored. Learning Connections are currently looking into how this can be done.

Comparison between local authority and non-local authority responses to evaluation questions

83. Overall, both sets of respondents were satisfied with the guidance and data collection process. However it is clear from the range of evaluation responses that there were two significant differences. These were:

• Satisfaction with definitions: non-local authority partners were significantly less likely to express satisfaction with the definitions used (though over two-thirds were still 'satisfied' or 'quite satisfied'). This perhaps reflects less familiarity with CLD terminology compared to those working in local authorities.
• Accessibility of data: non-local authorities reported more frequently that they already held the data needed, and that it was readily accessible to them. This may be in part due to the differences in scale, given that most of the non-local authority respondents were from voluntary organisations. Unsurprisingly, an analogous pattern was reported in the time and number of staff involved in data collection.

Summary of evaluation

84. In summary, the evaluation of the data collection exercise for 2007 was broadly positive. The majority of respondents expressed satisfaction with the guidance and the data collection process.

85. However, the low levels of returns by non-local authority partners makes it difficult to draw conclusions about how the process worked for them. It may be that those who were not satisfied with the guidance or the process simply did not make returns.

86. Encouragingly, however, of those that did take part the vast majority from both local authorities and non-local authority partners said that they could see the relevance of the exercise and would be willing to take part in it again.

87. Against this broadly positive picture, there remain a number of issues that should be considered in future collection of data on CLD activity:

• Indications that non-local authority partners were less satisfied with the definitions provided.
• Comments from some respondents about how the online return could be adapted to be more useful.
• Comments about the type of information respondents would like to see collected in future (but there were also suggestions that asking for too much information could deter respondents from participating in the survey).