Scottish Government

4.0 Methodology

The objectives meant that it was possible to set out three distinct phases to the work:

Phase One - the construction of an instrument to measure outcomes and identify the population against which to test this on a retrospective basis.

Phase Two - testing of the instrument across all three CLD priority areas.

Phase Three - analysis of the findings and methods to establish the contribution that the instrument can make to measuring outcomes and recommendations for future action.

4.1 Phase One - Construction of the instrument

The instrument ³ was developed over the spring and summer of 2007 and entailed a literature review (which included taking cognisance of a smaller-scale study underway with the University of Edinburgh into the social capital outcomes of CLD, subsequent creation of a range of outcome indicator statements that were then tested with CLD stakeholders across the country in a series of workshops; and latterly the formation of the instrument).

The four CLD⁴ themed workshops were well attended by a total of 81 stakeholders from community, voluntary and statutory sectors engaged in CLD activity. Participants in the workshops considered a range of CLD outcome indicators drafted by the consultants and informed by the literature review. These workshops were held so that practitioners, managers and academics engaged in CLD work could contribute to the construction of the tool by refining and shaping the indicators.

Their contribution resulted in a culling of the indicators from more than 100 to the 76 that formed the core of the instrument, which went on to be tested in the pilot. This culling was based on information drawn from both the focus groups and from subsequent statistical analysis.

The latter considered the frequency of workshop participant agreement with the desired outcome statements, as well as the correlation between responses and patterns of responses. This sought to establish whether the indicator demonstrated the desired outcome, if it was too general and could be applied to other outcome categories or if it was too vague and applied to none of the outcome categories.

Using statistical analysis tools such as weighting, analysis of correlations to other indicators, cluster trees, discriminant function analysis and simple correspondence analysis, it was possible to identify the strengths and weaknesses of each indicator statement against the outcome it was intended to evidence. This, together with stakeholder feedback on wording, enabled amendment and culling of the indicator statements and a stronger confidence in their validity.

Following this process the instrument was further refined with the assistance of the research advisory group to include appropriate end outcome indicators, a rating scale and final design considerations. It was agreed that the tool should be electronic and able to be completed on-line with alternative accessible formats available.

The advisory group also assisted in identifying the test populations, which it was agreed should include both rural and urban dimensions. Offers of interest in taking part in the pilot were considered and agreement reached on Aberdeen City and Aberdeenshire.

4.2 Phase Two - Testing of the instrument across all three CLD priority areas

This pilot phase of the research was a test of the instrument in order to test validity and to identify ways in which it offers reliable results; as well as to understand the impact of CLD activity in the pilot population.

The fieldwork pilot took place during the autumn of 2007 and the instrument was available for completion for six weeks during this period. A joint briefing meeting was held with 12 agency managers involved in the CLD Partnerships for both pilot populations, and was followed by similar joint briefing meetings for workers spanning all three Community Learning and Development strands. This resulted in 43 practitioners joining a group that supported the test on the ground.

Feedback from workers at this early stage enabled additional fine tuning of the support information for participants. It also established that hard copies would be required in larger quantities than anticipated. A total of 231 individuals took part in the pilot completing the study both on-line and in hard copy, of these, 83 preferred to complete using paper copies.

Once the test was underway workers received regular updates on the number of respondents by CLD strand.

4.3 Phase Three - Analysis of the responses and instrument performance following a fieldwork pilot

Following completion of the fieldwork pilot test of the instrument all data was collated and analysed by statistical and research consultants. This identified the key findings related to the outcome indicators (intermediate and end) as well as a review of the experiences of participants and workers in relation to use of the tool. Findings from the data also enabled issues of validity and reliability to be identified and addressed in the analysis of the data. Feedback from 11 workers and 6 participants was also taken into account in relation to the experience of using the instrument.

4.4 Constraints

A study of this nature has a number of constraints that are worthy of note:

• The data gathered is subject to the individual respondent's views on any given day. This ' good day - bad day' effect means that someone may answer one way on one given day and another on another given day depending on the circumstances of their mood on that date.
• The instrument also seeks to gather the views of participants on the distance travelled in relation to the Delivering Change outcomes and to attribute this to their engagement in CLD activity. Participants were asked to report where they were when they started the CLD activity and where they are now. For some this period might have been relatively short - weeks or months - while for others it might have been years. The retrospective nature of the instrument contributes two possible constraints:
  • The instrument does not assess the difference that the length of the retrospective period might have on the impact of the activity. In other words, it does not consider if a participant is involved in CLD activity for 10 days or 10 months. It relies on the participant to reflect back on when they started the activity without regard for when that might be. While potentially a constraint for the results of this study, this factor has more to do with an assessment of the impact of inputs and processes on CLD outcomes which this study does not address.
  • Asking participants to reflect back to when they started the activity relies on their past perceptions and memories as data. While initially considered a constraint for this study, there is no evidence that participants had concerns about or had difficulties understanding the task of retrospection. However, asking participants to provide a retrospective report remains a possible constraint to be mentioned here.
• It is also important to acknowledge the constraints of respondent perceptions which attribute change solely or substantially to the CLD activity. It is possible that the change is related to the impact of CLD; however it is also possible that such change is the product of factors that this instrument is unable to tease out. This is particularly true for the indicators for end outcomes. This does not invalidate the respondent's view but does not allow for additional factors to be considered.
• The instrument does not provide any assessment of the nature and quantity of the resources used to deliver the activity (inputs), and makes no assessment of the approaches or processes used in any given setting.
• There may be a time delay between the CLD activity and the outcomes, creating a distance between the change and the factors that might be attributed to causing the change.
• Worker influence was something that was discouraged as part of the test arrangements. However, workers reported that literacy issues would have prevented some respondents from participating in the study if support had not been available. A significant number of respondents reported completing the instrument on their own, with 179 of the 231 (77%) working on their own and only 52 of the 231 (23%) reporting that they were assisted in completion of the instrument. It is not possible to tell the extent to which the assistance of workers influenced the responses of these individuals.
• In a similar vein workers reported that despite reassurances to the contrary, some respondents appeared keen to ensure a positive response to statements. The workers mainly attributed this to the participants' association of the instrument with future funding arrangements.
• This response is indicative of some participants and workers who reported identifying the study as in keeping with evaluation, rather than research. It did not appear to be fully understood that outcome measurement is not performance measurement or programme evaluation.
• The correlations drawn from the data returned by workshop participants are based on 81 people working in a series of smaller groups. It is possible that a higher number of participants and groups would affect the patterns as they may alter how they categorise the indicators.
• The starting time for the pilot test of the instrument was delayed due to a recess taken by many CLD groups during the summer months. This had a knock-on effect to the subsequent testing timeline. This affected the length of time the instrument was available for testing - 6 weeks. Workers noted that a longer period may have secured more participation.
• It proved difficult to obtain the views of CLD participants on the experience of using the instrument. The views of respondents account for only six adult learners. A focus group planned with young people was not possible within the reporting timeline, and no respondents engaged in community capacity building were available for interview. However, considerable feedback was possible from the 11 workers who took part in a focus group at the end of the completion process. Their comments included views on the experience and the instrument from their own perspectives and role in the process, as well as feeding back views from many participants.
• The measurement scale used in the instrument was adjusted following feedback from a range of stakeholders in the process leading up to the final design. This reduced the range of options to three 'hardly ever', 'some of the time' and 'most of the time'. While this made an already bulky instrument appear less demanding, it had the effect of reducing the range of answers available and the feedback from participants indicated a desire for more options. The lack of options for reporting smaller amounts of change may have had an influence on the amount of change reported or the number of people who reported a change.