6.0 Conclusions
The research developed an instrument that was tested in order to establish its validity, to identify ways in which it offers reliable results, and to understand the impact of CLD activity in the pilot populations.
Scrutiny of the data results allowed us to test the instrument's validity and the reliability of findings. In effect, by asking questions of the data, analysing the answers and then asking questions of the answers we were able to then test the validity of the instrument itself.
So what did all this scrutiny and questioning tell us?
A total of 231 people took part in a study, which tested an instrument that had been developed with the support of workers in the field and subject to rigorous statistical analysis. Despite the difficulties of some aspects of the design of the instrument, they largely enjoyed the opportunity it provided them in terms of personal reflection. Their efforts were supported by managers and CLD workers across a range of settings and sectors, all of whom were keen to understand if their work made a difference to the lives of those they support and the communities in which they live.
The data supports their aspiration that it does.
Overall there is a positive distance travelled toward the defined outcomes by 22% of respondents who completed the personal development indicators. This means that CLD is supporting people to become more confident individuals, more effective contributors, responsible citizens and successful learners.
Equally, across a range of indicators, those involved in community capacity building provide evidence of improvement in how their groups function and the impact that this has on communities. Of those who completed the capacity building indicators, 22% indicate a positive distance travelled.
Even given the constraints of causality, this is a positive message.
These outcomes reflect not only those required of CLD in Delivering Change and HGIOCLD?2, they also set out the contribution that CLD can make to many areas set out in the Government's National Performance Framework; which is the bedrock of the groundbreaking November 2007 concordat between national and local government in Scotland. The study confirms that the instrument can assist in providing evidence that CLD is able to make a contribution of value to the education, skills, health and wellbeing of participants. It is tackling issues of inclusion, making a contribution to a fairer Scotland as well as supporting community cohesion, democratic participation and assisting people to enjoy the physical and environmental resources that Scotland has to offer.
The instrument also dovetails well with the Learning and Teaching Scotland and HMIE resources for improving schools and early years provision, Journey to Excellence as well as the capacities in young people being sought from Scotland's education system through Curriculum for Excellence.
The data that is available from this study is limited by a number of factors, most notably the difficulties presented by the lack of linkage between the outcomes and the inputs and processes. Equally the problems presented by the use of a measurement scale that is inadequate to provide a finer level of detail for reporting change merit mention. A more sensitive measurement scale would need to be addressed should there be any future use of the instrument. The Warwick Scale developed in 2006 by the University of Edinburgh for NHS Health Scotland to measure mental health wellbeing may offer some insights in this regard.
The findings of the pilot test indicate that the instrument does have merit for use in measuring the impact of CLD activity on outcomes. This is supported by the analysis of the data generated by completion of the instrument both in terms of what it can tell us about distance travelled and the patterns of responses.
CLD managers and workers are operating in an environment where they are keen to understand the impact of their work. They acknowledge that this tool focuses on outcomes in isolation from the nature and quantity of the resources used to deliver the activity (inputs), and makes no assessment of the approaches or processes used in any given setting. There is no hiding from the fact that this in effect provides the reader with information on outcomes without an understanding of the other elements that contributed to achieving these results for the participant population.
However, this study has been about the development of an instrument which supports national research into the outcomes of CLD activity. It was not about evaluation. Practice tools help that, they work with those involved in CLD activity, assisting them to gain an understanding of how to adapt approaches that will provide the best possible chance of achieving the desired outcomes. Such outcome based practice also enables a direct correlation to be drawn between the outputs (what was done and how) and inputs (the resources deployed).
Evaluation as part of good practice, enables stakeholders to consider, from different perspectives, whether such approaches work; and consequently what could be done differently in the future that would continue to achieve successful outcomes, or have more chance of achieving them in the future.
Such outcomes based practice is therefore essential. It has the best chance of providing a direct correlation between the CLD approaches and associated resources that deliver meaningful change. The practice that is adapted as a result of this information should flow through and be demonstrated in the results of national research, which will also support evidence of CLD outcomes.
To date there is little national research of this nature and the instrument that is the subject of this study seeks to support the establishment of a body of evidence in this field. While there is work to be done to further refine the instrument, the data that it can produce, suggests that were it to sit alongside a longitudinal research and more developed practice tools, CLD may be more able to evidence the impact of such work.
Research data flowing from the use of such an instrument would provide important trend analysis over time. This is largely absent for CLD, which given the body of research evidence that exists in parallel professions such as education, health and social work, gives cause for concern. Research in such fields is frequent, substantive and given considerable credence and CLD should be informed by the same rigour and given increased priority within government.
There are issues with this instrument around the participant's attribution of change to CLD as there are likely to be more factors that have an impact. However if this instrument is used alongside other research which demonstrates the relevance of different approaches on CLD outcomes, then these will complement and triangulate the findings of this instrument.
This instrument is designed to have validity for the general CLD participant population. As a self-completion tool it needs to be engaging and straightforward. Asking participants to comment on links to inputs and processes through the use of this tool would over-complicate and enlarge an already bulky instrument. It would also be in danger of asking questions which participants might find difficult to answer.
However, it is notable that participants in the test found the process of self-reflection a positive experience and workers noted that such processes are an important part of learning, noting a need to ensure that such skills development should be built into work with participants, particularly work with young people. The test of the tool also highlighted the need for support for respondents. CLD by its nature is in contact with and is required to prioritise the needs of those that are most disadvantaged. This includes those whose literacy skills can be poor, whose confidence is low and those for whom self-reflection would be unusual. The support of CLD workers for such participants can be important, and this is reflected in the feedback from workers in the test who highlighted the need to have the option of supporting study respondents. Given the additional needs related to equality and diversity among a range of the population, many of whom are the target groups for CLD engagement, it is essential that the tool, not only in the design of the content, but also its application, takes account of these needs.
The on-line nature of the tool proved to be both a barrier for some in this regard, while for others it worked extremely well. The substantial number of respondents completing on-line (64%) would indicate that further access to ICT among CLD participants may support a greater sample via this route. Given that 66% of the test population were adult learners, among whom some of those with the greatest literacy barriers might be encountered, this may support the use of the tool on-line.
The data gathered from this instrument would provide added value to complementary research into CLD outcomes and should reflect the use of more appropriate practice informed by research, the work of HGIOCLD?2, LEAP (Learning, Evaluation and Planning) and other practice tools, quality and evaluation-based frameworks and tools currently available or under development. There is a need to support the development of such practice tools with further and complementary research. This includes the work that is long overdue and recommended by the PIP project on the substantive and longitudinal study into CLD outcomes.
The data that this study provides is considerable. Time and resource constraints do not permit full and thorough analysis across all the demographic considerations. Those mentioned in this report provide the highlights for the purposes of establishing the validity of the instrument. It is acknowledged however that stakeholders in the pilot sites of Aberdeen and Aberdeenshire may wish further interrogation of the data for their own areas and across a more detailed set of indicators analysis. Given the considerable goodwill shown towards the study by both sites it would be helpful if an accommodation could be reached between the Government and the localities on how best to extract this information.