CHAPTER 2.4. HOW DOES THIS WORK FOR CULTURAL AND SPORTS POLICY?
Cultural Indicators
4.1 So, if we can find nothing of significance in other policy areas which might be useful to develop social and economic indicators with which to measure the impact of culture and sport on the quality of life and sense of well-being of both individuals and communities, is there anything happening specifically in policies for culture and sport?
4.2 Over the last few years, there has been a growth of 'cultural indicators' and a debate around how the contribution of culture in the community can be measured.
4.3 We use the term 'cultural indicators', as opposed to 'sports indicators'. This is because the debate is happening now within the field of cultural, as opposed to sports, policy. There is some discussion around the development of ways of measuring the impact of sport: for example, Coalter's work for sportscotland on the social benefit of sport which specifically links the role of sport across health, education, community development, crime prevention and economic development, and argues for research to measure its effects. 426 In 2004 Sport England launched its framework for sport which includes developing evidence for the contribution of sport to society and is now developing indicators with the Audit Commission. 427 But, perhaps because the benefits of sport are more easily articulated, particularly in relation to benefits to physical health, it is the case that the policy debates are not as extensive as in cultural policy. Nevertheless, it should also be recognised that the debate around cultural indicators is closely allied to the debates on sport partly because sometimes sport is included in a broadly defined sense of 'culture' and also because the arguments are transferable, one to the other.
4.4 As indicated in the literature review, the area of cultural indicators is still in the early stages of development and commentators have reflected on the lack of theoretical underpinning and the confusion between cultural indicators as a sub set of community indicators or as a set on their own. 428 Others criticise the confusion between 'research and argumentation' and highlight the lack of academic research in this area as being a handicap to development of work on 'measurement of outcomes and effects'. 429 Others go further: while not alone in this, Matarasso raises questions about what we measure and why in cultural policy. He writes:
Like the dog that famously did not bark […] most of the current work on indicators is notable for what is doesn't say. The missing element is where the indicators have come from - in other words what it is they are supposed to measure. 430
4.5 As we have demonstrated, the academic literature provides us with little empirical evidence of a causal link between culture and sport and quality of life/well-being. As a result, the work on indicators is based on either assumptions or suggestive evidence about the benefit of culture/sport for quality of life/well-being and is not based on a theory which is rooted in a strong evidence base. The issues for the development of indicators, as the commentators above neatly summaries are, then:
what are they for, or, what is the policy initiative which drives them?
can it be measured?
what measures would be useful?
4.6 It is all too easy for policy makers to avoid interrogating these questions and instead focus on the deceptively simple question of 'what can we measure?', with indicators, again, being driven by data availability.
4.7 Yet there is a growing debate around the view that 'we know it works'. Even if it is not always easily measured, there is sometimes no need to 'prove' the link between culture/sport and, in a broad sense, quality of life. It is argued, in this subset of the literature, that the issue (for policy makers) is to make sure it works better and (for researchers) to work out how and why. Starting from this given, we find that new terms are creeping into the policy literature are 'cultural vitality', 'cultural well-being' and 'cultural planning'.
Cultural Indicators: some international examples
4.8 The problems with statistical indicators has been explored in a recent review undertaken by the International Federation of Arts Councils and Cultural Agencies ( IFACCA) which demonstrates that it is a problem across the world. 431 This report points out that there is often a conceptual problem: a confusion about what indicators actually are and how they are used, typically seen in the confusion between a statistic and an indicator. IFCAA asserts that there is a lack of quality data and the lack of proper use of existing data - but here also highlight how frameworks are unwieldy particularly when working at macro level. There is a multiplicity of approaches, some replicating the work of others across the world. But the most fundamental problem exists where policy objectives are vague.
Cultural policy objectives tend to be couched in broad, abstract of even vague terms […] But such abstraction hinders the development of clear indicators for policy evaluation. 432
4.9 In the literature review we explore at some length the Urban Institute's Arts and Culture Indicators in Community Building Project ( ACIP) established in 1996 in the USA in collaboration with the Urban Institute's National Neighbourhood Indicators Partnership. ACIP has set out to investigate how arts and culture-related measures can be integrated into neighbourhood indicator systems, whose purpose is to monitor quality of life at community level. But before it could do that ACIP has tackled the lack of theory relating cultural participation to quality of life. The first two years of the project involved qualitative research to investigate what communities themselves recognise as culture. This produced a very broad definition of cultural participation, creating a further challenge for theory development:
the very broadness of ACIP's definition - combined with the fact that arts, culture and creativity are operating in an environment in which many other factors are operating simultaneously - vastly complicates the task of pinpointing the contribution of arts-related activities to the overall impacts observed. 433
4.10 This approach, to build from the communities' own take on culture, is one we will come back to in the context of cultural planning.
4.11 Also in the USA, the Knight Foundation has supported the development of core indicators to be used in cities across the country. 434 The indicators in this study are essentially quantitative but do not set out to measure the impact of the arts on quality of life/well-being but to measure the vitality of the arts. For this study and view, it is a 'given' that a measure of a healthy and sustainable community is the vibrancy of its cultural sector. The focus is, thereafter, on identifying attendances/participation, size of sector, the funding sources, number of key staff range of collaborations and number with schools programmes.
4.12 A similar focus comes from the New Zealand government's Ministry of Culture and Heritage/Te Manatu Taonga, which makes the case that
'[c]ultural well-being' could be expected to: 'encompass the shared beliefs, values, customs, behaviours, and identities reflected through language stories experiences, visual and performing arts ceremonies and heritage'. 435
4.13 An example of the application of this approach, also discussed in our review, can be found in a Knight Foundation supported study in Silicon Valley where the focus was on developing ways to increase a sense of community in an area which does not lack 'creativity' - used in its broadest sense - but which was seen to lack the social connectedness which is, so often, valued as a marker of quality of life. 436
4.14 The notion that social connectedness, or social capital, is part of quality of life is touched on in other policy discussions but here it is quite explicit. One of the key assumptions of the Creative Community Framework is that cultural participation enhances creativity. The framework spells out a series of links and progressions from cultural levers (arts education, policy, funding and leadership) through cultural assets (creative community, venues and facilities and civic aesthetics) leading to cultural participation and then on to cultural outcomes expressed as 'creativity connectedness and contribution'. From this essentially conceptual framework comes the index, and a set of indicators, which measures each stage. These are expressed quantitatively and are based on survey work which looks at both the levels of participation and attendance in a range of cultural activities as well as the size of the cultural sector - expressed in terms of facilities, new work created and in terms of investment.
4.15 It is interesting to note how the index measures areas which might be regarded as less tangible and less easily measured by statistics alone. Creativity, for example, is an area which might be seen as more difficult both to define and measure. However, one quantitative measure used has been the number of patents applied for and issued. In Silicon Valley - an area where the Knight-supported research suggested a lack of social connectedness, this measure demonstrates significant creativity - the number of patents applied for and issued and the number of times Silicon Valley firms are cited in other patent applications. 437 Creativity, in this case, is very much wedded to economic growth.
4.16 'Connectedness' is another part of the framework which is less tangible. Here the Knight Foundation research is able to draw on a US-wide survey on social capital which measures levels of social connectedness via surveys and then analyses across different groups. 438 There is an assumption that the arts are good for developing social capital and building communities although no evidence is offered as to why this is the case. We will return to this survey below as it raises some important questions about the link between quality of life and social capital.
UK Cultural Indicators
4.17 Turning to approaches in the UK, we have seen that some attempts have been made to develop indicators drawing on existing data sources, similar to the Audit Commission's approach to indicators for local authorities. But there are severe limitations. The Museums, Libraries and Archives Council ( MLA) has attempted to develop the existing statutory measures to demonstrate the importance of public libraries through what they term 'impact measures'. 439 One of the priority areas in this framework is to 'improve the quality of life for children, young people, families at risk and older people' and another is 'promoting healthier communities'. Both are based on an assumption that accessing library services improves quality of life and has an impact on health. Specifically, it is asserted, borrowing self-help health books contributes to better health. But the evidence is not at all robust. The underpinning research which supports this assertion is based on a Department of Health Briefing, published in 2003, which examines the use of self-help materials (books and on-line) for mental health issues. 440 However, while this research on mental health does point to 'a significant benefit from self- help materials', the benefit is to be found in specific interventions for a range of specific conditions, that is, cognitive behaviour therapy used for depression, anxiety, bulimia nervosa and binge eating disorder. 441 There is no evidence for claiming a benefit for those with, for example, bipolar disorders or indeed with physical disorders. There is some evidence that the use of self help materials is most effective when integrated with other approaches, and the research concludes that there 'evidence is lacking to support unqualified recommendation of self- help materials'. Moreover, the paper argues that there is a theoretical possibility, which has not been studied, that materials might cause harm either because of inappropriate use or by deterring users from seeking professional help. 442 The only conclusion to draw is that the MLA indicators are based on a partial and misleading reading of research material. Perhaps what this reminds us is just how much is at stake in the world of 'evidence-based policy making' particularly for cultural quangos.
4.18 Moving from the national to the local, in the UK there are examples of local authority, or regional based, initiatives focused on developing cultural indicators which do attempt an admirable level of rigour.
4.19 Work for Shropshire County Council by Morris Hargreaves McIntyre attempts to map the impact of a range of cultural activity - including sport recreation arts and museums on the local community. 443 A very specific aim of their study was to 'demonstrate and measure the impact of cultural provision on social and wider "quality of life" agendas'. To achieve this Morris Hargreaves McIntyre developed a set of indicators in a 'measurement framework' with which they are aiming to develop a baseline of information and then regularly measure change. 444
4.20 Morris Hargreaves McIntyre recommends that a significant number of indicators are to be drawn from existing local and national government sources (for example, crime statistics or local authority monitoring data). However, although attempting to draw together existing data, the framework itself runs to several pages and identifies a huge number of indicators. This, in itself, may not be a problem but there is still a reliance on data which are more easily captured - the number of books borrowed from the library, for example. Morris Hargreaves McIntyre also proposes questionnaires for users/attenders, participants, group leaders and organisations which are intended to measure levels of satisfaction. But one might question how far satisfaction with a service is a measurement of quality of life.
4.21 What we find with both the approach in Shropshire - and the impetus behind the MLA work - is an attempt to 'prove' what the literature review tells us is not possible, and that is that cultural and sport have a causal relationship with a wide range of aspects (domains even) of quality of life. The problem it seems starts with the definition, or policy focus, and then becomes even more problematic when moving to the indicator or measurement.
Cultural planning
4.22 More in line with the ACIP project in the USA is the work around indicators which has grown out of the concept of 'cultural planning' discussed in UK and Australian policy literature. 445 Cultural planning in essence takes as its starting point that culture is at the heart of economic and social and sustainable development and is not separate from it. Cultural planning advocates a 'bottom up' approach, that is one which starts by mapping the culture of an area covering a wider range of activities than the arts - for example heritage, cultural industries and built environment and, most importantly local customs and ethnic and cultural diversity (although interestingly not sport). One advocate of the cultural planning model, Colin Mercer, argues that in terms of measurement it is necessary to move 'up "the knowledge value chain" from data (statistics) to information (indicators) to knowledge (benchmarks) to wisdom (policy)'. 446 He makes the argument for developing indicators - qualitative and quantitative - which map culture in an area as a starting point for cultural planning.
4.23 This principle is applied by Mercer in work done in Essex for the county council. 447 The most recent report for 2001/2002 builds on work started in 1999 and, it should be noted, is mainly about the arts and does not include sport. Unlike the approach in Shropshire all data gathering is generic to this project and the county has benefited from an on line data gathering system developed by Comedia. As well as capturing the input/output data gathered from organisations, there are questionnaires which include multiple choice and open ended questions targeted at participants and audiences (both distributed at events).
4.24 While this is certainly a more manageable approach than the Shropshire one developed by Morris Hargreaves McIntyre there are still questions to be asked about the interpretation or conclusions to be drawn from the data. For example, the key qualitative indicator is levels of satisfaction which, one might reasonably argue, is not the same as quality of life. The authors of the report recognise that there is still a need for refinement but their critique of their qualitative indicators is that the self-report to open-ended questions may present a problem. 448 It could also be argued that there is not enough self-refection or sufficient effort put in to any attempt to triangulate some of the quantitative data with some qualitative texture. For example, there are some very big changes in the number of people employed in the sector over the two reported periods. 449 At the same time there is also a growth in volunteering but a decrease in the number of hours each person on average volunteered. Are these statistical problems created by the reporting system which will 'iron out' over time? Or is there a shift from paid to voluntary work? Or is there something else going on? A more developed qualitative approach might help to answer these questions.
4.25 However, and more profoundly, it is difficult to see how this methodological approach addresses Michalos' definition of cultural indicators as 'measures of people's beliefs and feelings about the arts' which he regards as part of subjective social indicators and which he contrasts with the kinds of data gathering and analysis found here in the cultural planning approach which he regards as 'objective social indicators'. 450 Nevertheless, the cultural planning model may be seen as a serious attempt to develop a conceptual framework for measuring the role and impact of culture within an area.
4.26 As mentioned in relation to the wider policy literature on quality of life and community development, missing from anywhere in the debate on cultural indicators is the question: what about the quality of the intervention? There is an almost complete absence of debate about the quality of the arts and sport provision - either in terms of facilities or in terms of activities or events, beyond counting numbers of new art works or sports facilities. But, does this matter? We found in some of the academic literature relating to sport debates about the amount of exercise (dosage) required to address issues of psychological well-being (stress, depression, etc). There is also some limited literature about the role of the coach. However, in the policy literature-and certainly in the world of cultural indicators - there is a lack of clarity on the question of quality of cultural or sporting intervention. The quality of the participants' experience will affect outcomes but no studies of impact have addressed this in their research design. There is an assumption that all cultural or sports participation is the same.
4.27 Finally, there is the linked but distinct area of social capital. There is an implication in some of the academic literature that the impact of sport/exercise on an individual's sense of well-being might be in part to do with meeting others in the same condition. The role of sport in building social capital is also referred to in some studies. When we move to the policy literature the idea of social connectedness becomes even more explicit. We were not asked to look at the social capital literature per se but it is one of the 'domains' of quality of life identified in the academic literature and the influential report in the US (which underpins the Silicon Valley work). The Social Capital Community Benchmark Survey makes explicit the link between social connectedness and quality of life. Based on survey work carried out across three years in 40 communities in the US the research concludes that: 'social connectedness is a much stronger predictor of the perceived quality of life in a community than the community's income or educational level.' 451 This research project's finding was that your personal happiness is not directly affected by the affluence of your community but it is quite directly affected by the social connectedness of your community.
4.28 Although this report does point to the role of the arts in strategies for developing greater social connectedness, the two biggest challenges which the research uncovered is the place of 'faith based civic engagement' and 'diversity' - specifically racial diversity as represented by whites, blacks and Hispanics. Admittedly both of these challenges - and the initial research for Putman's book Bowling Alone; Collapse and Revival of the American Community - are focused on the US and do suggest a cultural specificity. How far, though, can we draw specific conclusions for the UK from this US based work? Probably not very far. Nevertheless, is there, however, something in the concept of social connectedness which has implications for quality of life? And, if so, is social connectedness always a 'good thing' and does culture or sport always play a positive role? One might question this in relation to divided communities in the west of Scotland and the place of culture and sport in religious bigotry. Yet there is a refrain throughout the literature we have read which touches on this idea of building or connecting communities and impact culture and sport has on (re-)building communities.
4.29 What can we conclude from the literature on cultural indicators? There persists a confusion, also found in the wider policy literature, about definitions: what is quality of life or well-being? Overall there is still the outstanding issue of how to develop indicators which are both meaningful and manageable. If the policy focus is to improve the quality of life at every stage and in every aspect through culture and sport, then a complex set of indicators is required and these can be expensive and difficult. The answer is to address the policy focus - be very clear on policy objectives and carefully consider what indicators suit these - not simply adopt indicators which are easy to measure.
4.30 In the cultural policy literature, we find an approach which assumes that culture and sport have a positive impact on an area or city and that cultural vitality is one aspect of quality of life. This 'cultural vitality' is measured by the opportunities for participating and attending cultural events, the development of new work and the level of economic investment etc. This approach is to be found in the US studies and in the work of the New Zealand government.
4.31 In contrast, some of the UK work has focused on proving that culture and sport have a positive impact and the measurements flow from that. Conceptually at least, cultural planning appears to bring those two approaches closer together by promoting the view that culture is an aspect of planning which sits alongside economic and social issues. Its (so far) limited application in the UK does suggest a need to develop some key qualitative as well as quantitative indicators - including some means of measuring the quality of the intervention.