Scottish Government

CHAPTER SIX: LEARNING POINTS AND CONCLUSIONS

Key Findings

Development and Implementation

• Interventions need longer lead times, realistic aims and measurable objectives and outcomes.
• Partnership working was a factor in successful engagement.
• Organisational restructuring, bureaucratic procedures and mixed messages about the purpose of the funding delayed implementation.
• Promising pilots appeared to have more defined objectives; good partnership arrangements; good access to existing local resources; and a supportive wider organisational context.
• Lack of follow up was a key service weakness.

Impacts

• Attempting to achieve multiple objectives probably slowed down the development of each one within the time available.
• Key referrals were made to existing 'easy' to access health services; other services were well less used.

Policy learning

• Similar issues were both motivating and inhibiting factors for WMS uptake for different men.
• Men favoured mainstream health services, but one size does not fit all.
• Risks of smoking are accepted by men; but alcohol misuse and physical inactivity were not regarded as problems.
• Men liked physical tests and measurements, and valued follow up.
• Primary care services should develop a gender-friendly health improvement ethos to make both men and women feel welcome.
• There was a lack of referrals for health promotion services other than smoking cessation, possibly through lack of availability.

Project planning and evaluation

• Despite the 'demonstration' nature of the pilots, evaluation was not high on the agenda of most service providers, and there was a lack of planning for their contribution to the evaluation process.
• The project changed and evolved in response to population uptake during the implementation phase making it harder to measure cause and effect.
• A theoretical underpinning would help guide future projects.
• Research gaps have been identified in several areas.

Introduction

6.1 This chapter discusses the main themes and conclusions to emerge from the study. It presents the main lessons learned about the development, implementation and impact of the WMS, including the extent to which the pilots' objectives were achieved within the lifetime of the WMS project. It concludes by discussing the main implications to emerge for any future men's health policy or practice developments and some of the challenges that emerged from conducting this evaluation.

Development and Implementation

Scope and timescale

6.2 The expectation that the projects would have been able to address all three overall pilot aims (promoting healthier lifestyles and attitudes amongst men; providing men with the opportunity to undertake a health assessment; and effectively engaging all men), even within a time frame unaffected by subsequent implementation difficulties, was rather optimistic. Health promotion programmes can take at least a year to be implemented, and the first year is often characterised by little obvious impact ( HEBS, 1999; Whitelaw et al. 2006; SE, 2005c). Although the WMS project was commissioned to run for a period of two years, most pilots ran as functioning services for a very short period. The strategic context, which sought both community development and service delivery, also contributed to widening the scope of the projects.

Planning

6.3 A striking feature of the pilots' original bids was the lack of explicit discussion or apparent cognisance of possible threats or challenges to successful implementation, illustrated by the delays in starting up the pilots. Pilots lacked realistic and rigorous objectives, or theoretical underpinning. Those that had some of these features tended to report more success in achieving their aims than other pilots, and the authors believe that these features contribute to more effective programme implementation and evaluation design that is fit for purpose.

Partnership working

6.4 Most pilots were led from within the NHS, which brought a certain perspective and both barriers and opportunities. Those pilots that reported considerable investment in developing their partnerships during the process of creating and submitting their bid, or where partnerships existed prior to the WMS, were arguably more successful in engaging with their intended target group of men, and in making links with other support services. It is recognised that the development of successful community-based partnerships requires long-term commitment.

Introducing new initiatives within the existing health care system

6.5 Embedding WMS within health services meant that projects could draw upon existing resources. However, organisational restructuring, perceived lack of senior management commitment, and the administrative requirements of a large bureaucracy led to frustration amongst staff who were keen to get on with the job but felt a lack of strategic direction. More lead time needs to be factored in at the planning stage before projects are likely to be fully implemented.

Different values and perspectives about funding

6.6 The strategic aims of the Men's Health strategy and the wide scope of the pilot objectives meant a rather mixed message was received about the aims and purpose of the funding. In particular, there were tensions between the community development aspects of the WMS and the delivery of comprehensive health assessments. There was a perception held by some project staff that the Camelon model had been imposed on the pilots. There is a need to ensure either that objectives are clear and consistent or that a framework is established for setting priorities between multiple objectives.

Pilots that showed promise

6.7 While it was difficult to establish factors that were associated with successful pilot implementation due to the short time scale, those pilots that were beginning to show signs of promise ( e.g. getting clients through the door) had common features. These included having defined (realistic) objectives or a detailed project plan; motivated and committed staff; good partnership arrangements; good access to existing local resources and support services; and a supportive wider organisational context. Lack of follow up was considered a key service weakness by the users and providers.

Impacts

Achieving objectives

6.8 All of the pilots undertook activities directed towards the first objective of promoting healthier lifestyles and attitudes amongst men. The emphasis on general awareness raising activities varied, as did the degree to which these were documented by the projects. The scale of activity achieved through the pilots would not have been sufficient to have a significant impact at the population level, during their period of operation. Most pilots achieved the second goal of establishing a formal health assessment, although the nature of this varied. For example, some included a physical assessments others conducted a brief ' MOT' type assessment.

6.9 The third objective, effectively engaging with all men, was also partially achieved. Although 3,367 men were recorded as using the WMS, the pilots did not generally reach the hardest-to-reach men although there were some notable exceptions. However, it needs to be borne in mind that there were gaps in the available data returned to the evaluators. Hard-to-reach groups may be less likely to complete the individual level data and the pilots differed in the extent to which they documented and quantified their outreach activities.

6.10 Most staff thought that the pilots had been successful in achieving their locally determined short term goals, but not the medium or longer term ones. A few reported that their project had not achieved any of their goals due to insufficient time and the lack of realistic project objectives.

6.11 We think it is possible that, as a result of WMS staffs' push to try to achieve all the WMS pilots aims at the same time, that the time spent developing community links may have reduced capacity to undertake health assessments, or vice versa, and that the tensions between different strands of the strategy had the net effect of diluting the overall (potential) impact of the WMS initiative.

Contribution to Men's Health Strategy

6.12 The men's health strategy was built upon the key principles of establishing strong community links; developing appropriate support mechanisms; and providing a comprehensive health assessment. The projects with the strongest community links were those where pre-existing partnership working was in place and the timescale of the pilots provided limited opportunities for new community development work to establish results. The second objective was partially achieved as was evidenced by referrals to GPs, practice nurses and smoking cessation services. However, such referral pathways were already open to men without the need for referral by another service or health professional. Other potential interventions or support services were less well used, although a few pilots did set up their own follow on services, e.g. weight management. As indicated above, all of the pilots provided health assessments to a varying extent.

Policy learning

6.13 Despite the limited nature of the WMS pilot project, there are a number of emergent findings about men's health and men's views about primary care services in Scotland that could be used to inform any future men's health work.

Factors that encouraged and inhibited WMS use

6.14 Men who used WMS pilots reported doing so for three reasons: first, curiosity about the WMS as well as a motivation to seek information about their health (sometimes about a pre-existing condition) from other health professionals was associated with service uptake. Secondly spouses, partners or friends played an influential role in encouraging men to attend and, thirdly, the perceived convenience and accessibility of the WMS services were factors in service use. However, factors that encouraged some men, paradoxically, inhibited others; e.g. services provided during the evening in some urban locations were considered inappropriate by men because of concerns about being out in the evening in the areas in which they lived.

No 'one size fits all'

6.15 In general, men favoured services that were located in mainstream health care settings, offered more in-depth consultations and tests, and incorporated some form of follow up work. However, there was diversity in men's preferences and we conclude that no one model or approach to addressing men's health is suitable for blanket application across Scotland, and believe that the optimal standard for the development of men's health services is one that should be dictated by needs of the target group, health topic, setting and local circumstances; a view which is consistent with the position advocated by Judd et al. (2001). In order to reach men, it may be necessary to offer a range of services in different settings. Direct targeting of specific groups of men was often successful; reaching homeless men in East Glasgow by taking the service to the heart of their community was a prime example. Other pilots achieved success in engaging with LGBT men and gypsy travellers by working directly with key stakeholder groups from these communities.

Men's health: Scope for health improvement

6.16 As far as we are aware, this was the first attempt in Scotland to engage directly with men (in a comprehensive way) to explore their views about their health and well-being; what changes they feel they want or should be making to their lifestyles (to make them more health enhancing); and how and where they think primary health services should function to support their health seeking behaviour.

6.17 The health risks of smoking are clearly accepted amongst men, but there is a stark difference in their perception of other health behaviours, such as alcohol and physical activity, as health problems (see 3.29 Table 1). However, an encouraging proportion of men expressed a desire to change their behaviour, indicating that great potential for men's health improvement exists. The experience of tackling smoking suggests that capitalising fully on this potential may require concerted action across government both to lower barriers to healthy behaviours and to discourage unhealthy behaviours, in addition to individualised health improvement interventions.

Testing and follow up

6.18 Men valued having physical tests and measurements taken, especially blood pressure ( BP) and cholesterol, during their health assessments, and for many the prospect of these had encouraged them to come and use a WMS pilot. Yet it was clear that there was concern about (and resistance to) wholesale testing ( BP and cholesterol) of WMS users amongst some health professionals. It is beyond the scope of this evaluation to suggest what appropriate clinical practice (for BP or cholesterol) should be in the context of well men services, but we recommend that there is further exploration and development of protocols that would meet both the expressed and felt needs of men and the clinical evidence base, in order that this 'engagement opportunity' is not squandered. Developing follow-up services would also enable men to have access to sustained support in addressing any health concerns identified.

Local interpretation and implementation of nationally determined initiatives

6.19 The WMS pilot programme was perhaps overambitious in attempting to (a) address a complex issue, such as men's health, where there is no clear evidence base for interventions, and (b) contribute to the development of that evidence base. This pilot programme also encouraged a diversity of approaches to delivering WMS, with the intention of identifying which approaches were better than others, but it also had multiple objectives that projects interpreted in different ways. This was therefore a commendable attempt to commission health services developed with input from both the intended target groups and professional groups and organisations (a blend of so so-called 'top-down' and 'bottom-up' approaches). While this is considered to be ideal health promotion practice, in reality, it is difficult to achieve, and tensions and misunderstandings are commonplace within this approach, as evident within this project.

Primary care

6.20 Primary care remains a key setting in which to reach a large proportion of men. WMS users were in the habit of consulting their GP, but were reluctant to raise lifestyle, prevention type queries with them. There are a number of strategies that would improve current practice in engaging with them about health promotion issues in this context. Mainstream primary care services could develop a 'gender-friendly, health improvement' ethos, that encourages men (and women) to raise all kinds of health concerns with their GP, not just those related to a current health concern or illness.

6.21 Concerns about the availability of, and referral to, follow on services other than smoking cessation also remain. Having successfully engaged men in considering their health, the low levels of further intervention with respect to alcohol and physical activity represented a missed opportunity. More attention should be given to identifying and putting in place appropriate interventions before general health promotion activities take place.

Project planning and evaluation: Twin challenges

6.22 Although project proposals indicated that staff had agreed to conduct or engage with evaluation work as a condition of the funding, there was a general lack of engagement with the evaluation processes during implementation, illustrated not only by the nature and quality of the data received from many of the pilots, but also through insights gained through the staff interviews. It was apparent that little formal planning for evaluation had taken place; most pilots had not identified additional resources or mechanisms (existing or otherwise) to collect information and the majority did not have the capacity to undertake this function. Having said that, the vast majority of staff did recognise the need to collect activity information.

6.23 The diverse range of perspectives (individual, professional and organisational) that emerged about the most appropriate ways to address men's health ultimately influenced how staff viewed the intent of the WMS funding and, their notions about relevant criteria that should be used for evaluation. The commissioning process (of projects of this nature) needs to take account of this phenomenon and to ensure that adequate time is factored in to clarify such issues, and address any tensions that will inevitably arise during the bid development stage. This is particularly important when dealing with diverse, multi-agency partnership groupings.

6.24 There was a lack of appreciation that this pilot project was effectively a 'demonstration' project, which was entirely different from previous projects in which staff may have been involved. Frontline staff could have been better informed about the nature and purpose of the evaluation. However, individual and organisational deficits associated with health promotion evaluation capacity are systemic throughout Scotland ( NHS Health Scotland, 2006). Commissioners and programme planners may also wish to require or recommend that more time and resources are focussed on capacity building work for frontline staff employed on demonstration projects in the future.

6.25 Most projects also evolved and changed in order to engage more effectively with their intended target group. This level of project 'instability' presents evaluators with challenges in determining which particular aspects of a service should be linked with any of observable impacts or effects that might emerge.

6.26 There is a need for project planners intending to implement new complex, multi-agency, community-based projects (like the WMS) to develop partnership models of working that can be evaluated. Future funding should require all potential bidders to develop and submit proposals that include an evaluation plan, alongside the proposed project plan. This might encourage project planners to articulate the 'theory' of their project. None of the WMS pilots ⁶ indicated any sort of theoretical basis; a persistent criticism of health improvement projects (Hawe et al. 1990; Tones and Tilford, 1994). More crucially, such proposals may also encourage planners to think in more detail about project and evaluation requirements, and consider all related resource implications. Evaluation plans should also be developed with external evaluators at a much earlier stage, particularly for larger scale projects. The use of planning models ( e.g. logic models) may have assisted in addressing these points (Cooksy et al. (2001), WK Kellogg Foundation (2004)).

Further Research

6.27 The evaluation suggests that further investigation is required to explore the professional and organisational capacity, particularly within primary care, to deal with possible increased demand for health improvement work by men.

6.28 Due to the short operational time frame of many of the WMS pilots a question remains about how effective sustained community development can be in challenging men's attitudes towards their health and health services use.

6.29 It is also important to identify effective ways of engaging with young men and men without partners with health improvement efforts. These groups emerged as hard-to-reach by the WMS pilots.

6.30 Lastly, the impact of the WMS on health outcomes over time could not be addressed in this study because of the time scales involved. However, some projects have the potential to follow up their clients and it should possible to explore this issue in the future.