The Independent Evaluation of 'Starting Well' Final Report

Listen

The Independent Evaluation of 'Starting Well' Final Report

Executive summary

Introduction

In 1999 the Scottish Public Health White Paper sought bids from health partnerships in Scotland to develop good practice in the areas of child health, coronary heart disease, sexual health and colorectal cancer. Through the Glasgow Healthy City Partnership, a multi-agency child health bid was developed and awarded funding. A key part of the rationale for this investment was the widespread belief that 'early years' interventions can help to break the cycle of poverty that limits the opportunities available to children born into the most disadvantaged circumstances. The project, 'Starting Well', was granted 3 million over a three-year period and was launched in November 2000.

The project drew extensively on the US literature on home visiting. The essence of this evidence-base is that, compared with standard health care provision, intensive home visiting had significant impacts on a range of child and family health related outcomes. The key elements of the US programmes on which Starting Well focused were: intensive visiting of families within the home; the development of supportive relationships between families and their visitors; and, an emphasis on health promotion approaches. However, the complex nature of home and health visiting makes this an evidence base that is not straightforward to implement and there were a number of ways in which Starting Well departed from the model associated with David Olds. These included:

  • The targeting of deprived communities rather than vulnerable individuals;
  • The inclusion of all new babies as opposed to only first babies;
  • A lesser focus on the antenatal period than recommended by Olds, due to the availability of Community Midwifery services in Scotland, and to caseload issues within the project;
  • The use of paraprofessionals as part of the home-visiting delivery mechanism in addition to professional health visitors;
  • The vastly different primary care context within which the evidence was derived (for example, the absence of a universal health visiting service; and related to this,
  • The requirement to integrate aspects of project delivery with existing professional and organisational structures as opposed to an entirely standalone intervention.

Notwithstanding these important modifications, the overall aim of Starting Well was originally set out as to 'demonstrate that child health can be improved by a programme of activities to support families, coupled with access to enhanced community-based resources for parents and their children'. This aim was addressed in three principal ways:

  • the introduction of an augmented programme of home visiting to all families of new babies born within two geographical areas within the City of Glasgow, selected due to their relative socio-economic disadvantage (Greater Easterhouse 1 and Gorbals/ Govanhill/ North Toryglen, known respectively as the East and the South);
  • the development of enhanced local community supports and structures within these areas; and,
  • the development of integrated organisational services to respond to the needs of children and their families both within the local areas and across Glasgow as a whole.

Evaluation

Following the decision to award funding to the Glasgow Healthy City Partnership, the Scottish Executive commissioned a multi-method independent evaluation led by the Department of Public Health, University of Glasgow. The main aims of the independent evaluation were as follows:

1. To measure the impact of the project on children and families;

2. To understand the theory, processes and context of the Starting Well intervention; and,

3. To analyse the policy implications of the project.

The independent evaluation of Starting Well was designed to be both formative and summative. To this end, the research team has produced several reports during the course of the evaluation, and two journal articles are currently in press. The final report provides an assessment of Starting Well's impact on a range of outcomes and processes and is organised around the main aims of the evaluation.

  • An assessment of the health related outcomes for children and families as assessed through a quasi-experimental survey. This was supplemented by a relatively simple contextual description of study areas that is summarised in Appendix IV.
  • A summary of the rationale lying behind the Starting Well demonstration project, as expressed by key stakeholders in the first year of the project, helps to introduce a number of key processes underpinning its implementation. These are:
    • the nature of the relationships developed between families and their health visitors;
    • the development of an augmented model of home visiting;
    • and, the development of mechanisms to support strategic change.

This part of the report concludes with a discussion of strategic stakeholders' retrospective reflections on the initial rationale for Starting Well.

  • Key findings, policy implications and recommendations are drawn together in the Conclusion.

Assessing Impact: the Quasi-Experimental Study

The most resource intensive component of the evaluation involved a quasi-experimental comparison of the two intervention areas with a socio-demographically similar area in the north of the city. This cohort study compared the health and development of intervention children over the first eighteen months of life with a group of families receiving statutory health visiting. Key health related outcomes included: quality of the home environment; maternal depressive symptoms; child dental registration; and measures of maternal service satisfaction. A designated health visitor approached all families with newborn children for consent between 01/06/01 and 31/06/02, yielding a total of 627 participants, or around 50% of all births 2.

Participating children were assessed on a maximum of three occasions (immediately after birth, then at six and eighteen months) using a combination of mother-report questionnaires, observation in the home and structured interviews with the mother. Questionnaires covered: background maternal, household and area characteristics; maternal mental health and health behaviour; and attitudes towards parenting and current health-visiting service. Each participant that could be contacted at six and eighteen months received a home visit from a trained research nurse who administered the HOME Inventory, a standardised interview-and-observation tool that assesses the quantity and quality of stimulation available to a child in its home environment (Bradley & Caldwell, 1979 3). Interpreters were made available to assist participants with no or limited English. Finally, individual-level data such as the number of home visits were collected from routine sources including health visitor records.

There are a large number of complex findings but the most important are summarised below.

  • 627/1321 (47.5%) eligible families were recruited over a 13-month period; 367 from intervention areas and 260 from comparison areas receiving the generic service.
  • Cross-sectional analyses concentrated on 359 participants completing both baseline and 6-month assessments and 294 completing all three assessments to 18-months 4. These sub-samples represent 57.3% and 46.9% of opt-ins, respectively.
  • Comparisons of aggregate-level routine data on opt-ins and opt-outs suggest no obvious bias associated with recruitment or persistence in the study.
  • Multivariate regression analysis revealed: lower rates of depressive symptoms amongst intervention mothers at 6 but not 18-months; no improvement in the quality of the home environment at 6-months but a small positive effect at 18-months (p=0.088); higher levels of client-satisfaction with levels of health visitor support; and higher levels of dental registration at both assessments.
  • Minority ethnic mothers achieved lower HOME scores and were more likely to suffer from high levels of depressive symptoms. These findings are interpreted as indicating real need amongst this group but should be treated with some caution due to the fact that key instruments (the HOME Inventory and the Edinburgh Postnatal Depression Scale) have not yet been validated in a British Asian cohort.
  • These modest findings provide some evidence of a positive Starting Well effect although the policy relevance associated with some findings such as those related to dental registration is open to question. More longitudinal data and analysis are necessary to determine the longer-term clinical and social significance of these intermediate outcomes and to assess the degree to which a 'step-change' in child health has been achieved.
  • Despite doubts as to the transferability of the North American evidence-base to the British context and a number of evaluation limitations, findings relating to maternal depressive symptoms and HOME score are supportive of shorter-term benefits to the psychological health of study mothers and potentially longer-term cognitive and emotional developmental benefits for study children.
  • Simple comparisons of area-level context (described in appendix IV) suggested a basic similarity between the intervention and control areas that did not help interpret the above findings. Lower-level comparisons, however, (for example at the level of postcode sector) revealed the potential for more sophisticated multi-level analyses that may help tease out the relative contribution of individual and area-level factors to these outcomes. More extended individual-level regression analyses remain our immediate priority, however, if future opportunities can be found to explore these possibilities, we may not only explain more of the variance in outcomes but also gain a more informed sense of the kinds of emergent community-level factors that constrain and facilitate both the operation and effectiveness of Starting Well.

In the future it would be valuable to determine whether or not Starting Well has had a direct influence on more child-centred outcomes such as readiness for school in general or cognitive development in particular. Whether or not this will be possible with the existing cohort remains to be seen, but we are optimistic. We have made strenuous efforts to put in place mechanisms for retaining contact with existing respondents and to maximise the availability of essential baseline information. Extrapolating from response rates thus far, our current assumption is that by the beginning of 2005, when the first of the study children will be 42 months old, we might reasonably expect to be able to contact approximately 500 families and that about 70 per cent of these (N=350) would respond positively to a further round of data collection. On this basis there is a strong case to be made for further follow-up that we propose to make in due course.

Theory, Processes And Context

Theory of change

A theory of change approach (Connell and Kubisch, 1998) was used to map stakeholders' views of how and why the intervention was being implemented and to capture expectations of change within a 3-year programme of activity. This was undertaken through interviews and focus groups with key strategic players, observation at steering group meetings and documentary review.

As with similar complex community initiatives, the strategic stakeholders within Starting Well struggled to articulate a Theory of Change that was wholly testable but developed a relatively robust internal monitoring system to capture the implementation of its plans, and, as with the independent evaluation, adopted a strong focus on process learning. The key assumptions underlying the initial Starting Well Theory of Change were:

  • families in deprived areas would engage in the project;
  • through the development of trusting relationships with home visitors (health visitors, support workers and community nursery nurses), families would engage in health promoting activities with the home and in the wider community;
  • health visitors working more intensively with a smaller caseload and supported by evidence-based practice guidelines, would be able to take a broader view of a family's health;
  • the employment of health support workers, predominantly from within the intervention areas would enhance the support provided by health visitors;
  • through intensive work with individual families, health visitors would be able to develop a greater understanding of child and family health needs at a community level;
  • new area infrastructures for child health would result in more responsive local statutory and community supports for families;
  • a senior level project steering group would provide the driver for strategic change; and
  • this whole system and individual family level intervention would result in a step change in child and family health in the longer term.

Processes

A substantial part of the evaluation investigated three key issues that were an integral part of Starting Well's Theory of Change:

  • the extent to which intensive home visiting led to the development of therapeutic relationships between families and their home visitors;
  • the implementation issues involved in developing a skill mix approach to home visiting; and
  • the degree to which intensive home visiting at an individual family level led to improved community and strategic responses to child and family health problems.

1. Family Case Studies

A primary aim of this evaluation component was to examine the formation and the operation of the relationship between the child's key-care giver and their health. A second broader aim was to describe the developing views of both sets of participants on key aspects of the service. Key findings include:

  • Analysis of fifty-nine individual interviews with a diverse group of 'Starting Well' mothers and their health visitors attempted to understand the interpersonal processes that underpinned the project's operation at the level of individual families.
  • A process model was identified that linked demonstrably intensive home visiting input to a diffuse set of benefits summarised as 'enhanced support' (comprising: increased confidence; reduced anxiety; reduced isolation; the opportunity to confide; and experience of advocacy).
  • The process model describes how intensive visiting equated to more time and direct contact with mothers during a period of universal need which encouraged the rapid formation of a trusting relationship, an individualised care package and the provision of more and better quality information on needs and life circumstances. This in turn, was associated with the identification of a broad range of problems and problem-solving activity and an enduring two-way (functional) dialogue between mother and health visitor. In sum, these processes promoted perceptions of enhanced support. Lack of maternal receptivity to the service and health visitor caseload pressures explained variation in process and outcomes.
  • Intensive visiting can be an effective way of delivering a more patient-centred, 'holistic' model of care. Precipitating factors include: the convenience of the home setting; the shift in power relations inherent in the mother's control of access to the setting; and a concomitant need for the health visitor to maintain access by a) providing a flexible service and b) establishing a positive, non-directive relationship.
  • Project health visitors praised teamwork, training and aspects of the approach (intensive support, skill mix) as strengths but had experienced resistance, scrutiny, and larger, more demanding caseloads than initially anticipated. These latter factors may, at times, have impeded their capacity to deliver the service as intended.
  • Support was voiced for a universal intensive service in the first postnatal months, provided that it had the capacity to target sub-groups of women with higher levels of identified needs, for example, primiparous, isolated or depressed women.

2. Skill Mix

The process of developing and implementing an augmented model of home visiting was investigated in a report accepted by the Scottish Executive in June 2002, which was based on qualitative interviews with both strategic and operational staff at two time points. The final report builds on the findings of that fieldwork, where relevant, but focuses primarily on two specific issues: the extent to which Starting Well's model of home visiting was perceived to have acted as a vehicle for changing health visiting practice, and, the degree to which the project's health support worker model worked in practice.

The key findings related to these issues are set out below:

  • In a relatively short space of time Starting Well developed two project teams incorporating a new type of worker (the health support worker) alongside a professional group of long-standing (health visitors) in order to implement its home visiting model.
  • The project attempted to develop a standardised approach to health visiting but the degree of consistency achieved within practice was variable due to caseload size and some professional resistance to the notion of standardisation. This has wider implications for the use of standardised family health plans.
  • Pressure of caseload size limited some of the project's aspirations and led, for example, to the need to rethink the ability of the model to be applied universally even within a deprived community.
  • The project teams developed very differently in the two intervention areas. These differences were due to levels of individual, professional and organisational buy-in/resistance to the Starting Well model.
  • The two emerging 'models' differed in the degree to which they advocated integration within GP practices and in the dilution of the Starting Well approach.
  • The lessons that can be learned from these two manifestations of the project need to take account of the central role of organisational context in defining and supporting practice.
  • In relation to skill mix, whilst much good practice was identified in bringing together health visitors and nursery nurses, issues of role clarity remained problematic throughout the life of the project. This has implications for the wider development of skill mix approaches, especially where new professional groupings such as community nursery nurses are introduced as single individuals to an existing primary care team. Time, training and supervision are all necessary at both local and strategic levels.
  • Health visitors did not, in the main, develop their practice in relation to community development 5 and this has implications for the implementation of Hall Four, the most recent recommendations from the Joint Working Party on Child Health Surveillance, (Hall and Elliman, 2003).
  • The role of the health support worker developed into a diffuse and flexible one. It was a role that was, generally speaking, viewed positively by team members but one that required careful supervision.
  • The employment of the support workers through a voluntary sector organisation allowed a supportive model of engaging individuals with a knowledge of the local area who might not previously have been engaged in the labour market. This was a model that was perceived to have been beyond the current capacity of the NHS.
  • The dual management structure, however, led to operational difficulties around the day-to-day deployment and supervision of health support workers, some of which had their roots in a lack of professional ownership of the social inclusion aims of the health support worker approach. This may become an important policy issue in the future as public sector organisations develop ways of expanding their workforces.

3. Community and Strategic Change

Starting Well was conceived as a project that would impact not only at the level of an individual child and family health but also at the level of community and organisational capacity to respond to the health needs of local families. In assessing its impact at these levels the final report considers two sets of issues: the degree to which community infrastructures and supports have been strengthened through Starting Well activity; and the extent to which the project is perceived to have impacted on the strategic planning and provision of child and family health services. The key findings are summarised below in relation to 5 key questions:

1. To what extent did the process of intervening intensively with individual families result in an understanding within project teams of key community health needs?

  • More intensive contact with families helped health visitors to understand health needs at a community level. This understanding was also shaped by communication within the project teams, working with other agencies and pre-existing knowledge of the local communities. A wide range of needs was identified but none were believed to be 'new' issues.

2. Did a shared understanding of needs feed into local implementation groups?

  • The process of sharing perceptions of community level need was rather haphazard within the two project teams and its success appeared to be a function of the level of collaborative working, with one team in particular demonstrating a significant lack of cohesion.
  • The increasing burden on health visitor caseloads and the early lack of clarity in the role of the community support facilitator (as perceived by other members of the project teams) led to a lesser emphasis on advocating for community change within the health visitor role, which questions the assumption that health visitors, and the changing systems with which they work, are ready for the challenges posed by Nursing for Health (Scottish Executive, 2001).

3. Did local implementation groups develop and support local, community solutions?

  • The local implementation groups were perceived to have been successful in disbursing monies from their development funds to local organisations but less effective in securing representation from both key statutory agencies and local parents.
  • The role of the community support facilitators and the bilingual worker became key in bringing about more sustainable changes at a local level in response to identified needs and in liasing with other relevant child health fora. Whilst there is evidence of much good practice at a local level, the implementation groups suffered from the poor level of representation from community and wider statutory groups that plagues health and social care projects more generally.

4. To what extent were difficult issues referred to the project steering group?

  • During the course of Phase I of Starting Well, only a small number of issues were passed from the local implementation groups to the project steering group (for example, the lack of breast feeding facilities within a local shopping centre and the high level of maternal mental health problems within the communities). None of these resulted in significant change at a strategic level and the members of the local implementation groups showed little knowledge of the role of the steering group. As with many complex interventions, a lack of connection between the city-wide and the local, and between the strategic and the operational, served to limit the effectiveness of organisational structures that the project established.

5. Did the steering group advocate for more strategic, Glasgow-wide solutions?

  • The project steering group did not succeed as a mechanism for strong partnership working around the child and family health problems experienced in poor communities. A lack of ownership of the project beyond the health partners, and a lack of commitment to tackling broader strategic questions, were discussed by strategic stakeholders. On the other hand, there was some evidence provided of more constructive strategic work occurring 'behind the scenes' (for example, joint working around Sure Start and the development of additional money advice supports to vulnerable families). Given that partnership working lies at the heart of current public policy, it is imperative that further development in this area is taken seriously.

Theory of Change Revisited

Towards the end of the evaluation, the same group of individuals who articulated the project's initial Theory of Change, were asked whether they viewed the project as having been successful (in other words, was the Theory of Change doable in retrospect?). They were also asked for reflections on the meaningfulness of the original project plans. The key findings were that:

  • Those involved at the most strategic level within Starting Well were largely optimistic that the project had met its objectives in relation to the home visiting and community support components of the project. Most were much less positive about the objectives set around partnership working, and the extent to which these had been realistic, given levels of commitment and shared ownership, was questioned.
  • There were mixed views as to whether the types of short-term objectives set by the project were meaningful in leading to a longer term step-change in child and family health.

Whether or not the initial aims were appropriate it is clear that Starting Well did not, for example, demonstrate the step change that it aspired to as measured by the most important health related outcome that we investigated, and nor did the process of service development and delivery run as smoothly as predicted. Despite this, however, the complexity of the Starting Well experience should be recognised. It was highly valued by many of the staff and individuals involved, and there are valuable lessons to be learnt from it about the implementation of future initiatives. It may even be that further analysis of the quasi-experimental data will show stronger intervention effects. There is also a possibility that such effects only emerge as the Starting Well children get a little older, provided that attempts are made to look for them. The case for doing so will be made in due course. For now we conclude with a consideration of some of the reasons why Starting Well has not met all of the expectations of those who commissioned and designed it.

Wider Policy Implications

The most important implications for policy and practice that arise directly from the findings of the evaluation have been summarised above. Here we draw together a number of conclusions and reflections on the process of designing, implementing and learning from a Demonstration Project such as Starting Well. In particular we highlight four key issues: project planning; the meaning of 'demonstration'; the use of the existing evidence-base; and, achieving professional and organisational change.

  • Project planning has to strike a balance between constriction and chaos. Planning and monitoring change in complex systems require time and capacity development; these were in short supply at the point of commissioning the Demonstration Projects. It is a positive sign that this has been recognised at a national level in commissioning Phase II although the meshing of independent evaluation findings with the development of this second phase has been less than ideal since the time for such findings to shape future plans is extremely limited.
  • Far greater clarity and consensus is required for future projects and for Phase II of Starting Well if the project is to avoid being pulled in two opposing directions. This is particularly salient in relation to the debate around the most appropriate 'model' of Starting Well where there are stakeholder concerns that current policy thinking about 'integration' will force the project down a particular road regardless of its initial aims and underlying principles.
  • In assessing the degree to which demonstration projects have or have not applied evidence-based practice, the complexity of the application of evidence needs to be considered. For a range of contextual, methodological, practical and philosophical reasons it may not be appropriate to transpose evidence from one setting to another.
  • Implementing new ways of working within and across professional boundaries, and establishing meaningful community and partnership approaches, should not be viewed as straightforward, uniformly supported or inevitable outcomes of delivering a project. Greater realism will be required to turn around well-established ways of working.

Notwithstanding the very real issues of design and implementation highlighted above, there is much to learn from the Starting Well experience. Although the commitment to improving the early years experience of the poorest children is not in doubt, the evidence base to guide effective action is less secure than once was thought. This is particularly true of home visiting programmes in the UK. In these circumstances, the renewed emphasis on promoting social justice by reducing child poverty in all its forms, and the growing recognition of the importance of evaluating promising public health interventions exemplified by the second Wanless report (2004), suggest that the lessons to be learnt from Starting Well are important ones that should not be neglected.

Page updated: Thursday, March 24, 2005