Scottish Government

CHAPTER 6: DATA COLLECTION AND MONITORING

Summary

PATH's achievements to date

6.1 One of PATH's responsibilities from its inception has been making recommendations on data collection and evaluation of smoking cessation services. The main objectives of its data collection and monitoring remit are:

• To promote good practice in data collection for smoking cessation services;
• To develop a strategy for the national monitoring of smoking cessation services;
• To support smoking cessation services with data collection, evaluation and monitoring;
• To offer guidance for smoking services on using client data in accordance with data protection and related legislation;
• To support the development of smoking cessation information and research strategies.

6.2 An updated document covering recommendations for the future development of Scottish Smoking Cessation Services with regard to data collection, monitoring and evaluating, information management and essential resources was produced by PATH in December 2004 (i.e. during its first phase of funding). Since then, its key recommendations have been met and a smoking cessation minimum dataset ( MDS) (now in a revised version) and a national smoking cessation database (which is now being managed by the Information Services Division ( ISD) of NHS Scotland) have been developed and are in use. The national database is part of ISD's Data Development programme, which aims to keep pace with the ways, range and scope of service delivery and clinical practice in various services, and reflect these in the development of NHS data collection at a national level.

6.3 The MDS is the mandatory information that smoking cessation services must collect on each client setting a quit date. It is intended to enhance the validity and consistency of smoking cessation data for national and local monitoring and enable the Scottish Government and NHS Scotland to gather consistent national statistics on smoking cessation for the first time. PATH/ ASH Scotland has published guidelines for its use, which were updated in March 2007. These were devised in consultation with smoking cessation professionals and an expert Working Group. In addition, a best practice guidance document - updated in November 2006 - was published by PATH/ ASH Scotland covering aspects such as data protection, client confidentiality and access to information when using client data. It is intended to help smoking cessation services interpret what the relevant legislation means in relation to using their clients' data and summarises the principles underpinning:

• The Data Protection Act (1998);
• The Caldicott Principles for Patient Confidentiality;
• The Freedom of Information (Scotland) Act (2002).

6.4 The development of the MDS is overseen by a Project Board that includes representatives from PATH/ ASH Scotland; ISD Scotland; the Scottish Government; Public Health experts from NHS Health Scotland, Health Boards and a University; smoking cessation co-ordinators (working at Health Board level); the academic research community; and nursing. An Expert Review Group is currently undertaking a review of the MDS. It is examining requests and suggestions for changes to the MDS and the implications of these for the national database, as this must remain fit for purpose. It will also make recommendations on how services can increase follow-up rates. Its members are drawn from a similar range of professional backgrounds. There is limited overlap in the membership of the Project Board and the Expert Review Group.

6.5 The national smoking cessation database has been developed by ISD Scotland and is available for use by smoking cessation services. It was initially launched in July 2005, and a redeveloped version was introduced in March 2006. ISD Scotland has produced a User Guide for the database, which is currently in its third version. The database is a web-based system which operates through NHS-net. It enables automatic transfer of anonymised data for national monitoring and collects the smoking cessation MDS and additional information for local purposes. The system also includes features to help service management, including reporting facilities, letter templates and flagging-up clients for follow-up. At present it is only accessible over the NHS-net, which means that it cannot be accessed by non- NHS staff involved in smoking cessation. According to data provided by PATH, there were 213 registered users across Scotland in May 2007.

6.6 The first national statistical report from the database, covering the period 1 January 2006 - 31 December 2006, was published on the ISD Scotland website on 26 March 2007. Almost all of the data in this report is taken from the national smoking cessation database (and therefore based on the data recorded in the MDS). However, local information systems were used for three NHS Health Boards - Argyll & Clyde, Grampian and Greater Glasgow & Clyde. Data for Tayside pharmacies was also provided from a local information system. ISD hope that, in time, all data will be available via the MDS and the national database and will not need supplementing with data from local systems. The future looks promising in this respect, as NHS Grampian began using the national database in January 2007 and the MDS Expert Working Group includes two representatives from NHS Greater Glasgow & Clyde. The statistical data presented in the report are based on quit attempts made/quit dates set during the 2006 calendar year. The report also includes self-reported quit outcomes based on client follow-up at one month and three months post quit date. In addition to this routine reporting, ISD propose to compliment this with a programme of more detailed topic-based analysis and research. Potential topics include comparisons of quit rates with England and Wales; pharmaceutical products used in quit attempts; interventions used and their outcomes; service up-take by deprivation category and rurality; and outcomes related to specific target groups, such as young people, pregnant smokers and the most deprived communities.

6.7 During its second phase of funding, PATH has worked closely with ISD on the implementation and management of the national database. This has been a good example of partnership working, where PATH has the expertise in smoking cessation and ISD understands how to implement and operate the actual database. For example, staff from ISD often visit Health Boards to provide them with support and training to enable them to undertake the required processes. PATH staff also contribute to delivering the required training, though ongoing support (e.g. relating to the software) is provided by ISD. Both full training and 'top-up' training can be provided.

Feedback from stakeholders and discussion

6.8 The interviews with stakeholders were used to:

• Obtain feedback on the process-related lessons learned from those responsible for developing the MDS and the national database;
• Obtain feedback (including feedback on the guidance) from those providing data via the MDS for inclusion in the national database;
• Obtain feedback (including feedback on the user guide) from those using the national database (both nationally and locally);
• Ascertain the views of ISD Scotland in their capacity as manager of the national database.

6.9 Feedback from stakeholders was very consistent. Nearly all stakeholders who were familiar with this strand of PATH's work recognised the necessity of developing a Scottish MDS and national database. They appreciated that developing these has been a highly complex task and congratulated PATH and ISD for their achievements. Thus they were very supportive of the concepts.

6.10 However, stakeholders were also very vociferous when asked for their views on using these tools. For example, Health Board representatives described providing the required information as "a complete nightmare" and "the bane of my life". One recently-appointed co-ordinator of smoking cessation services said that "my predecessor just used to scream at it".

6.11 Many reasons were identified for these views, although it should also be recognised that many respondents felt that the situation was improving and were very appreciative of the training provided by ISD and PATH and the ongoing support of ISD (e.g. with respect to software-related problems). One of the main difficulties is that the system is trying to capture common information from smoking cessation services that operate in entirely different ways across the various Health Boards. Managers also said that compiling and submitting the required information was very time-consuming and required additional staff and/or considerably more work by existing staff than initially anticipated. These factors often resulted in significant

resource and cost implications. In particular, some Health Boards (e.g. those which already had good local data collection procedures in place) found that they had to manage two systems simultaneously as they moved from one to the other.

6.12 It is not intended here to try to pre-empt the findings of the MDS Expert Review Group, but several recurring themes emerged during the stakeholder interviews. These included:

• Difficulties associated with getting clients to agree to set a quit date:
• • It does not capture the time that is often required to get clients to agree to set a quit date (which can frequently take at least 6 - 8 weeks);
  • Trying to get clients to set a quit date results in some of them ceasing to attend local smoking cessation services;
• Its inability to capture reductions in smoking, given its focus on quits;
• The time taken to try to follow-up service users at one, three and (especially) twelve months - several respondents felt that follow-up should cease after three months (with one pointing out that no other NHS service routinely tries to follow-up its users after 12 months);
• Problems associated with how to record people who quit and then lapse, often several times (as frequently happens);
• The seeming inappropriateness of classifying someone who has effectively quit, apart from the occasional puff, as a failed quitter;
• Problems with providing the required information within the specified timescales.

6.13 It should be noted that many of these issues relate to the rationale behind the MDS, not to PATH's performance in its development. For example, it is important to recognise that the database was set up to gather the minimum amount of data needed to measure and report on quit attempts at a national level in a consistent manner. Furthermore, it was designed to use the same criteria and definitions (e.g. of a failed quitter) as those used in a body of established research which is recognised internationally. ⁴³ Several of the stakeholder interviews indicated that those responsible for collecting the data locally do not always seem to understand and appreciate the reasons for including some questions (and, indeed, for excluding others).

6.14 More than one Health Board representative voiced suspicions that some Health Boards were "cheating" when providing their local data. This was seen as being a significant concern, given that local data on quit rates are often being used locally for comparisons with similar and/or neighbouring Health Boards and also for assessing performance against key targets.

6.15 Despite all of the negative feedback, several Health Board personnel said that they are finding the database useful for compiling local reports and expect to find it increasingly useful as it builds up over time.

6.16 One Health Board representative stated that local academics had been employed to undertake the follow-up work with service users. This approach was felt to have been effective and had reduced pressures on NHS staff.

6.17 An important point was raised by two academic stakeholders who had been involved in the initial discussions in the early 2000s about establishing the national database. One of the initial objectives was that researchers should have ready access to the database for academic study. As all of the data are anonymised, they felt that there should not be problems associated with client confidentiality, and any issues associated with consent would have been removed by the client's original agreement to provide their data for the MDS. However, none of the staff from PATH or ISD who had been involved in these early discussions and agreements were still in post, and academic researchers have experienced considerable problems accessing the database (which have been compounded by it only being accessible via NHS-net). As a result, important opportunities to explore smoking cessation services from a national perspective (e.g. identifying which Health Boards are particularly successful at working with pregnant smokers, and exploring the reasons for this) are being missed.

6.18 The above feedback suggests that those using the MDS and the national database are gradually starting to experience their usefulness in practice as well as in the abstract. It was particularly interesting (and somewhat ironic) to note that although many respondents complained about the amount of information they had to provide, several also wanted to include additional local information that captured the specific characteristics of their service. For example, although most smoking cessation services provide 'closed' programmes for would-be quitters, some run 'open' programmes. It is therefore vital that every organisation using the MDS and the national database recognise that these are minimum requirements and other data can also be collected locally.

6.19 The stakeholder interviews and other discussions suggested that there is a potential conflict between the 'micro' requirements of those using the database primarily for local purposes and the 'macro' requirements of ensuring that full use is made of all of the data for national purposes (e.g. comparisons of the effectiveness of different approaches). It will be important in the future to ensure that the database does not place unnecessary demands on service providers whilst also meeting the broader requirements of several other stakeholders, including the Scottish Government, NHS Health Scotland and the academic community. It is therefore vital that future developments of the MDS and national database continue to be overseen and directed by an external Project Board representing both local and national interests.