Scottish Government

Chapter Seven Kirkpatrick Level 3: What did people do as a result of the training?

7.1 The previous chapter explored what participants learned (in terms of confidence, knowledge and skills) from their ASIST training. Changes in learning, however, do not necessarily lead to changed behaviour. This chapter will examine the extent to which participants transfer their ASIST learning into practice in either a professional or personal capacity (Kirkpatrick level 3 outcome).

7.2 In the case of suicide intervention, obtaining direct measures of the application of learned skills into practice (i.e. observing a real life interaction) is both practically and ethically impossible. Therefore, our evidence regarding Kirkpatrick level 3 outcomes is derived mainly from self-report measures (participant online survey and interviews), supplemented by reports from managers whose staff have been ASIST-trained, and other indirect measures reported in the ASIST literature.

7.3 For the purposes of our analysis, we have defined an 'intervention' as the use of one or more of the elements of the ASIST suicide intervention model ( SIM). The elements include:

• asking someone if they were having thoughts of suicide
• exploring reasons for wanting to live and / or die
• reviewing the person's risk
• making a safeplan.

7.4 We believe that this broader definition of 'intervention' produces a more accurate reflection of the application of ASIST learning into practice, than defining 'intervention' as the use of all of the SIM elements. For example, in the case of an ASIST trainee who had asked someone if they were thinking about suicide, but the response was negative: this would still count as an 'intervention' for the purposes of our analysis, as the person had used their ASIST learning to recognise risk and ask directly about suicide.

7.5 This chapter also reports some of the qualitative evidence we gathered in the survey and in local implementation studies which shows how people used the ASIST model (or parts of it) to intervene, as well as situations in which interventions may have not gone well.

Intervening with a person at risk

Intervening before and after training

7.6 In our survey of ASIST participants, more than half (58%) reported they had intervened with a person at risk of suicide prior to their ASIST training. The number of participants who reported intervening following training rose to over three-quarters (78%). This finding represents a 20% increase in intervention following training.

7.7 A comparison by employee group shows that:

• NHS staff reported an increase in intervention from 70.3% before training to 90.4% following training.
• Voluntary sector staff reported an increase in intervention from 63.8% before training to 86.6% following training.
• Local government employees reported an increase in intervention from 58.4% before training to 76.6% following training.

7.8 Our analysis suggests that the percentage of interventions both before and after training are highest among NHS staff, followed by the voluntary sector and are lowest among local government employees. However, the 20% increase in intervention following training is consistent among all three employee groups.

7.9 The percentage of participants who reported applying their ASIST skills into practice in our survey is higher than the figure reported elsewhere in the ASIST literature. The literature typically reports an average of 50% of participants who say they have used their ASIST skills at least once with a person at risk of suicide following training. The difference between findings from our survey and findings reported in the literature might result from any of the following:

• Variation in follow-up times: International evaluations of ASIST have typically obtained follow-up measures within three to six months post-training, whereas our survey sampled participants who have attended the training up to four years ago, hence, having more of an opportunity to apply their skills.
• Targeting of training: ASIST participants in Scotland might be individuals who come in contact with people at risk of suicide (in a professional or personal capacity) more often than participants in other places.
• Measurement: In our survey we have defined and measured an 'intervention' as the use of one or more of the elements of the SIM model. The definition of an intervention in the ASIST literature is vague, and it is not clear whether respondents had understood an 'intervention' to include the use of elements of the model as well as following through the model in full.

Intervention outcome

7.10 Eighty-seven percent (87%) of survey participants, who have intervened following training, reported having (one or more) experiences using ASIST when it went well. The following quotes from participant interviewees illustrate how they perceive the effectiveness of their intervention:

I did go through the model with him and at some point he said, 'You know what, I couldn't really do this...'. Exploring his ideas with someone else had allowed him to think things through and realise that suicide is not the answer.

He continues to present sometimes in a histrionic way and difficulties in family relationships are still apparent. However he has been able to make continuing use of the resources network which he started to build up from after our early discussion.

Several weeks later, and lots of chats, she is holding down a trainee position with the organisation that could result in a full time job.

The client was going through a period of self harm, there were many factors in her life which led to this. On one occasion when I went to visit her she said when I left she was going to take her own life. I spent two hours talking with her. Eventually she agreed to keep herself safe until I could take her to a project the next day that deals with suicide intervention.

I used the model with a couple of other people - a close relative who lost his wife of 60 years and another personal contact who had been out in touch with me. Neither were suicidal but appreciated being asked.

7.11 These quotes illustrate the positive impact that the use of various elements of the ASIST model (exploring invitations, asking directly about suicide, exploring reasons for dying and living, reviewing risk, making a safe-plan, and linking to resources) had on individuals who were perceived to be at risk of suicide.

7.12 Only 4% of survey participants reported having had experiences using ASIST when it did not go well. Here are some quotes from these participants:

I have been in a situation where I asked the right questions, but the person chose to verbally deny that he was thinking about suicide and killed himself a couple of hours later. Nothing prepares you for that and it seems that time and loving friends / family are the only healers.

On the occasion it did not go well, the model was not at fault. Possibly no intervention would have changed the outcome.

Without ASIST I would have felt worse if his suicide attempt had been successful. I know I did the best I could. I now know the change in body language, voice and barrier that accompanies that intention, and know not to take them personally and to persevere with positive intentions.

7.13 The remaining 9% were unsure as to whether their interventions went well or not:

I don't feel I did any harm. Maybe made him aware I was there. There was no reaction from him, either positive or negative, about me asking.

Elements of the model

7.14 Just over a quarter (26%) of survey participants reported having followed all stages of the ASIST model in their interventions. More than half (59%) reported having used parts of the model. (Note that this figure includes cases where participants asked someone whether they were feeling suicidal and the answer was 'no'.) The other 15% either used a different model, hadn't used any model at all, or were unable to recall.

7.15 Only one of the 22 ASIST participants we interviewed used the ASIST prompt card in an intervention. Whereas some participants did not feel the need to use the card, others were uncomfortable about using it. Here are some quotes from the interviewees who have not used the prompt card:

No - didn't feel the need.

No - was fairly soon after training and still fresh in my mind. I carry it with me though.

No - I felt that it would seem like I didn't know what I was doing, or that I was trying to quantify her experiences.

No - it was in my handbag and I couldn't stop the conversation to get it. I carry it everywhere though.

Not in front of her or with other people.

No, don't know why. I do carry it in my purse.

Things that work well in putting ASIST into practice

7.16 Participants interviewed for this study highlighted several elements they felt have gone well in their interventions with individuals at risk using the ASIST model. The main elements that were perceived as helpful included:

• being able to recognise the signs in someone thinking of suicide
• having the confidence to ask a person directly whether they are thinking about suicide
• having a structured model to follow through
• being able to establish a "safe plan" and link the person to resources.

Challenges in putting ASIST into practice

7.17 The most challenging aspects of using ASIST, according to participants, are asking people directly about suicide and being personally involved. Here are some quotes from our survey participants and interviewees:

It was difficult to ask the question, but training had given me courage.

Asking the question was the most difficult bit, but it actually helped her and it made it easier to move on in the conversation. It actually reassured me that it's not so difficult to be direct as I had previously thought.

It was hard because I'm personally involved - she's so close to me. It's always easier when you can distance yourself.

7.18 One interviewee found it very difficult to deal with a situation where her patient had refused to let her disclose her suicidal ideation to other staff involved in her care:

She said she had been buying lots of Paracetamol and was going to take it. I asked her how many she's got and she said 12 packets. I asked her permission to talk to the staff and she refused. I was in a big dilemma, as on the one hand I have a duty to tell the staff if someone is threatening to kill themselves, but on the other hand that would break any confidence she has in me, which is the basis for our relationship, and if she finds out I told somebody when she asked me not to, it could drive her over the edge. I decided not to tell anybody and tried to make a safe plan with her - to get rid of the pills in order to avoid temptation.

Reports from managers

7.19 To supplement our findings from participants' self reports, we asked service managers in a number of local authorities and organisations whether they had noticed any changes in the work-related behaviour of ASIST-trained staff members, which could be reasonably connected to the training.

7.20 Several project managers in our LIS areas reported that they had actually observed their staff putting ASIST into practice (either in a face-to-face situation with a client, or on the telephone). In one project in Glasgow, the manager said that she could see the empathy that staff had with clients as a result of ASIST training. They were confident and not afraid to ask the question about suicide intent.

7.21 In Midlothian, a service manager in a mental health service reported having heard stories from staff of their use of ASIST with clients to good effect. She had no doubt that it raised awareness and made people more confident to intervene. However, she felt that participants' confidence may not be sustained if they do not use it regularly.

7.22 Another manager in Shetland reported that their staff discussed their interventions with clients in staff meetings. This was seen to be a marked change directly resulting from workers' attendance on the ASIST training.

7.23 In addition, in our interviews with ASIST trainers (many of whom are themselves service managers), there were quite a number of cases highlighted where ASIST participants had applied their learned skills within 24 hours of completing the training.

Other indirect measures of Kirkpatrick level 3 outcomes

7.24 Our review of international ASIST literature identified two evaluation studies that employed additional indirect measures (other than participant self-reports or reports from managers) for the application of knowledge and skills into practice.

7.25 Perry and McAuliffe (2007) evaluated the implementation of ASIST in a large community hospital in Canada. To complement staff self-report measures, the authors measured: (a) the proportion of clients that staff routinely assessed for suicide risk; (b) identification of suicidal risk among mental health patients in the Emergency Department; and (c) admission rates of suicidal patients presenting in the Emergency Department. All of these measures were taken repeatedly over a four-year period. Following training there was an increase of between 14-21% in the identification of suicidal risk among mental health patients and more staff assessed a higher proportion of their clients for suicide risk. There was also a steady reduction in suicidal patients' admission rates (from 56 to 42%), reflecting (according to staff) the clearer process of exploring reasons for dying and living and an increased focus on strengthening the client's protective factors in the community, which enabled some admissions to be averted. The findings could suggest that knowledge and skills were transferred effectively from the training context to the workplace, however there is no way to ascertain a casual relationship to be attributed to training.

7.26 A second paper evaluated ASIST training which was provided to primary and secondary school staff members in Virginia, USA (Cornell et al, 2006). Over a period of two years, the evaluators measured: (a) the number of referrals to mental health services; (b) the number of students questioned about suicide; and (c) the number of contracts made (not to engage in suicidal behaviour) with potentially suicidal individuals. The authors carried out two studies: the first compared the above measures before and after training, and the second compared the above measures between ASIST trainees and controls.

7.27 In the first study, the authors found that, following training, trainees made more referrals to mental health services than they did pre-training. However, in the second study, the authors found that, overall, the control group made more referrals than the trainees did. It is not clear whether an increase in the number of referrals to mental health services is interpreted by the authors as being a desirable outcome. On the one hand, it is said to reflect increased awareness of signs of suicide risk, but on the other hand, it is also said to reflect a lack of confidence in one's ability to help individuals who are at risk.

7.28 The number of students questioned about suicide did not differ pre- and post-training. Moreover, when trainees were asked about whether they had wondered if a student might be suicidal but decided not to question that student, they reported this to happen on average 6.7 times a year. For the control group this figure was only 0.7 times a year. This finding may reflect trainees' improved ability to detect suicidal signs, but it does nevertheless suggest that they do not always act on their concern.

7.29 Finally, in the first study, the authors found no significant difference between the number of contracts not to engage in suicidal behaviour made before training and three months after. However, in the second study the authors found that within two years post-training, trainees had made more contracts with suicidal individuals than did a control group.

7.30 In summary, the international ASIST literature offers some evidence, which does not rely solely on self report measures, to suggest that ASIST participants indeed transfer their knowledge and skills into practice. While this evidence supports and complements participants' reports of their own behaviour, it should be viewed with caution, because:

• Any interpretation of the findings as evidence for the successful transfer of ASIST learning into practice lies on the assumption that measures of admission rates, referrals, assessments, etc, are indeed a reflection of behavioural change in course participants. While this is a possibility, there are likely to be other factors that influence these measures which could not be causally attributed to training.
• The findings from the literature are largely inconclusive and could be interpreted in a variety of ways.

Profile of intervener and non-intervener

7.31 In this section, we outline a profile of the 'intervener' (someone who has intervened with a person they believed to be at risk of suicide following their ASIST training) and a profile of the 'non-intervener' (someone who hasn't intervened following training). These profiles could be useful for reflecting on the targeting of future training in order to maximise effectiveness.

7.32 Over three-quarters (n=412) of our survey sample reported that they intervened with a person at risk following their ASIST training. The remaining quarter (n=122) did not intervene.

7.33 A comparison by employee group shows that:

• More than one-third (35%) of interveners worked in the voluntary sector.
• More than a quarter (28%) of interveners were local government employees.
• Less than a fifth (19%) of interveners were NHS staff.
• The remaining (18%) belong to other employee groups.

7.34 These figures closely match the characteristics of the entire sample, and therefore interveners cannot be distinguished from non-interveners by employee group.

7.35 In order to outline the profiles of the intervener and the non-intervener, we compared survey participants who have intervened following training, with participants who have not, on several key features. (See Table 7.1.)

Table 7.1: Key characteristics of people who have intervened following ASIST ('intervener'), people who haven't intervened ('non-intervener'), and the overall sample

7.36 These findings indicate that individuals who put their learned skills into practice following training were most typically:

• female
• professional caregivers
• had previous experience of intervening with a person at risk of suicide.

7.37 The first two bullet points above (female, professional caregivers) also represent key characteristics of ASIST participants who have not put their skills into practice, and indeed are key characteristics of ASIST participants in general. However, the non-interveners group contained a smaller percentage of female professional caregivers compared to the interveners group.

7.38 The third bullet point is of most interest, as it seems to highlight a key feature differentiating interveners from non-interveners. Over two-thirds of respondents who have intervened following training have had previous experience of suicide intervention. In comparison, over two-thirds of respondents who have not intervened following training did not have previous experience of suicide intervention. A possible explanation is that people who are more exposed to high risk individual have more opportunities to practise ASIST skills once they've attended training. Another possible explanation is that people who have intervened before have more confidence to intervene again.

7.39 In order to find out whether interveners and non-interveners differed in terms of their perceived levels of suicide intervention confidence, knowledge and skills we compared survey participants' scores (on a scale of 1 (very low) to 5 (very high)) before and after training. (See Table 7.2.)

Table 7.2: Perceived levels of suicide intervention confidence, knowledge and skills before training and at the time of the survey among interveners and non-interveners

* Scale: 1=very low, 2=low, 3=moderate, 4=high, 5=very high

7.40 As can be seen in Table 7.2, interveners reported higher levels of confidence, knowledge and skills both before training and at follow-up, than non-interveners. All these differences were found to be statistically significant at the 0.01 confidence level using the t-test for independent samples.

7.41 These findings suggest that having confidence, knowledge and skills in relation to suicide intervention plays a significant role in the likelihood that an individual will intervene with a person at risk. Individuals who have intervened following ASIST reported higher levels of confidence, knowledge and skills, both before and after training, than individuals who have not intervened following ASIST.

Reasons for not intervening

7.42 As stated earlier in this section, less than a quarter (22.8%) of participants in our survey sample had not intervened with a person at risk of suicide following their ASIST training. The vast majority of them (96%) reported the reason for not intervening to be that a situation had not arisen. The remaining 4% gave other reasons for not intervening, such as ASIST was not perceived to be appropriate, they felt their skills were too rusty, or that they had assisted others to intervene.

Intervening with different groups

7.43 One of the factors that might influence whether and how a trainee would apply their learned ASIST skills into practice, is the type of relationship they have with the person at risk. We asked survey respondents to indicate whether they had ever intervened with a client, personal contact or colleague:

• More than three-quarters (77%) had intervened with a client.
• More than one-third (38%) had intervened with a personal contact.
• Few (13%) had intervened with a colleague.

7.44 It appears that the majority of ASIST interventions occur in a professional setting, between a trained staff member (or volunteer) and their client. Slightly over a third of respondents reported intervening with a personal contact, and even less with a colleague. This is likely to result from having less opportunity to make contact with suicidal individuals in a non-client capacity, but might also reflect the challenges involved in carrying out an intervention with a personal contact or a colleague at work. Moreover, 13% of survey respondents felt ASIST was not always appropriate for use with personal contacts. Here are some illustrative quotes from our participant interviews:

There is more tension when it is people you know, if you've got more invested in a relationship with them and you care more. With people in a professional capacity - you might not see them again.

With a personal contact it's more difficult - you want to protect them and don't want to believe that anyone close to you can see no way out. You have to not take it as a personal affront and manage your emotions better. I felt confident I could handle it well though, because of training. When it's a professional contact you can maintain your distance.

7.45 We also asked survey respondents to indicate the age group of the person they have intervened with most recently:

• Less than one-third (29%) had intervened with someone aged between 16-25 years.
• Almost two-fifths (39%) had intervened with someone aged between 26-45 years.
• Less than one-fifth (19%) had intervened with someone aged between 46-64 years.
• The remaining 13% had intervened with under-16s, over-65s, or could not recall the age of the person they had intervened with.

Female vs. male interveners

7.46 The following analyses examine possible differences in the extent and pattern of suicide intervention between female and male participants in our survey.

Intervening before and after training

7.47 We compared the percentage of females and males who had intervened with a person at risk of suicide before and after their ASIST training in order to check whether gender is of importance to suicide intervention. (See Figure 7.1.)

Figure 7.1: Percentage of males and females who had intervened with a person at risk of suicide before and after ASIST training

7.48 The difference between the percentage of females and males who had intervened with a person at risk before ASIST was only 3%. This difference was not found to be statistically significant using the chi-square test for independence. Following training, the percentage of interveners had grown for both males and females. However, the gap between males and females has increased to 11% more female interveners following training. This difference was found to be statistically significant at the 0.01 confidence level using the chi-square test for independence.

7.49 Our analysis suggests that while ASIST training increases the likelihood of intervention for both males and females, the increase is significantly higher for females.

Intervening with different client groups

7.50 Additionally, we wanted to examine whether there were any differences between male and female interveners as a function of:

• the gender of the person at risk (male / female)
• the type of relationship with the person at risk (client / personal contact / colleague).

7.51 The findings from our survey of ASIST participants are summarised in Table 7.3.

Table 7.3: Percentage of male and female participants who have intervened with male and female clients, personal contacts and colleagues following ASIST training

7.52 Two main findings come out of the table:

• Consistently, over the three relationship groups, males tend to intervene slightly more with males and females tend to intervene slightly more with females. However, these differences are quite small.
• Overall, it seems that males intervene with clients slightly more than females do, and females tend to intervene with personal contacts and colleagues slightly more than do males do. Again, these differences are quite small.

7.53 Here are some illustrative quotes from female interviewees, talking about the challenges of intervening with males:

I would be slightly more concerned about men's responses in the sense that they might be less likely to admit to feeling suicidal and that it would take more work to get there. (Female interviewee)

Not much of a difference between intervening with men and women - only thing is that I find it harder to watch a man cry, which is something that often happens once you 'ask the question.' (Female interviewee)

Perhaps I would find it more difficult to ask men the question, because of how they might respond. (Female interviewee)