Scottish Government

Appendix 2: Markets Model Sensitivity Analyses

Introduction

In this appendix we describe the sensitivity analyses that were carried out relating to the sizing the drugs markets part of the study. The main objective of that part of the study was to provide estimates of the size of the illicit drugs market in Scotland, both in terms of quantity and cost. Estimates have been derived by combining information from various sources, all of which are based on estimates. For example the size of the cannabis market is derived from estimates such as the estimated number of people in Scotland who use cannabis (from the SCVS and SALSUS), the estimated frequency of cannabis use (also from those surveys) and the estimated amount a cannabis user uses per day (from more anecdotal data).

There are two issues that need to be considered. One is that the estimates are often derived from samples, and as such there will be some amount of statistical error attached to an estimate, typically the standard error of the mean for the average values we have used as inputs to the markets model. This statistical error can then be used to derive a confidence interval, giving some kind of indication of how reliable the estimate is. The other issue is that by only using point estimates, we are ignoring the variability within the population; some heroin users use more heroin per day than others.

Some of the data sources attach a measure the uncertainty to their estimates, for example SALSUS provides confidence intervals for a number of their estimates. The SCVS does not provide confidence intervals for all of the figures in the report, but it is possible to calculate them using standard statistical techniques for deriving standard errors from samples. Therefore it is possible to provide error bounds for the estimated number of people who use particular drugs in Scotland. The national prevalence study, from which we derive the number of heroin users, also provides confidence intervals.

For surveys and the national prevalence study, the statistical exercise of providing standard errors or confidence intervals somewhat ignores the fact that the estimates are based on assumptions that either cannot be tested or are difficult to examine. For example, the surveys assume that the respondents are telling the truth about their drug use. The surveys also assume that their sampling is perfectly representative, and in particular does not under-sample from individuals who are likely to use drugs. There are many statistical assumptions attached to the capture-recapture method that the national prevalence estimate is based on, and these are described elsewhere. We could also derive error bounds for estimates, such as the estimated frequency of heroin use from DORIS, but those error bounds would ignore the issue that the DORIS sample may not be representative of all Scotland's users of heroin (for example the DORIS sample may over-represent drug injectors who may use more heroin per day than drug users who inhale the drug). Thus although standard errors / confidence intervals can be derived, other sources of error are essentially ignored.

For sources, such as the estimated amount of cannabis used by a recreational drug user in a day, there is as much uncertainty in what the possible range attached to an estimate would be as there is over the point estimate. As an example, if we assume that an average recreational cannabis users uses 1 gram of cannabis per day, we could suggest that the range of cannabis used per day is between 0.5 grams and 1.5 grams, but that range is almost as much of a guess as the estimated average amount in that, just as there is no strong evidence that the average user users 1 gram per day, there is no strong evidence that the range of amounts of daily use goes from 0.5 grams to 1.5 grams.

There is also the issue that, instead of assuming a point estimate, with or without a range, there may be an underlying distribution for each estimate. As an example, if it is taken that an average recreational user of cannabis uses 1 gram per day, and the range typically goes from 0.5 grams to 1.5 grams, then perhaps more cannabis users use around a gram per day, therefore the shape of the distribution would be more like a normal distribution (with most people using around the mean value) than a uniform distribution (with people being equally likely to use anything between 0.5 and 1.5 grams). We could spend a lot of time creating what we feel to be realistic distributions but, as with the estimated point value and estimated range, the estimate distribution is, essentially, little more than a guess.

There are several reasons for undertaking sensitivity analyses. The main reasons we have done this in this study are to

• Examine which input estimates have the most effect on the total estimate
• Speculate at some level of error bounds for the total estimates
• Look at how specific estimates affect the total estimate

In these sensitivity analyses we focus on the first two issues. We begin by looking at which input variables have the most effect on the total size of the drugs market in Scotland. We then go on to speculate about the level of error that should be attached to the total estimate.

Methods

The approach we have taken for the markets analysis is to use various distributions for the different inputs into model used to estimate the size of the illicit drugs market. There are two approaches we take to assigning these distributions. They are:

• Assume a uniform distribution, ranging from 0.5 times the point estimate to 1.5 times the point estimate
• Use a normal distribution to approximate the distribution from the source dataset (where available)

When we assume that all of the inputs or estimates range from 0.5 times the point estimate to 1.5 times the point estimate then we have to truncated these where necessary ( i.e. the number of days someone use heroin in a 90 day period cannot be more than 90).

Assuming wide ranges (from 0.5 to 1.5 times the point estimate) is helpful for examining how the different input estimates (such as the amount of cannabis used by a recreational user per day) affects the total size of the market. However, there is a potential drawback that the distributions that are truncated (such as the number of days within a 90 day period that a heroin user uses heroin) have less of a range and then, possibly, are restricted in the effect they can have on the total estimate. While this may affect the sensitivity analysis, it perhaps reflects the real situation where some distributions (such as the number of days someone use a certain drug) is truncated at the maximum number of possible days the person can use the drug ( e.g. nobody can use heroin more than 365 days per year). Truncating the distributions in such a way, and still using a uniform distribution, would have the effect of lowering the distribution for the total costs (and thus lowering each of the error bounds).

Using a distribution, such as a normal distribution to approximate the underlying error of the mean in the source datasets provides much tighter error bounds. These error bounds could be considered too narrow, particularly since they ignore any error associated with possible methodological issues ( i.e. under-representation or under-reporting in the SCVS).

We can set up the sensitivity analyses by simulating 1,000 values to approximate for each of the distributions used to describe the input variables. As suggested above, we can take three different approaches to creating these distributions. Assigning uniform distributions with the range 0.5 to 1.5 times the point estimate is relatively straightforward. This has been done for all input variables, truncated where necessary. For the approximate distributions of the source datasets, we can do this for the data from the National Prevalence Study and DORIS as we have access to those datasets. In terms of the National Prevalence Study, 1,000 simulated values of the total number of problem drug users was created as part of that study. Those simulated values do not have a standard error attached to them. For the input variables derived from DORIS, we can re-analyse the source data and derive the standard error of the mean for each of the variables used. We can approximate the distribution of the number of recreational drug users who use cocaine, cannabis etc. by treating the number of drug users ( e.g. 100,111 users of cocaine) as the product of a simple percentage of the people who use cocaine multiplied by the total population. This ignores the fact that the total number is derived from two different surveys ( SCVS and SALSUS) and the sampling frames of each study. Thus a proportion can be obtained, and the standard error of the mean of proportion can be estimated using standard statistical methods. A sample size needs to be assumed. Despite SALSUS having a much larger sample size than SCVS, we went with the SCVS sample size, however we ignored the design effects of both samples. Once the standard error of the mean of the proportion was obtained, this could be scaled up to the population level once again.

It was not possible to approximate a distribution for the more anecdotal data, such as those data derived from the IDMU studies or the price data from SCDEA.

Data

The broad uniform distributions and the distributions approximating to the error found in the source datasets are listed for the variables derived from original data are listed in Table 1 (for variables relating to problem drug use) and Table 2 (for variables relating to recreational drug use). Table 3 presents the broad and best guess distributions for remaining variables.

Table 1 Low and high values of inputs, and standard errors, used in the sensitivity analyses

Table 2 Low and high values of inputs, and standard errors, used in the sensitivity analyses

Table 3 Low and high values of inputs used in the sensitivity analyses

Results

When the 1,000 simulated values for each of the inputs are combined to provide a total estimate of the size of the drugs market, the total estimate can be correlated against each of the input values to find out which one is the most highly correlated. This can be done in various ways, including using rank correlations (where each value of the input variables are ranked and the ranked total estimate is correlated against these ranks). Partial correlations can also be calculated, in which each of the other variables is held constant to examine the particular correlation between any single input and the total estimated size of the drugs market. The approach we took was, however, simply to correlate all input variables with the total size of the drugs market. This more simple approach was taken for ease of analysis in identifying which inputs were most correlated with the total market size, and thus influenced the estimate the most. We also took the simplest approach to simulating the random values in that we took simple random samples. The sensitivity analyses may have had more power if a more sophisticated approach to simulating values was taken, such as Latin Hypercube sampling.

The broad ranges were used to examine the relationship between the input variables and the total cost of the illicit drugs market in Scotland. Table 4 lists the most correlated input variables, in order of decreasing importance.

Table 4 Input variables most correlated with the total size of the drugs market

Thus the number of problem drug users is the input variable that has the most effect on the total size of the drugs market in Scotland. The cost of heroin comes second. It is interesting to note that the estimated number of days cocaine users use cocaine within a year (essentially the frequency of cocaine use) is third. This is one of the variables that we have, perhaps, the least information on. The cost of cocaine comes next followed by the frequency of recreational cannabis use, the amount of cocaine a typical recreational user uses per day (again something we have very little solid information on) and then cost of crack cocaine and the cost of cannabis.

To give some idea of the error surrounding the total estimates, we can use the simulated distributions, in this instance the simulated distributions derived to approximate the error in the source datasets (where available) and the broad uniform distributions for the remainder.

Table 5 gives the lower 2.5 and upper 97.5 percentiles (which would correspond to the lower and upper bounds of a 95% confidence interval, along with the 25 ^th and 75 ^th percentiles (the inter-quartile range) and the median

Table 5 Summary of the simulated distribution of the total cost of problem drug use, recreational drug use and the combined total

From Table 5, the inter-quartile range of the total estimated size of the drugs market in Scotland is approximately £1,250 million to £1,580 million.

We can also explore some of the main assumptions used when creating the markets model. For example we can derive the total cost of the illicit drugs market in Scotland using the Scottish Drug Misuse Database data to calculate what proportion of problem drug users use heroin, crack cocaine, cannabis etc. We can also examine what would happen to the total cost of the drugs markets if we were to take the number of recreational drug users from the SCVS respondents who said they used drugs in the last month, rather than the last year.

Table 6 compares the total size of the problem drug use market and the total combined drugs market in Scotland when using the Scottish Drug Misuse Database data instead of the DORIS data for apportioning the problem drug use group into heroin users, crack cocaine users etc.

Table 6 Comparison of the size of the drugs market using Scottish Drug Misuse Database data and DORIS data.

Thus from Table 6, we can see that there is quite a substantial difference when using the DORIS data to derive estimates of the numbers of problem drug users who use heroin, crack cocaine, cannabis etc. The size of the drugs market attributed to problem drug use rises from £499 million to £901 million, a rise of 81% over the £499 million estimate. Focussing on the heroin market alone, the rise is 31%, while there is more than a threefold rise in the costs associated with illicit methadone, crack cocaine, cocaine and cannabis. It could be that the DORIS sample substantially over estimates the amount of secondary drug use by problem drug users. While it could be argued that the DORIS sample may be biased towards people who use, for example, more heroin as they are primarily drawn from treatment sources, there is less of an argument that their use of other drugs is much higher than heroin users who are not in treatment.

Table 7 Comparison of the size of the drugs market using SCVS 'last year' and 'last month' estimates

Thus although the total size of the drugs market decreases from £1,419 million to £1,172 million when only those who stated they used drugs in the last month are taken to be recreational drug users, the decrease in the size of the drugs market attributed to recreational drug use decreases from £518 million to £271 million.

There are of course a range of other sensitivity analyses similar to those summarised in Table 6 and Table 7 that can be done, looking at the impact the model assumptions have on the total size of the illicit drugs market. Such analyses highlight the fact that the final estimates should be treated with caution, as it is clear that the results are highly dependent on the assumptions and data sources used.