Chapter 3: Research Methods
3.1 This chapter describes the research methods used in the evaluation.
3.2 The evaluation comprised the following four main components: (1) the statistical analysis of data relating to case trajectories ( i.e. the progress of cases) in the High Court for 12 month periods prior to and following the introduction of the reforms; (2) interviews with criminal justice personnel in order to obtain their views on the operation of the High Court prior to and subsequent to the reforms; (3) questionnaire surveys of jurors and professional witnesses in order to establish their perceptions of their treatment by the High Court prior to and subsequent to the reforms; and (4) a programme of court observation both prior to and following the reforms.
3.3 The four main components of the research are described in more detail below.
3.4 In addition to these four main components, the researchers also undertook: (5) an analysis of data extracted from High Court sitting lists between January 2002 and April 2004; and (6) an analysis of sentencing statistics in order to assess the impact on High Court business of the increase in sheriffs' sentencing powers from a maximum of three to five years' imprisonment.
3.5 Neither of components (5) and (6) of the research require any detailed methodological explanation in this chapter. The analysis of sentencing statistics is presented in chapter 10 of the report. The analysis of data from High Court sitting lists was reported in the first interim report for this project and is not repeated here. Its main purpose was to inform the early stages of the research (and it was extremely useful in this context) but the data collected on case trajectories in the 12 months leading up to the reforms (our own 'pre-reform' sample) is more recent and is of more use as a comparator, as it was collected in the same way as the case trajectory data from the 12 months immediately following the reforms (our 'post-reform' sample). It suffices to say that there are no notable discrepancies between the sitting list data, which stems from the period January 2002 to April 2004, and our own pre-reform dataset, which covers the period March 2004 to April 2005, so nothing is lost by not reporting on the former here.
Collection and Analysis of Statistical Data
3.6 The aim of this aspect of the research was to compare the case trajectories of two samples of High Court cases: one collected in the 12 months immediately prior to the reforms, i.e. 1 April 2004 to 31 March 2005 (the 'pre-reform' sample) and one collected in the 12 months immediately following the reforms, i.e. 1 April 2005 to 31 March 2006 (the 'post-reform' sample). These two time periods were determined in the Research Specification for this project.
3.7 It was decided that the analysis should focus only on cases that had concluded within the relevant sample period, 24 so that the eventual outcome of the case (whether a guilty plea, a finding of not guilty following trial, and so on) was known. Ideally, the analysis would not have been undertaken until all of the cases had concluded, but within the time constraints of the project this was not possible for the post-reform sample. In order to compare like with like, the same rule was applied to both samples. In the event, 64.2% of cases indicted in the post-reform sample and 69.4% in the pre-reform sample had concluded by 31 March of the relevant year. The fact that a similar proportion of cases had concluded across the two samples means that valid comparisons can be made.
3.8 As promised in the Research Specification, case trajectory data for both the pre-reform sample and post-reform sample was made available to the researchers from the High Court Case Management System ( HCCMS), which went on-line in April 2004. This process was not without its difficulties. Some initial technical problems in identifying the relevant cases to be extracted from HCCMS and ensuring that no concluded cases were left out of the extract meant that the monthly extracts envisaged in the Research Specification did not prove possible and the data was actually delivered in a single batch (for the pre-reform sample) and three separate batches (for the post-reform sample). This has not affected the quality of the analysis.
3.9 The data extracted from HCCMS was supplied to the researchers in the form of Excel spreadsheets. These were transferred onto a customised SPSS database 25 in order to enable statistical analyses to be undertaken. Some of the data could be copied across directly into SPSS from Excel but the data relating to the case trajectories themselves was transferred in a two stage process. It was first transferred onto paper records and only when the research team were satisfied that they understood the trajectory of a particular case was it manually inputted into SPSS. The vast majority of trajectories were relatively straightforward to map, but for a very small minority of cases (mostly involving multiple accused), it was necessary to ask for clarification from the Scottish Court Service. Two versions of each database were prepared: one based on individual indictments (which often name several accused); the other contained a separate entry for each individual accused.
3.10 It should be noted at this point that the data is subject to a number of limitations. First, there was some information that, while it would have been useful to have, was not recorded in HCCMS at the time the research was being undertaken. One example is the reasons for any continued preliminary hearings. These are minuted manually (this minute is generally attached electronically to the HCCMS record), but the time constraints of the project meant that it was simply not possible to examine these minutes (and there was some evidence that they vary greatly in terms of the detail provided, so it might not have been a useful exercise anyway). Another example is the size of any sentence discount that was given. It was originally hoped that the research might assess the extent of sentence discounting for guilty pleas. The Research Specification stated that this should be done if it proved possible, but it did not prove to be so, as only the eventual sentence was recorded in HCCMS and not the size of any 'discount' given. 26
3.11 Second, the comprehensiveness and quality of the information contained in HCCMS is entirely reliant on manual data input. As in any system of this nature, there is always potential for error and our experience was no exception to this (although where we received information that was obviously wrong or did not make sense, we attempted to clarify the situation by asking for additional information from Scottish Court Service, as noted above). We also found that recording practices differed depending on the individual clerk in question with, for example, some clerks always recording reasons for adjourned trial diets, whereas others did not.
3.12 All of this means that the SPSS databases did not present quite as complete a picture as we would ideally have wished. It did prove possible to fill in at least some of the gaps, however, with the help of our project manager, Ian Clark, who spent some considerable time interrogating HCCMS and the electronic minutes of High Court cases. As a result of this exercise, we obtained more reliable figures on, among other things, the number of witnesses listed on the indictment and who gave evidence and the number and length of extensions to the 110-day and 140-day time bars. We were not, however, able to obtain data on the number of witnesses that were actually cited in each case, or on the number that were added to the indictment by section 67 notice, as this information is not systematically recorded in HCCMS or minuted in the court record and it was not possible within the timescale of the project to obtain it from Crown Office.
3.13 The findings from the statistical analysis are reported in chapter 4.
Interviews with Criminal Justice Personnel
3.14 Interviews with criminal justice personnel took place over the course of the project as follows.
3.15 Prior to the implementation of the reforms: four High Court judges; two Advocate Deputes; six defence practitioners; one High Court manager; 27 two expert witnesses; and one social worker.
3.16 Following the implementation of the reforms: six High Court judges; two Advocate Deputes; seven defence practitioners; four representatives of victim support organisations; two expert witnesses; four High Court managers; three representatives of the Scottish Legal Aid Board ( SLAB); and one procurator fiscal. Some of these individuals were interviewed on more than one occasion.
3.17 The interviews were usually undertaken by two members of the research team. They were guided by a discussion guide, a sample copy of which is contained in Appendix 1. The discussion guides used varied slightly according to the role of the interviewee. The interviews were relatively informal in nature and the participants were not restricted to the topics listed on the discussion guide, being encouraged to give their views on all aspects of the reforms.
3.18 In addition to these formal interviews, the research has been informed by several informal discussions with parties involved in the operation of the High Court. These often took place whilst the researchers were attending the High Court to observe court proceedings, and included High Court clerks, Advocate Deputes, defence practitioners, police officers and other support staff. The researchers also attended the High Court Consultative Forum held in Edinburgh on 21 June 2006 where several other useful informal discussions took place.
3.19 The results of the interviews of criminal justice personnel are reported primarily in chapters 5, 6 and 7. Information gathered during the interviews also serves to supplement other chapters of the report.
Questionnaire Survey of Professional Witnesses and Jurors
3.20 The third aspect of the research was a questionnaire survey of professional witnesses and jurors undertaken in order to establish their perceptions of their treatment by the High Court prior to and subsequent to the reforms.
3.21 The professional witness survey involved sending self-completion questionnaires to a sample of professional witnesses both before and after the reforms. As agreed, the administration of the questionnaire survey was undertaken by the Scottish Executive, with the analysis of results being undertaken by the researchers. The questionnaire relating to the pre-reform period (subsequently referred to as the 'pre-reform' survey) was posted out in January and February 2005. Of the 390 sent out, 216 usable questionnaires were returned, a response rate of 55%. The questionnaire relating to the post-reform period (subsequently referred to as the 'post-reform' survey) was sent out in December 2005. Of the 200 sent out, 91 usable questionnaires were returned, a response rate of 46%. Chapter 8 of this report presents a comparative analysis of the results. A copy of the questionnaire used can be found in Appendix 2.
3.22 In the very early stages of the research, the idea that a questionnaire might be sent to victims and witnesses other than professional witnesses was discussed but it was decided not to proceed because of the sensitivities involved in High Court cases, which can involve, for example, murder and rape charges. It was considered important to try and obtain the views of this key user group, however, so it was decided that the researchers would interview representatives of victim support organisations with this end in mind. 28
3.23 A separate questionnaire survey of jurors was also undertaken which, like the professional witness survey, was administered by the Scottish Executive but analysed by the researchers. 29 A total of 420 self-completion questionnaires were sent out to jurors at the beginning of January 2006 ( i.e. after the implementation of the reforms). Of these, 193 usable responses were received, a response rate of 46%. The sample included both jurors who actually served on a jury and those who were cited but not selected.
3.24 It was not necessary to send out juror questionnaires prior to the reforms. A survey of jurors was undertaken as part of Lord Bonomy's review (reported in Samuel and Clark, 2003) and this has been used as the yardstick against which the post-reform survey is compared. In this pre-reform survey, a total of 650 self-completion questionnaires were sent out in March 2003. Of these, 320 usable responses were received, a response rate of 49%. As was the case for the post-reform survey, the sample included both jurors who served on a jury and those who were cited but not selected.
3.25 Chapter 9 of this report presents a comparative analysis of the results of the juror survey. A copy of the questionnaire used in the post-reform juror survey can be found in Appendix 3. The questionnaire used for the pre-reform juror survey (undertaken as part of Lord Bonomy's review) was almost identical 30 and can be found in Annex F of Samuel and Clark (2003).
Observation of Court Proceedings
3.26 A programme of court observation both prior to and following the reforms was undertaken. This comprised the following observations of High Court proceedings:
3.27 Prior to the implementation of the reforms: seven days at Aberdeen; three days at Glasgow; two days at Edinburgh; two days at Peterhead; one day at Inverness.
3.28 Following the implementation of the reforms: six days at Aberdeen; four days at Edinburgh; two days at Glasgow. Observation following the reforms has focussed primarily (although not exclusively) on preliminary hearings. In designing the programme of court observation, an attempt was made to cover not just all three of the courts involved in holding preliminary hearings, but also to observe as wide a range of individual judges as possible. Thus in Aberdeen, for example, each of the six observations involved a different judge.
3.29 Detailed notes of all of these observations were made. The results of the observations are reported primarily in chapters 5, 6 and 7, alongside the interviews conducted with criminal justice personnel. Information gathered during the observations also serves to supplement other chapters of the report.