Scottish Government

ANNEX F: STATISTICAL SIGNIFICANCE AND RELIABILITY

Statistical significance

F.1 The formula used for calculating significant differences between sub-groups is as follows:

the standard deviations for two sub-groups are calculated as SD1 and SD2

1) Calculate an "overall" or "pooled" SD for the two groups together. This is very close to the weighted average; weighted by the relative sizes of the sub-groups in the sample.

2) Use this pooled measure to calculate the Standard Error of the Difference ( SED) between the sub-group means, i.e.:

3) Divide the difference between the sub-groups scores that you observe, by the SED. If the size of this result (technically referred to as the "t-score") is greater than 1.96 ( i.e. either less than -1.96 or greater than +1.96), then the difference is statistically significant at the 95% confidence level. In other words, there is sufficient evidence that scores in the underlying population are different for the two sub-groups. Thus:

Statistical Reliability

F.2 The respondents to the questionnaire are only a sample of the total 'population'. We cannot therefore be certain that the figures obtained are exactly those we would have if everybody had been interviewed (the 'true' values). However, we can predict the variation between the sample results and the 'true' values from a knowledge of the size of the samples on which the results are based and the number of times that a particular answer is given.

F.3 The confidence with which we can make this prediction is usually chosen to be 95% - that is, the chances are 19 in 20 that the 'true' value will fall within a specified range. The table below illustrates the predicted ranges for different sample sizes and percentages results at the '95% confidence interval', based on a random sample.

Table F.1: Predicted ranges for different sample sizes at the 95% confidence interval

F.4 For example, on a question where 50% of the people in a sample of 1,177 respond with a particular answer, the chances are 95 in 100 that this result would not vary by more than three percentage points, plus or minus from a complete coverage of the entire population using the same procedures. However, while it is true to conclude that the "actual" result (95 times out of 100) lies anywhere between 47% and 53%, it is proportionately more likely to be closer to the centre of this band ( i.e. at 50%).

F.5 Tolerances are also involved in the comparison of results from different parts of a sample. A difference, in other words, must be of at least a certain size to be considered statistically significant. The following table is a guide to the sampling tolerances applicable to comparisons.

Table F.2: Sampling tolerances

Table F.3: Demographic sub-group comparisons