ANNEX 1: GUIDE TO STATISTICAL RELIABILITY
The variation between the sample results and the "true" values (the findings that would have been obtained if everyone had completed the questionnaire) can be predicted from a knowledge of the sample sizes on which the results are based, and on the number of times that a particular answer is given. The confidence with which we can make this prediction is usually chosen to be 95%, that is, the chances are 95 in 100 that the "true" values will fall within a specified range.
The table below illustrates the required ranges for different sample sizes and percentage results at the "95% confidence interval":
Approximate sampling tolerances applicable to percentages at or near to these levels | Actual Sample Size | 10% or 90%
+ | 30% or 70%
+ | 50%
+ |
|---|
Overall | 500 | 2.3 | 3.5 | 3.8* |
|---|
*For example, if 50% of all respondents were to give a particular answer, the chances are 19 in 20 that the "true" value will fall within the range of +3.8 percentage points from the sample results.
Comparing percentages between sub-groups and overall total
When results are compared between separate groups within a sample, different results may be obtained. The difference may be "real", or it may occur by chance (because not everyone completed a questionnaire). To test if the difference is a real one - i.e. if it is "statistically significant" - we again have to know the size of the samples, the percentages giving a certain answer and the degree of confidence chosen. If we assume "95% confidence interval", the difference between two sample results must be greater than the values given in the table below:
| Actual Sample Size | 10% or 90%
+ | 30% or 70%
+ | 50%
+ |
|---|
Overall (500) vs: |
|---|
Sub-groups of: | 50 | 8.6 | 13.1 | 14.3 |
|---|
| 100 | 6.2 | 9.5 | 10.3 |
|---|
| 200 | 4.6 | 7.0 | 7.6 |
|---|
| 300 | 3.9 | 5.9 | 6.4 |
|---|
| 400 | 3.5 | 5.3 | 5.8* |
|---|
*For example, if 50% of the total sample (500) give a particular answer, and 55% of respondents in a sub-group of 400 give the same answer, there is not a statistically significant difference between the responses of the two groups.
Looking at the fifth column of the above table shows that there needs to be a difference of +5.8 percentage points between the two results in order for the difference to be statistically significant.
Therefore, if 56% of the latter group give the same answer, then this is a statistically significant difference (since there is a 6 point difference between the two).
Comparing percentages between sub-groups
The following table indicates differences required for significant comparisons between sub-groups.
Approximate sampling tolerances applicable to percentages at or near to these levels | 10% or 90%
+ | 30% or 70%
+ | 50%
+ |
|---|
Sub-group of 50 vs: |
|---|
100 | 10.1 | 15.4 | 16.8 |
|---|
200 | 9.2 | 14.0 | 15.3 |
|---|
300 | 8.9 | 13.5 | 14.8 |
|---|
400 | 8.7 | 13.3 | 14.5 |
|---|
Sub-group of 100 vs: |
|---|
200 | 7.0 | 10.7 | 11.6 |
|---|
300 | 6.6 | 10.0 | 10.9 |
|---|
400 | 6.3 | 9.7 | 10.6 |
|---|