EXECUTIVE SUMMARY
Background
1. This report outlines the key findings of the Scottish Executive 1 website evaluation study. The Scottish Executive has undertaken a number of changes to the website since its last evaluation in 2004, including the introduction of more multi-media content and changes in the topic list structure and labels. These changes were introduced to improve the content, design and usability of the site, and the focus of the study was to assess the effectiveness of these recent developments.
2. The key aims of the research were to evaluate the use, content, design and accessibility of the improved website, to inform the continuous process of the site development, and to evaluate the 'Topics' structure used within the website.
3. The results presented in this report are based on successive stages of research including desk research, a consumer omnibus survey, an online self completion survey, qualitative website observation and interviews and a card sorting exercise conducted during January through to May 2007. Findings from the research are reported thematically; however, the initial desk research stage is reported on separately, as are the findings from the card sort exercise.
Users profile and use of the website
4. Amongst those who undertook the online self completion survey, the general public (23%) and students (20%) collectively account for two fifths of all respondents visiting the website. The website is also popular with visitors from Local Authorities (13%) and Academic/Educational Institutions (10%). Well over a third of all users visiting the site during the survey visited the site for the first time (37%). Only small proportions of users tended to use the site frequently using the site daily (5%), most days (7%) or up to a couple of days per week (15%). A further third of users used the site less frequently; using the site one to three times a month or less often. A majority of non-users, interviewed during the consumer omnibus survey, stated that they had no interest in the Scottish Executive website and would require a specific reason to visit the site (60%).
Overall satisfaction
5. The majority of visitors to the Scottish Executive website were either very or fairly satisfied with their experience of the website (82%). Three quarters of all respondents rated the section of the website they had visited as either very or quite good (75%).
Multimedia
6. A third of all users said that they made use of the online newsletters which appear to be by far the most popular interactive feature (32%). Online consultation forms have been made use of by around one in ten users. More 'multimedia' style features such as webcasts, audio and video podcasts and voice enabled pages are only used by a small proportion of website users.
7. When respondents were asked to find interactive, 'multimedia' style features during the website evaluation, there was some confusion as to where to find multimedia features on the website. This would support findings from the initial survey that multimedia features were being used by only a small proportion of visitors to the website.
Disability access
8. A small proportion of respondents identified themselves as having some form of disability (5%). Despite the small percentage indicated, this represents a total of 651 respondents.
9. The most popular aids appear to be voice software, text to speech software or the use of large fonts or enlarged screens. The use of pre-specified web links was cited as a way of circumventing the problem of identifying the location of specific information.
10. Most respondents commented that podcasts in either video or audio formats would prove useful for those with hearing or visual disabilities. Those with auditory disabilities might benefit from subtitling on the videocasts.
Topic lists
11. There were a number of topic areas which respondents stated that they could not easily find or eventually found but within an unexpected area. In most cases, respondents eventually found the topic information they were looking for, but identified it via the onsite search rather than through the topic list menu.
12. There were some issues with the use of acronyms, abbreviations and technical terminology. In particular, respondents cited difficulties understanding what a 'podcast' was, saying that the use of words such as 'video' or 'audio' would prove more helpful in finding interactive features.
Topic structure
13. The research strongly indicates that issues with the topic structure are principally usability problems and not caused by gaps in content. Specifically, the problems are related to structure and labelling of topics and their groups.
14. The main issue identified is a high extent of perceived overlap between topic groups. Many instances of this were observed and this finding is consistent with our experience of other government websites.
15. A key recommendation is therefore to introduce an A-Z of topics which would supplement (but not replace) the topic structure. Where topics are hard to locate as they could credibly be located in more than one topic group, the introduction of an A-Z would allow users to access these more easily. This idea was suggested in the qualitative stage of research and received widespread approval from respondents during discussion at the card sorting stage.
16. In addition to the overlap, a number of other topic structure issues were identified:
- A wide range of topics appeared not to have a 'home' or at least not a sufficiently visible one. Examples include infrastructure and enterprise.
- Other topics were perceived to be too broad to fit into any single topic group. Examples of these included Architecture, Equality, Promoting Scotland and the Voluntary Sector, for which a new dedicated topic group is recommended.
- Some group labels were considered "nebulous". Alternative labels are provided in the report.
- The overall number of topic groups did prove to be a challenge for respondents and it is recommended that efforts are made to control this when the topic structure evolves again.