Scottish Government

Main Findings

• SSAScot's activities support the organisation's aims and objectives.
• SSAScot deliver a range of quality services effectively and efficiently.
• All services provided by SSAScot received high satisfaction ratings from all respondents, regardless of level of usage.
• SSAScot staff were regarded highly by all respondents.
• SSAScot is a cost effective organisation, offering both value for money and added value to its members and funders.
• Types of services required by members vary according to the type of customer.
• Non use of services can be explained in terms of the fact that some services are not required by specific customers rather than in terms of dissatisfaction with the quality of services offered.
• In the future, at the same time as SSAScot looks to provide specific services to members, the organisation must also be aware of the needs of key stakeholders.

Background

Blake Stevenson Ltd was commissioned by the Scottish Executive (Analytical Services and Skills for Life and Work Division) to evaluate the effectiveness of SSAScot in achieving its business aims and objectives, and to consider if SSAScot is providing customer satisfaction.

The evaluation has taken place at a time of change and challenge for SSAScot. It is a relatively new organisation, coming into being at the end of 2002. SSAScot and its member organisations have been experiencing a period of significant development over the past eighteen months with new SSCs being licensed, aspirant SSCs developing bids, and former NTOs or other skills bodies which will not be SSCs, considering their position in, and contribution to, the skills environment.

Methodology

This evaluation is the result of documentation analysis and field research involving a series of telephone and face-to-face interviews with key stakeholders and member organisations. A sample of 27 interviews were undertaken.

The interview schedule gathered quantitative and qualitative information on the services provided by SSAScot.

The interviews were analysed manually to identify the extent to which services support the aims and objectives of SSAScot; to assess the success and cost effectiveness of completed activities; to assess levels of customer satisfaction; to identify the views of key customer groups; and to arrive at conclusions and recommendations.

Following the analysis of telephone and face-to-face interviews, all key stakeholders and member organisations who had been interviewed were re-contacted and asked for their participation in a focus group discussion which would explore in greater detail some of the issues raised in the earlier interviews. Five people participated; four Sector Skills Bodies and one stakeholder. The discussion focused on barriers to accessing services; potential developments; and the future of SSAScot.

Findings: Extent to which SSAScot activities support their aims and objectives

SSAScot states its vision to be 'empowering Sector Skills Bodies in Scotland'. Its mission is to act as 'the collective voice of Sector Skills Bodies in Scotland'. In order to deliver the above, SSAScot's key roles are as follows:

• To facilitate Sector Skills Bodies collectively;
• To provide guidance and advice to Sector Skills Bodies and key stakeholders;
• To provide a forum for the exchange of information on skills issues;
• To promote awareness and understanding of Sector Skills Bodies.

Members of SSAScot who were interviewed as part of the evaluation (both Sector Skills Bodies and key stakeholders) understand and agree that the main roles of SSAScot are to facilitate, to provide information on relevant issues, and to promote Sector Skills Bodies.

All members interviewed agreed that SSAScot perform these roles efficiently and effectively.

Findings: Levels of satisfaction with SSAScot services

In order to identify those services and activities that provide the most benefit to members, the following criteria were used:

• any services used by 50% or more of the respondents which gained an effectiveness rating of above 60%;
• services used by 50% or less of respondents but with an effectiveness rating of 80% or more.

In addition we also analysed responses to the questions, 'Which SSAScot service has had the greatest impact on your organisation in 2003/2004?' and 'What SSAScot services would you use in the future?'

Levels of customer satisfaction with the activities and services of SSAScot were high. Members believe that SSAScot delivers a range of quality services effectively and efficiently.

Figure 1: Services used by 50% or more of all respondents, with an effectiveness rating of 60% or more.

Figure 1 indicates that the most frequently used services all have high satisfaction levels associated with their use. On this basis we can say that it is these services, based around information and advice provision and facilitation, that provide the most benefit to members.

Services Available When Needed

However, the concept of benefit should be understood in the context of need as well as in terms of frequency of use. For example, a member organisation may not need one service as frequently as it requires another. If the less frequent need is satisfied, the benefit may be as great as any other.

For this reason we have also included in our analysis services that are used by less than half of the respondents but which also have a high satisfaction rating.

Figure 2: Services used by 50% or less of all respondents, with an effectiveness rating of 80% or more

Services which have Greatest Impact

Respondents were asked to identify the services that had the greatest impact on their organisation in 2003/2004. They were most likely to mention the e-intelligence weekly email bulletin, the Annual Conference, and the services connected to the Modern Apprenticeship scheme. Respondents placed great value on the networking opportunities offered by SSAScot, the provision of information, and the organisation and co-ordination of events.

Services Respondents want SSAScot to Provide in the Future

The six services members most want to see retained in the future were:

• e-intelligence weekly email bulletin
• Annual Conference
• Communication and dissemination of up to date information on the Scottish Education and Training system
• SSAScot website
• SSAScot supporting the engagement between sector skills, education and training bodies
• Co-ordinating attendance of Sector Skills Bodies at committee meetings

SSAScot Staff Team

The majority of those interviewed had high praise for the SSAScot staff team and the current staff complement should be considered one of the organisation's main assets.

Many commented on the professionalism of the team and all believed each member of staff to be extremely efficient, knowledgeable, helpful and friendly.

Cost Effectiveness of Services - Value for Money

Overall, members stated that using the services provided by SSAScot added value to their work. In the words of one member, their organisation "...couldn't operate as cost effectively without SSAScot...."

The cost effectiveness of SSAScot was assessed against the criteria of:

• SSAScot Leverage,
• SSAScot Added value,
• SSAScot Substitution,
• SSAScot Displacement
• SSAScot Overspill

Overall it is our assessment that SSAScot is cost effective, and provides both value for money and added value to its members and funders.

Explaining Non Use of Services

Non use of services can be explained in terms of the fact that some services are not required by specific customers rather than in terms of dissatisfaction with the quality of services offered.

For focus group participants, the main reasons for not using services were:

• lack of clarity about the specific activities associated with a service;
• feeling that their organisation could provide this service 'in-house'; and
• that the service was not directly relevant to them.

Matching Services to Customer Needs

When considering customer response, it is important to recognise the varying levels of need Sector Skills Bodies have. Through this study, we identified three particular groups of Sector Skills Bodies that require specific levels of support:

1. Established Sector Skills Councils which have developed effective relationships with relevant bodies and stakeholders and are of a sufficient size to allow for project work to be undertaken in-house.

This type of organisation is most likely to be looking for:

• electronic news updates on skills and training issues; and
• opportunities for networking with other Sector Skills Bodies and key stakeholders including conferences, events, and workshops.

2. Very recently licensed SSCs, or former NTOs which have not received a Sector Skills Council licence. These bodies tend to have a small number of staff in many cases and have not yet managed to develop contacts with key stakeholders or other Sector Skills Bodies.

This type of organisation would be looking for a range of services including:

• provision of information and advice on skills issues;
• a channel for enquiries between SSCs and key stakeholders;
• networking opportunities with other Sector Skills Bodies and key stakeholders including conferences, events and workshops;
• organising and hosting of events on their behalf; and
• assistance in undertaking specific project work.

3. Sector Skills Councils or bodies which are based outside of Scotland without sufficient knowledge of the Scottish skills and training system.

This type of organisation would be looking for:

• information and advice on the Scottish skills and training system;
• a channel for enquiries between SSCs and key stakeholders; and
• networking opportunities with other Sector Skills Bodies and key stakeholders including conferences, events and workshops.

Rather than providing a generic range of services to all members, SSAScot should consider the type and individual needs of the particular organisation and tailor specific services to meet these needs

Findings: Developing services for the future

Meeting the Needs of a Diverse Customer Base

SSAScot faces the challenge of meeting the needs of their customer base. Whilst there is a general concern from members that the needs of specific groups should not be overlooked, the organisation must also be aware of the needs of key stakeholders.

Identifying the 'Core' Services Delivered by SSAScot

A significant challenge facing SSAScot is to clearly distinguish between its own services and those that might be delivered by the Sector Skills Development Agency (SSDA).

Members were able to identify what they saw as the 'core' functions of SSAScot:

• acting as an independent Scottish voice for members;
• facilitation;
• provision of information (specifically the e-intelligence weekly email bulletin);
• provision of support services (organising conference and seminars); and
• networking.

The Potential to Develop New Services

A number of positive suggestions for services and activities that SSAScot could provide in the future were given. These were:

• streamlined training forums;
• administrative support;
• research support; and
• providing a guide to innovative practice.

Conclusion and Recommendations

This evaluation of the services and activities provided by SSAScot to its members demonstrates high levels of customer satisfaction and that the services and activities provided support the aims and objectives of SSAScot.

Members and stakeholders rated highly the effectiveness and efficiency of the services and activities offered by SSAScot. Very few could think of ways that these could be improved.

A number of recommendations are made for the future development of services and activities:

1. SSAScot should clearly define its role vis a vis the SSDA. There is some confusion and conjecture as to how the two organisations will relate to one another in the future. Members were aware that a draft agreement has been drawn up between SSAScot and the SSDA but are unsure of the contents or the implications.

2. If membership fees are to be retained SSAScot should draw up a set of clear criteria on which to base them. Due to future developments and the partnership agreed between SSAScot and SSDA, it is essential that SSAScot clarify whether membership fees will have to be paid by those outwith the Skills for Business Network.

3. SSAScot may wish to consider streamlining or changing the range of services offered. This evaluation has highlighted the services most likely to be used and those least likely to be used. SSAScot may wish to consider how it makes the most effective use of the resources it has by adapting the type and range of services offered. Constant monitoring of service take up will be important to spot changes in demand.

4. SSAScot should consider probing into why some services are less likely to be used. It may be that members do not require some of the services available, it may be that they can perform these services in house, or it may be that members are unaware that the services are available. When considering any review of services, SSAScot needs to clearly distinguish between these different categories of members.

5. Moving forward, SSAScot should consider how effectively it promotes its services and activities. Some respondents were unclear about the range of services offered. For example, some were unaware that SSAScot could undertake specific project work for members.

6. To ensure that services and activities match the requirements of members, SSAScot should continue with its annual customer satisfaction survey. SSAScot and its members exist within a dynamic environment and service provision should reflect this. SSAScot have undertaken annual customer satisfaction surveys in recognition of this

SSAScot needs to establish a mechanism for feeding back the findings of any survey to key decision makers and ensuring that actions are taken as a consequence.

7. SSAScot should build on its main asset, namely its staff complement. The staff team is an important resource for SSAScot and will need adequate support to continue to function effectively.