Scottish Government

6. Chapter Six Added Value of the BLA Pilot

Additionality

6.1. One of the most important questions for any programme evaluation concerns the degree of additionality achieved. Whatever the benefits or positive impact identified by programme stakeholders, if beneficiaries signal that they would have acted similarly in the absence of the programme, the estimated benefits must be readjusted.

6.2. Full additionality in the BLA programme is limited: only 5% of the survey companies who opened a BLA account felt that they would not have undertaken any training if the programme had not existed. On the other hand, only 13% said that the training they had carried out would have occurred in exactly the same manner without the programme. For the vast majority of survey companies (82%) their BLA involvement meant that training had taken place at a larger scale or earlier than would have been the case without the programme. These findings are somewhat typical for in-company training schemes in that both deadweight (where outcomes would have occurred anyway) and purely additional outcomes (would not have occurred at all) are relatively small proportions and most companies report partial additionality.

Table 6.1: What would have happened in the absence of BLA Pilots

Source: MORI/ CPC Telephone Survey (Employers)

HIE roll-out case study - full additionality

A company in the HIE roll-out areas indicated that, because of the financial consequences, they probably would not have done the training had it not been for the BLA programme. The manager admitted that he " really needed a prod" and the financial support helped; " the BLA programme just greases the mechanism".

6.3. Almost half the companies suggested that only part of their training activity would have occurred without the BLA pilots. On average this amounted to 42% of the total - close to the share of training activity funded by public money, although this does suggest that there may be some deadweight in assistance (where part of the public funding is necessary to bring about additional outcomes but not all). Over a third of companies said that training would have been delayed by an average of 13 months.

6.4. Information about additionality is available for 29 case study companies. The picture is fairly similar with the majority of companies (62%) reporting partial additionality: they would have undertaken training without the BLA programme, but not as much (38%) or only later (24%). The percentages for full additionality and deadweight are slightly higher than in the telephone survey (17% and 21% respectively).

Table 6.2: What would have happened in the absence of BLA Pilots

Source: MORI/ CPC Telephone Survey (Employers)

6.5. The evaluation compared deadweight and additionality in the BLA programme to deadweight and additionality in three similar schemes:

• The Small Firm Development Accounts ( SFDA) pilot - the SFDA pilot was implemented by the Department for Education and Skills (Df ES) in 2002 in response to the Performance and Innovation Unit report 'In Demand: Adult Skills in the 21st Century'. The aim of the pilot programme was to increase the number of small businesses actively engaged in developing their workforce, particularly at basic and intermediate levels, to improve business performance. The pilot took place in four counties in the East Midlands region and ran from April 2002 to May 2003.
• The Company Learning Account ( CLA) programme - the CLA programme is a business support mechanism which has been designed in response to concerns highlighted by the business community in Wales: that public-funded support can be, or is often perceived to be, too prescriptive, output-driven, inflexible, and bureaucratic. A pilot phase of CLAs started in September 2002 with the recruitment of companies into the programme, offering companies:
  • Support from a management development adviser;
  • Support for innovative management development activity; and,
  • Support for workforce development activity.
• Employer Training Pilots ( ETPs) - ETPs, launched in 2002, were an England-wide pilot scheme focusing on increasing demand from employers for training for their low skilled staff and creating a training supply-side driven by that demand. The pilots were designed to overcome market failures that are most acute at low skills levels. The model delivers training to Level 2 via the workplace, to suit the operational needs of the employer's business.

6.6. There is no across-the-board, systematic evidence on ETP significantly increasing employer provision of or employee engagement in training, but in both the CLA and SFDA schemes there appears to be a relatively small proportion of deadweight - as in the BLA scheme.

Table 6.3: Full additionality and deadweight by initiative

Sources: Evaluation of the Small Firm Development Account (Df ES, 2004), Evaluation of the Company Learning Account Pilot Programme (El WA, 2005), The Impact of the Employer Training Pilots on the Take-up of Training among Employers and Employees (Df ES, 2005).

6.7. The previous chapter briefly discussed the difficulties case study companies experienced in trying to identify business benefits to CPC researchers in clearly measurable terms. It is remarkable that the few case study companies that managed to give clear quantitative 'evidence' of the impact of the BLA training all commented that they would have done this training also without the BLA programme. Below are three examples from the SE geographic pilot. They are clear success stories in terms of positive business outcomes, but cannot be attributed to the BLA programme. Although this is anecdotal evidence only, it seems to suggest that companies will undertake training without public support, if it affects their bottom line.

SE geographic pilot case study

A medium-sized company introduced a new computer system for its information management and used the BLA programme to provide its staff with the necessary IT skills to work with this system. The new computer system made the company more professional: in particular, customer relations management had improved and there were more return customers - in retail this was up by 30-40%. They would also have done the computer training without the BLA programme: the company was growing and they needed the new computer system. The main benefit of their involvement in the BLA involvement was the guidance and the fact that someone else did most of the work for them.

SE geographic pilot case study

An IT company offering software solutions to its clients used the BLA programme to enable three people to use the Java platform. These new Java skills enabled the company to start working on a new (Java) software product so all Java proceeds are the tangible result of the BLA training. They would have done the training anyway, because the company needed to upgrade to the Java platform to stay up-to-date in the market place. The main benefit of their BLA involvement was that they got the training done earlier.

SE geographic pilot case study

A micro photography company had ventured into digital photography on their own as much as they could, but then hit a brick wall. The compatibility between programmes and computers was beyond them and they needed someone to come in and help them with it, so they could achieve the same kind of efficiency and speed in processing digital pictures as conventional pictures. Since the BLA training, they can deliver 1000 pictures digitally in a day. They would have done the training anyway, but the fees of the training consultant had recently gone up from _150 to _500 and the BLA money made it easier for them to cover the costs.

6.8. As these three case studies show, the fact that companies would have undertaken the training without the BLA programme does not mean that managers did not value their BLA involvement: one manager indicates how the BLA programme enabled them to bring their plans forward (time additionality), another manager makes the slightly vague statement that it made the training 'easier' and a third respondent focuses on the benefits of the advice and guidance.

Sustainability

6.9. A second key issue for the BLA evaluation - as for any evaluation - is the sustainability of the outcomes:

• Will companies continue to use the BLA tools ( TNA and/or BHC); and,
• Will companies continue to train in a more systematic way?

Future use of BLA tools

6.10. Only 6 case study companies commented that they were likely to continue to use the BLA tools when asked about their future intentions. These companies were predominantly case studies in the HIE roll-out area (4 out of 6), but the sample sizes are too small to warrant any clear conclusions. Local delivery agents agree with their former clients that employers are unlikely to go through the TNA process on their own in future. The employer survey paints a rosier picture, with no less than 90% of the telephone survey employers expecting to undertake a TNA on their own in future.

6.11. Unlike the case study interviews, the telephone survey systematically asked each employer whether s/he was planning to continue use the BLA tools in future; the case studies only present anecdotal evidence. Still, the research team fears that the 90% survey outcome partly suffers from the positive bias that troubles many telephone surveys. The in-depth case studies allowed the evaluation to go beyond first reactions, which almost inevitably were positive in the face-to-face discussions with employers as well. Questioning employers further almost always led to a less positive, more nuanced response. For example, one case study company knew he should use the TNA process in future but " it is like annual staff reviews: you always include training in there, but you never do it". The broad consensus among local delivery agents that companies were unlikely to use the BLA tools on their own in future strengthens the research team's suspicion.

6.12. The main problem is that most of the TNA work seems to been done for the companies by the local delivery agents. As mentioned earlier, one in five survey companies did not even recognise the TNA tool and case study companies indicate that they experienced the BLA process as a 'chat' with the local delivery agent. How can companies continue to use the BLA instruments if they are not even aware that these instruments were being used?

Future approach to training

6.13. When looking more generally at companies' future approach to training, the case study companies offered a small number of real success stories in terms of a complete overhaul of the company attitude to training. However, other case study comments were far less promising and showed that companies:

• Saw their BLA activities as a useful but one-off training boost;
• Interpreted future training activities merely in terms of additional staff hires; and,
• Thought the absence of the BLA support and guidance would seriously hamper any future training attempts.

HIE roll-out case study

The company's manager pointed out that the main difference after the BLA programme was that there now was more enthusiasm for training throughout the whole company. Previously, it had been something they did if it was required; now they did not do it because legislation required it but because employees were more positive about training and because they had realised that there were clear benefits to the individual staff and to the company. The manager thought that he would continue to use the templates used by the local delivery agent: he used to find it very difficult to put his ideas on paper and the new templates would definitely help. He did not think that lack of funding would stop the company in future from training: the whole company attitude had changed.

Tourism pilot case study

The manager of a tourism company was positive about the BLA process but admitted that drafting a TNA or training plan on his own was not something he would do again in future without any support. This was because he was in the " fire-fighting business"; the day-to-day running of the business is too time-consuming.

6.14. The employer survey seems to confirm this impression of partial success. When survey responses about employers' past and future approaches to training are compared, it is apparent that more employers will switch to a positive, more structured and systematic approach to training. Twice as many companies said that they would adopt a formal and systematic approach to training in future post participation in BLAs (30%) when compared to their attitudes pre- BLA (14%). Almost half said that they would take a positive and systematic approach in future (up from 34%). Fewer would train in an ad hoc manner (21% down from 39% pre- BLA) and few would not train (3% down from 10%). Taken together, 53% of companies gave a more positive response when asked about their future compared to their past approach to training.

Table 6.4: Companies' approach to training in future

Source: MORI/ CPC Telephone Survey (Employers)

6.15. The impression of a 50/50 split is confirmed in table 5.5. More than half the survey companies said that as a result of BLA pilots they would approach training more systematically and just under half expect to increase the number of employees undertaking training. Just under a third said that this would involve spending more money on training but 12% did report that they expected to spend less. About half of the survey companies expected to approach training much the same way, spend the same amount of money on training and train the same number of employees in future.

Table 6.5: Nature of training post- BLAs

Source: MORI/ CPC Telephone Survey (Employers)

6.16. This impression of a 50/50 split with regard to the sustainability of the BLA outcomes compares favourably to the findings of the SFDA evaluation, where one third of companies planned to approach training more systematically in future.

6.17. A major issue is that the BLA programme does not address in a sustainable way one of the main barriers to training as identified by case study companies: the opportunity cost of the employer's time spent on planning and organising the training. Most employers seem to be aware of the importance of skills development in the longer term, but cannot find the time to engage in training. Without the local delivery agent there to force them to make that time, they are likely to revert to their old habits of short term 'fire-fighting'. Of the 29 case study companies that commented on what they considered to be a major barrier to training, the vast majority of companies (89%) pointed to time; 20% of companies also pointed to financial constraints.

6.18. The research team's evaluation of the SFDA scheme confirms the importance of time as a barrier to training. Just under 50% of respondents would have liked to participate in the second year of the programme but were prevented by a lack of time. This issue appeared to impact more acutely on micro firms. Time constraints were also a factor in SFDA companies' decision to only participate in certain elements of the scheme or to undertake a more limited number of training activities.

6.19. To some extent, aftercare provision was addressed by the BLA programme. For example, in the SE area the BLA Training Pack for companies contained a section on ongoing support after the BLA, with the contact details of LDS and Business Gateway. Likewise, arrangements for aftercare provision were included in the service level agreement between SE and LDS: " LDS will continue to offer support to post- BLA companies in conjunction with the Business Gateway, in terms of sourcing appropriate training provision to meet their staff and business development needs". There have been examples of continued support being given to former BLA clients by LDS training partners, in particular in cases where there had been a pre-existing relationship with LDS - with one training partner alone already having had further contact with 9 BLA clients at the time of the evaluation. However, there has been no provision in the BLA programme for systematic follow-up visits or calls to post- BLA companies; in general, the employer him/herself has to take the initiative to contact LDS or Business Gateway to ask for further support. At this time, it remains to be seen how many of the BLA companies will choose to do so.

6.20. It is obviously difficult to assess how companies' future training intentions will compare to their actual training activities in say a year or two years time and whether time constraints will indeed limit the long-term outcomes of the BLA programme. But it is reassuring that at least half of the BLA survey companies are optimistic with regard to BLA sustainability.

Wider benefits for the Scottish economy - testing market failure

6.21. The vast majority of Scottish businesses are small companies (98% employ less than 50 employees; 93% employ less than 10 employees). The rationale behind the BLA programme appears to have followed the following logic: economic growth requires productivity growth, productivity growth requires skills development and small companies invest less in skills development than larger ones. Intuitively, it appears logical and convincing that skills development in this type of company, that constitutes 98% of the Scottish economic infrastructure, must be beneficial to the overall Scottish economic performance. However:

• For improvements at the macro/national level to be influenced by additional custom at the micro/firm level, the additional custom must go beyond mere displacement. For example, if marketing training in a hotel means that this hotel increases its turnover at the expense of another nearby hotel, where are the benefits to the Scottish economy as a whole?
• A number of stakeholders indicated that opportunities for productivity gains in small companies in sectors such as tourism, retail and the beauty industry are fairly limited. There might be important one-off productivity and efficiency gains from training hotel staff in sales techniques, for example, but as one LEC employee pointed out: " once all the front staff are trained in customer-service - provided there is not a lot of staff turnover - there is not that much more you can do to improve sales". This comment was echoed by several case study companies who valued the BLA contribution to their company, but indicated their BLA involvement would not lead to further training activities in future; on the contrary, training needs had been dealt with through the BLA programme.

6.22. These caveats matter less if the BLA programme is considered solely from a lifelong learning (as opposed to economic growth) perspective, since training and learning activity contributes to personal development and a better skilled workforce, irrespective of the benefits to the company this employee is working for at the time of his/her training. But, as one LEC employee pointed out, the BLA programme never made clear its core objective: was it supposed to contribute to economic growth - in which case the focus on small companies can be questioned - or was it about skills and workforce development? For SE and HIE management alike the clear link between workforce development and business development was a key element of the BLA design.

6.23. The Scottish Executive's Lifelong Learning Strategy (2003) mentions two specific market failures which were to be tested and possibly addressed through the BLA programme:

• Possible latent skill deficiencies where employers do not recognise that more investment in workforce development could improve their performance; and,
• Difficulties for businesses, particularly SMEs, to access information on appropriate training opportunities - with a focus on the provision of flexible, good quality learning that is accessible in the workplace.

6.24. Assessing market failure is a difficult process and this evaluation does not claim to be able to provide any conclusive answers. However, the scope of this evaluation offers one possible route to give a partial assessment of the possibility of latent skill deficiencies in SMEs: one can look at the extent to which the BLA process has helped companies identify training needs they had not previously been aware of. This is obviously not a perfect measure, but can give some indication of the degree of market failure.

6.25. In general, companies are aware of at least some of their training needs before the TNA. Of the 24 case study companies that commented on their previous knowledge of their training needs, all without exception indicated that they had been aware of all (14 companies) or some of their training needs (10) before the TNA, which seems to suggest only limited latent skill deficiencies. Where companies were aware of some of their training needs, this tended to be mostly limited to specific technical needs; more often than not, the local delivery agent helped identify additional training needs in non-industry specific areas such as management or marketing. When case study employers were asked questions about the additionality of the training, they often also pointed towards these non-technical training courses as courses they would not have invested in without their BLA involvement. Also, owner/managers often are aware of the specific skills development needs of their (shop floor) staff, but local delivery agents pointed out that it is often difficult to convince owner/managers that they might need training themselves as well. Finally, it is important to point out that many of the case study companies that indicated that they had been aware of all training needs prior to the TNA, still signalled that the TNA process had been valuable and beneficial to them: the companies might have been aware of the needs, but the local delivery agent succeeded in highlighting and clarifying them.

HIE roll-out case study

A company in the HIE roll-out area included 10 employees in its training plan, 2 at management level and the rest at shop floor level. The manager himself was not included in the training plan: the local delivery agent tried to convince him to do some management training at the local college and the manager accepted that this probably would have been beneficial, but he simply did not have the time.

Tourism pilot case study

A tourism pilot case study that later withdrew from the BLA process believed that they still benefited from their involvement in the programme: the main benefit was the identification of training needs. They might have been aware of these needs already, but the BLA process highlighted and clarified them.

6.26. A similar exercise can be undertaken to try to assess the possibility of an information failure in terms of companies experiencing difficulties in accessing information about training opportunities: it is possible to look at the extent to which companies themselves knew or suggested training providers to the local delivery agents for at least some of their business needs.

6.27. The vast majority of case study employers (73% or 24 companies) knew or suggested training providers to the local delivery agents for at least some of their business needs; this was again especially the case for more technical training courses. This company involvement in sourcing training provision casts some doubt on the information market failure and indeed on the necessity of the LDS contribution. However, as mentioned earlier the main strength of LDS lies in offering impartial advice about the range of training provision available. LDS confirms that there have been instances where employers opted not for the training provider they had suggested themselves but for another provider suggested to them by LDS, for example because they noticed that the other provider offered a cheaper service.

SE geographic pilot case study

A company in the SE geographic pilot used the BLA programme to train its entire staff on the new computer programme. They already knew the training provider and did not need any support sourcing the training: people approach them all the time, letting the company know that they have written new software programmes especially for the industry.

6.28. When asked who was involved in sourcing training survey companies made the following key points:

• Training providers figured large in all pilots, but were less prominent in the tourism pilot (61% in the tourism pilot compared to 76% overall);
• BLA advisers played a role in all pilots (47% overall);
• LDS were significant in the manufacturing pilot but (as expected) played a more limited role in the HIE pilot (5% in the HIE pilot compared to 26% overall);
• Company representatives themselves played a more limited role - This appears to go against the impression created by the in-depth case studies and discussions with local delivery agents, but a possible explanation lies with the 76% of employers who identified the training providers themselves as involved in sourcing training provision. This percentage might well include cases where the employers already knew the training providers and directly discussed their training needs with these providers, who then identified training courses for them.

Table 6.6: Those involved in sourcing the training provision

Source: MORI/ CPC Telephone Survey (Employers)
Note: Percentages do not add to 100% because of multiple answers

Cost/benefit analysis

6.29. As mentioned earlier, 570 small businesses had participated in the BLA programme by December 2005 in the sense that they had at least undertaken a TNA. The majority of these companies (344 or 60%) had gone on to open a BLA account.

6.30. Total (expected) spend by the Scottish Executive for the three years of the BLA programme is £1,484,000. Almost two thirds of this amount (£872,371 or 59%) represents the (expected) value of the BLA accounts or the actual subsidy awarded to companies. One third of the amount (£501,595 or 34%) was spent on the different local delivery agent contracts. The remainder was spent on other costs including the programme evaluation.

Table 6.7: Value of BLA accounts

Source: SE / HIE Monitoring Data (December 2005)

Table 6.8: Value of local delivery agent contracts

Source: SE / HIE Monitoring Data (December 2005)

6.31. The average cost of delivery per TNA is £880; the average cost of delivery of a BLA account is £1,458 - these amounts do not include the cost of the actual BLA subsidy to companies. The average total cost per TNA delivered (including the BLA subsidies) is £2,410; the average total cost per BLA account opened is £3,994.

Table 6.9: Delivery cost per company (cost of delivery agent contracts)

Source: SE / HIE Monitoring Data (December 2005)

Table 6.10: Total cost per company (cost of delivery agent contracts and BLA accounts)

Source: SE / HIE Monitoring Data (December 2005)

6.32. As mentioned earlier, the BLA programme funded up to 50% of a company's training cost, up to a maximum of £7,500 for the manufacturing and HIE geographic pilots and up to a maximum of £3,500 for the tourism and SE geographic pilots. It is remarkable that the average BLA contribution per company is about half the available amount of support in all four pilots: £3,900 and £3,500 for the manufacturing and HIE geographic pilots respectively and £1,800 for the tourism and SE geographic pilots. This seems to imply that public subsidy might have been set too high. On average, the BLA programme funded 42% of the company training plan in the geographic SE pilot and 48% in the three other pilots.

Table 6.11: Average cost of BLA training / BLA contributions per company per pilot

Source: SE Monitoring Data (July 2005) and LDS Monitoring Data (January 2006)

6.33. When comparing the average BLA contribution with the average company turnover, it is apparent that the BLA contribution represents a marginal contribution only: 0.6% in the tourism and geographic HIE pilots, 0.3% in the manufacturing pilot and 0.1% in the geographic SE pilot.

Table 6.12: Average BLA contribution per pilot per company compared to average company turnover per pilot

Source: SE Monitoring Data (July 2005) and LDS Monitoring Data (January 2006)
Note: The turnover results for the tourism pilots are based on a small sample size and should be approached with some caution.

6.34. The average BLA contribution per learner varies between £166 for the tourism pilot and £571 for the geographic SE pilot.

Table 6.13: Average cost of BLA contribution per learner

Source: SE Monitoring Data (July 2005) and LDS Monitoring Data (January 2006)

6.35. The fact that no conclusive evidence on the impact of the BLA pilot on business performance could be found means that any cost/benefit analysis can only be built around possible scenarios. The total cost of delivery per BLA account opened is £3,994; average company turnover is £1.1 million. This means that, for benefits in terms of increased company turnover to exceed costs, the BLA programme must lead to an average increase in company turnover of 0.4%, on top of any increase in performance due to other (non- BLA) factors. Although it is impossible to assess the likelihood of this increase to take place, it must be noted that this is by no means an implausible scenario.

Added value of the Bla Programme - conclusions

6.36. The additionality of the BLA programme compares favourably with other workforce development programmes: full additionality is limited (5%), but so is deadweight (13%); in the vast majority of cases (82%) the BLA programme achieved partial additionality in terms of training undertaken earlier or to a larger scale than would otherwise have been the case.

6.37. Assessment of the sustainability of the programme is difficult as it is based on company perceptions of their future training intentions. In general company perceptions are positive: about half of the BLA companies believe they will approach training in a more systematic and structured way in future.

6.38. The research findings provide some evidence on the importance of information market failures as barriers to SME training. Neither latent skills deficiencies nor lack of access to information about training opportunities are seen as the main barriers to training in SMEs. The main barrier appears to be the opportunity cost of the manager's and staff's time spent on organising and undertaking training. The BLA programme points towards the right approach for policy-makers to address this barrier: the local delivery agent support fulfils a dual role in both limiting the amount of time the employer has to spent on organising the training and forcing employer to invest that remaining time.

6.39. The total cost of delivery per BLA account opened is £3,994; average company turnover is £1.1 million. This means that, for benefits in terms of increased company turnover to exceed costs, the BLA programme must lead to an average increase in company turnover of 0.4%, on top of any increase in performance due to other (non- BLA) factors.