Scottish Government

1 Executive Summary

Introduction

1.1PACEC were commissioned to evaluate the delivery, effectiveness and value for money of the SMART: SCOTLAND programme between 1999 and 2008. The evaluation was based largely on representative surveys of grant recipients and unsuccessful applicants; and on interviews with stakeholders in the programme.

1.2SMART: SCOTLAND aims to:

• assist small and medium-sized enterprises ( SMEs) to research, develop and exploit new, technically innovative, products with good commercial potential;
• stimulate technical innovation and encourage best practice throughout business;
• strengthen the scientific and technological bases of industry;
• improve the future competitiveness of the Scottish economy by supporting technically innovative SMEs, recognising that these are a dynamic source of new wealth creation, employment and export sales; and
• help contribute towards a climate which encourages investment in innovative technology by individuals, companies and financial institutions and which stimulates a market in technological advancement.

1.3 The main objectives of the evaluation were to assess:

• the extent to which SMART: SCOTLAND is delivered effectively;
• the extent to which SMART: SCOTLAND is an effective policy intervention, and
• the Value for Money ( VfM) of the SMART: SCOTLAND scheme.

1.4 The evaluation aimed to answer specific questions, reported on in the rest of this summary, covering:

• Scheme design,
• Intermediate outputs,
• Effects on business performance and behaviour, and
• Economic costs and benefits.

1.5 The framework for assessing the economic impact comprised the following elements:

• Gross attributable impacts: The change in GVA and employment as a result of the SMART grant.
• Deadweight: What would have happened in the absence of the SMART grant.
• Displacement: The negative effect on competitors
• Leakage: Benefits falling outside Scotland
• Linkages and Multipliers: Positive supply chain effects
• Persistence: The duration of the impacts

1.6 The evidence base assembled in this study comprised:

• Monitoring information of all 363 organisations who received SMART grants (totalling nearly £60m) between 1999 and 2008.
• Survey of 288 grant recipients with a further 18 follow-up interviews. This survey was the main source of evidence for the estimates of economic impact (Gross attributable, deadweight, displacement, leakage and persistence in 1.5 above).
• Survey of 114 of the 438 organisations who were unsuccessful in their application for a SMART grant. Limited use was made of this comparison group since it systematically differs from the recipient group on the criteria of receiving a grant. However, the common ground, of applying for a SMART grant made it the best available comparison group.
• Scottish Input-Output data was used to estimate linkages and multipliers (in 1.5 above), and to convert turnover to GVA.
• Treasury discount rates were used to ensure that cost and benefits are expressed as Net Present Costs and Net Present Benefits.

1.7 A summary of the assumptions used to make these estimates (which are given in full in the Technical Appendix in Chapter 12) are:

• Full time Equivalent ( FTE) employment is the sum of full time plus half part time employment
• Turnover to GVA ratios and supply chain multipliers were taken from the 2004 Scottish Input-Output tables; all other estimates are based on responses to the PACEC survey of grant recipients.
• Gross effects are the respondents' assessments of what happened as a result of receiving the grants, including any increased production and productivity as well as increased research activity.
• The counterfactuals are the respondents' assessments of what would have happened if they had not received their grants and is split between Deadweight (where jobs would have been gained in the absence of a grant) and Safeguarded (where jobs would have been lost).
• Displacement effects are the respondents' assessments of the extent to which sales would be taken up by Scottish competitors in the hypothetical event of their ceasing to trade immediately.
• Leakages are based on the proportions of staff living outside Scotland.
• The average durations of GVA effects are respondent's assessments of the impacts being either short term (2.5 years) or medium term (5 years),
• In cases where an answer was not given for a particular question by a particular grant recipient (either because they did not take part in the survey or because they did not respond to that particular question) an answer was inferred from those who did respond to that question, who had the same characteristics in terms of early/late grant date, type of scheme (feasibility/development), and, if possible, Scottish region.

Scheme design / qualitative fit

How well does SMART: SCOTLAND fit with other public sector support? Are there gaps or overlaps?

1.8 The survey results suggest that SMART and other public sector funding only substitute for and complement one another to a small extent. However, it was also shown that nearly half of grant recipients had used other (non-financial) support and advice in relation to their project in addition to support from the SMART scheme itself, of which two-thirds had used support from the tertiary education sector. In combination, these findings suggest that SMART occupies its own niche among the available sources of public funding, but that it is complemented by other, non-financial public sector support.

What is the comparative effectiveness of the technical and commercial feasibility studies as opposed to the research and development projects? Is one significantly more effective?

1.9 Based on the grant recipients' assessments of the extent to which their projects met their objectives, it was found that the two types of project were equally effective: 81% of both types of projects either wholly or largely satisfied their objectives.

What is the added value arising from Market Research and Business Review Report?

1.10 The number of firms surveyed that had undergone the reviews was small, but the findings indicate that, on balance, the reviews were correlated with lower-than-average project effectiveness and business performance. Importantly, it does not appear that the apparently negative associations of the reviews are a product of their relatively recent vintage. Firstly, by and large, later period projects were associated with more positive findings. Secondly, further comparison of the firms that had undergone the reviews with their direct comparators (projects post March 2006) confirms that they underperform the rest of their cohort.

1.11 These findings do not confirm that lower performance may be caused by the reviews, however. There is a potential bias among the firms opting to undergo a market research and business review. One of the best predictors of actual company performance is their growth ambitions at the start of the project, and the firms who opted to undergo a market research and business review were found to be significantly less likely to have rapid growth ambitions at the outset, and therefore more likely to have poorer performance overall.

What is the benefit of the ceremony and/or awards? Do recipients showcase the awards e.g. on their web-sites or publicity material? How important is it to have SMART: SCOTLAND Awards as a high profile recognition of innovative action within a business? Will this provide encouragement to others to follow suit?

1.12 The prestige of the award was mentioned by several case study recipients as making it easier to get additional funding.

1.13 One recipient felt strongly that there should be a network of "alumni" who have successfully completed projects and generated employment, and further advice and opportunities, such as SMART Plus. "Let the market make a decision about success, but please, SMART, follow it up." There could be a high profile annual dinner addressed by Scottish government and political leaders, which would both raise awareness and facilitate networking.

What is the effect of the issue of Post Completion Conditions ( PCC)? Is 5 years the optimum duration? Is a PCC period necessary or desirable?

1.14 The Conditions appear to have a moderate effect in terms of retaining production within the EU, at least temporarily. 19% of respondents said that it was likely or very likely that they would have located production outside the EU in the absence of the conditions, and 21% said that it was likely or very likely that they would shift production outside the EU at the end of the five year embargo period. The conditions appear desirable for these reasons. However, it is not clear from the survey results whether the five year period is optimal.

What is the role of the UK Government's Small Firms Loan Guarantee Scheme in financing projects?

1.15 The role of the SFLG Scheme in financing projects is very small: only 2% of grant recipients used the scheme as an additional source of finance for their projects.

Do recipient firms go on to claim R&D Tax Credits?

1.16 30% of grant recipients had claimed R&D tax credits, either for their SMART project, or for other projects. A similar proportion was unsure.

Target and intermediate outputs

To what extent are distinct market failures being addressed by SMART: SCOTLAND?

1.17 The original rationale for the scheme focused on the existence of a funding gap for R&D / innovation projects by SMEs, arising from risk and uncertainty associated with these activities. Two-thirds of the grant recipients surveyed were prevented from pursuing their objectives prior to receiving support from the schemes because of a lack of finance / lack of ability to attract finance. Smaller proportions also specifically mentioned risk and uncertainty.

To what extent has SMART: SCOTLAND genuinely encouraged technological innovation in SMEs, and to what extent do supported projects involve significant technological innovation?

1.18 It was shown that grant recipients recorded moderate increases in each indicator of innovation, and a corresponding moderate decline in lack of innovation, following their SMART projects. It was also shown that more than three-quarters of the firms that had taken outputs from their projects into the market assessed the level of technological innovation embedded in them as being significant or high ¹.

To what extent are the benefits generated by SMART: SCOTLAND additional?

1.19 The overwhelming majority of projects would definitely or probably not have gone ahead without SMART support. And those that would definitely or probably have gone ahead anyway would generally have happened later, on a smaller scale or narrower in scope than they actually did. It follows that almost all SMART projects examined were additional; and so, therefore, were the benefits arising. The unsuccessful applicants' responses about the fate of their projects lent weight to the grant recipients' views on what would have happened to their projects without SMART.

What proportion of supported projects have resulted in successful outcomes?

1.20 Two-thirds of projects resulted in new or improved products / services reaching the market, and small proportions also resulted in existing products / services using new or improved processes reaching the market.

To what extent - both before and after completion of supported projects - has SMART: SCOTLAND levered in private sector finance into supported firms, and what factors influence these investments? Does grant support make the business more likely to attract finance than unsupported businesses?

1.21 Around four-in-ten grant recipients sought additional finance for their projects (i.e. over and above their SMART grants); and approaches for additional finance, including approaches to private sector sources, were by-and-large successful.

1.22 In the follow-up interviews several companies said that securing SMART funding made other funding easier to win. "It is the gateway to all other funding; it gives credibility and an entrée."

Are grant recipients more receptive than other SMEs to equity finance?

1.23 Venture capital equity finance and business angel equity finance were both highly ranked amongst the sources of alternative finance sought by the minority of grant recipients who actively pursued an alternative to SMART funding. Equity finance was ranked less highly by the grant recipients who sought additional funding. In other words, equity finance becomes less attractive when grant finance is available. It is not clear from the survey whether grant recipients are more receptive to equity finance than other SMEs. However, these results are not dissimilar from the results of other support for innovation evaluations - e.g. of the Teaching Company Scheme, Knowledge Transfer Partnerships, LINK programmes and Faraday Partnerships - undertaken by PACEC.

Does grant support increase a business's attractiveness as a target for takeover or merger?

1.24 This may be the case. Between the time of grant application and time of survey, grant recipients were more likely than unsuccessful applicants to have become subsidiaries, or to have acquired subsidiaries.

To what extent do SMART: SCOTLAND projects lead to new intellectual property?

1.25 More than three-quarters of grant recipients said that their projects had resulted in them developing new intellectual property and that two-thirds said that they had actually obtained intellectual property, e.g. a patent.

1.26 Some of the interviewees said they found the patent application process difficult and expensive and would have welcomed advice and guidance on intellectual property as part of the SMART package.

Is there evidence of any significant change in these effects over time?

1.27 In terms of project effectiveness and market place success, there were no significant differences between earlier (i.e. pre-April 2004) and later projects (i.e. post-March 2004). However, later projects were significantly more likely than earlier projects to result in the development and acquisition of intellectual property.

Effects on business performance / behaviour

What have been the medium/longer-term impacts of SMART: SCOTLAND with respect to the development of SMEs?

1.28 A net majority of grant recipients said that their SMART project had improved their company growth / performance. Three-quarters of grant recipients said that the effects they described were on-going (4 or more years), whilst a quarter said that the effects were short term (2-3 years).

Do supported businesses go on to spend more on R&D than the sector average?

1.29 Grant recipients had increased their R&D activity more than the unsuccessful applicants had between the time of grant application and time of survey. This suggests that supported businesses go on to spend more on R&D than the sector average because other surveys tend to show that SMEs on average spend very little on R&D.

Do beneficiaries increase their capability to innovate?

1.30 A number of measures suggested that beneficiaries do increase their capability to innovate. For example, large majorities of grant recipients acknowledged that they had: improved their innovation culture; become better able to manage innovation; improved their innovation / technical understanding; and, invested more in innovation in general.

What are the barriers to supported business undertaking future R&D projects? Are barriers different to those evident prior to grant support?

1.31 Two-thirds of grant recipients said that they were prevented from pursuing their objectives prior to receiving support from the schemes because of a lack of finance / lack of ability to attract finance. Nearly three-quarters of respondents said that being a SMART grant recipient made it much easier or a little easier to obtain finance.

Does grant support help SMEs exploit academic research / leading edge scientific knowledge? How can better outcomes be achieved?

1.32 This appears to be the case. A majority of grant recipients felt that their projects had led them to collaborate more with universities, and a majority also reported that they had exploited academic / leading edge research. Grant recipients were also most likely to obtain non-financial support for their projects from the tertiary education sector. Almost all firms that had used tertiary sector support described it as Very useful or Useful; and two-thirds said that they would continue to use the support. In light of these findings, ways in which better outcomes could be achieved in this respect are not obvious.

Economic costs and benefits

What have been the gross and net additional effects of the SMART: Scotland programme?

1.33 In terms of employment, the projects supported by the programme during the evaluation period generated a gross total of 1,381 FTE jobs. The net additional employment effect (gross less deadweight and displacement) was 1,057 FTE jobs. But allowing for linkages and multipliers, the full net additional effect was 1,788. In terms of GVA, the gross total was £122 million, net additional was £94 million and full net additional was £155 million. Taking into account the likely continuation of these actual full net additional effects, cumulatively, the programme generated £674 million in full net additional gross value added.

How have the programme's benefits compared to its costs?

1.34 On average across the evaluation period and each project type, the programme cost per £1 million in cumulative full net additional gross value added was £100,000. Measured in 2008 prices the programme cost per full net additional job was £38,000

How have the costs and benefits varied according to project type?

1.35 Feasibility projects were more than three times more employment cost effective than development projects and more than four times more cost effective in terms GVA. It is likely that it may take longer for the effects of development projects to feed through for several reasons.

• Firstly, because they are around five times larger in terms of value of grant than the feasibility projects.
• Secondly, development projects generally have a more significant level of technological innovation than feasibility projects and are generally more complex in terms of developing new products and processes, and in terms of producing new scientific and technical knowledge.
• Thirdly, businesses awarded feasibility grants were less concerned, compared to those with development grants, to commit resources and time to developing prototypes, overcoming technical problems, producing new scientific or technical knowledge and collaborating.

What have been the wider benefits and costs of the programme?

1.36 Positive spillover effects of SMART projects were fairly common amongst grant recipients' customers. These effects were less commonly associated with unsuccessful applicant's projects that had gone ahead anyway. SMART projects were associated with relatively strong multiplier effects. On the other hand, the displacement effects associated with SMART projects were relatively modest.

General conclusions

Effectiveness of delivery

1.37 Effectiveness of delivery can be assessed from the survey results related to additionality and customer satisfaction. Almost all SMART projects covered by the survey were genuinely additional. It was also shown that every aspect of the scheme was rated relatively highly by grants recipients. The only aspect that was rated as less than Good on average was the application process (and even that was rated as being nearer to Good than Fair). On the basis of these findings, it is reasonable to conclude that SMART: SCOTLAND had been delivered effectively.

Effectiveness of policy intervention

1.38SMART: SCOTLAND'S effectiveness as a policy intervention was assessed in relation to the programme's five aims:

• assist small and medium-sized enterprises ( SMEs) to research, develop and exploit new, technically innovative, products with good commercial potential;
• stimulate technical innovation and encourage best practice throughout business.
• strengthen the scientific and technological bases of industry;
• improve the future competitiveness of the Scottish economy by supporting technically innovative SMEs, recognising that these are a dynamic source of new wealth creation, employment and export sales; and
• help contribute towards a climate which encourages investment in innovative technology by individuals, companies and financial institutions and which stimulates a market in technological advancement.

1.39 Concerning the first aim, the survey evidence showed that two-thirds of grant recipients had managed to take new products or services arising from their projects into the market place, and that more than three-quarters of the firms that had done so assessed the level of technological innovation embedded in them as being significant or high. Substantial minorities of grant recipients attributed increased turnover (37% of respondents) and / or export sales (21% of respondents) to their SMART projects; and follow up questions designed to measure the quantum of change confirmed these findings. These findings signify widespread, but moderate, commercial impacts. However, it should be noted that the commercial impact of earlier period projects has been somewhat greater than that of later period projects, the implication being that many of the projects covered by the survey will not yet have had sufficient time to realise their true potential. It is concluded, therefore, that SMART: SCOTLAND has been reasonably effective in relation to this aim. A major caveat is that the market research and business reviews do not obviously add anything to the outputs of projects; and it would be prudent to examine the issue of their effectiveness more closely than was possible as part of this evaluation.

1.40 Concerning the second aim, one of the key outputs of the programme is the result that more than three-quarters of grant recipients said that their projects had resulted in them developing new intellectual property ( IP) and that two-thirds said that they had actually obtained intellectual property (e.g. a patent). Coupled with the high levels of reported additionality, this suggests that the programme has helped to stimulate a far greater deal of IP than would otherwise have been the case. Given that the majority (71%) of applicants already undertook R&D or innovation activity of some kind before applying for a grant, the research found that the programme helped achieve a 16 percentage point increase in this proportion, to 87% of firms. There was some evidence of these effects spilling over into other businesses associated with grant recipients, especially their customers. Overall, however, these various effects do not appear to be dramatic. It is therefore concluded that while there may be scope to improve the spillover effects to other companies and further encourage best practice throughout business, the programme has been very effective in stimulating technical innovation, as reflected in the high proportions of companies developing and obtaining new intellectual property.

1.41 Concerning the third aim, it was shown that the large majority of supported firms improved different aspects of their capability to innovate. There was also evidence that supported firms had improved their connectedness to the academic science base. It is fair to conclude, therefore, that the scheme has been effective in relation to this aim.

1.42 Concerning the fourth aim, it was shown that SMART: SCOTLAND has assisted innovative SMEs to undertake projects, almost all of which would not have happened, or would have been diminished, without the programme. It was also noted that the programme often had had effects on different aspects of business performance, including on company asset values, export sales and (most frequently) employment. Almost three-quarters of firms reporting these effects considered that they would endure beyond the short term. Again, it is concluded that the programme has been effective in relation to this aim, although there is likely to be scope to increase the incidence and magnitude of effects in question.

1.43 It is less easy to demonstrate clearly, with direct reference to the survey results, whether the programme has been effective in relation to the fifth aim, because the concept of an investment climate is somewhat nebulous. However, there were some useful clues. SMART grants generally covered less than half the cost of projects, most of which would not have gone ahead otherwise. This in itself implies that the programme stimulated private investment in technology development projects. It was also shown that SMART grants had helped to lever-in further private sector finance into grant recipients' businesses. The concept of a market in technological advancement is particularly difficult to test, but it was shown that a small proportion of grant recipients had increased their income from intellectual property. More generally, it was shown that the programme had encouraged moderate increases in innovation activities amongst grant recipients and had increased their capability to innovate. On these bases, it is concluded tentatively that the programme has been reasonably effective in relation to its fifth aim.

1.44 Reflecting on the five aims collectively, it is also concluded that SMART: SCOTLAND has been a reasonably successful policy intervention overall. It is worth reiterating, however, that there is likely to be scope for increasing the incidence and magnitude of the effects that the programme is intended to generate. One way of achieving this might be to encourage more firms to tap into other support alongside the support they received from the programme itself. It was found that just less than half of the firms actually did this.

Value for money

1.45 Value for money is best judged in terms of the scheme's ability to produce additional outputs and outcomes, and in terms of the costs of achieving these outputs and outcomes. On average across the evaluation period and each project type, the programme cost per £1 million in cumulative full net additional gross value added was £100,000. Measured in 2008 prices the programme cost per full net additional job was £38,000. These are the average figures but it should be noted that there were considerable variations in value for money according to project type.