Creative Credits: A randomized controlled industrial

1 THE VENTURE CROWD Crowdfunding equity investment into business CREATIVE CREDITS A RANDOMIZED CONTROLLED INDUSTRIAL POLICY EXPERIMENT Hasan Bakhshi, John Edwards, Stephen Roper, Judy Scully, Duncan Shaw, Lorraine Morley and Nicola Rathbone June 2013 2 THE VENTURE CROWD Crowdfunding equity investment into business About Nesta Nesta is the UK’s innovation foundation. An independent charity, we help people and organisations bring great ideas to life. We do this by providing investments and grants and mobilising research, networks and skills. Nesta Operating Company is a registered charity in England and Wales with company number 7706036 and charity number 1144091. Registered as a charity in Scotland number SC042833. Registered office: 1 Plough Place, London, EC4A 1DE www.nesta.org.uk © Nesta 2013. CREATIVE CREDITS A randomized controlled industrial policy experiment 3 CONTENTS Foreword 5 Executive summary 6 Section 1: Creative experimentation Section 2: Creativity, open innovation and growth 2.1 2.2 2.3 2.4 Introduction Open innovation and business performance Creativity and innovation Creativity and open innovation in SMEs Section 3: Creative Credits: an RCT+ experiment 3.1 3.2 3.3 3.4 Introduction Creative Credits, how it works and its logic model The Creative Credits experiment – rationale The Creative Credits experiment – implementation Section 4: The Creative Credits experiment: results 4.1 4.2 4.3 4.4 Introduction Network additionality of Creative Credits Output additionality of Creative Credits – Innovation and sales impacts Behavioural additionality Section 5 – Implications for policy design and evaluation 10 11 11 12 14 14 16 17 17 19 21 25 26 27 28 32 35 5.1 Introduction 5.2 Implementing the RCT+ evaluation approach Randomization Longitudinal data collection Mixed methods 5.3 Implications for Innovation Voucher Schemes 37 37 37 39 39 41 Annex 1: Implementation and Evaluation 43 A1.1 Implementation A1.2 Evaluation A1.3 The qualitative research process 43 44 50 4 CREATIVE CREDITS A randomized controlled industrial policy experiment Annex 2: Profiling Creative Credits Applicants A2.1 Introduction A2.2 Non–applicants survey A2.3 Characteristics of Creative Credits Applicants 52 52 52 52 Annex 3: Profiling Creative Credits Projects 54 Annex 4: Econometric Tables 56 Annex 5: Estimating the benefit–cost ratio for Creative Credits 66 Annex 6: Case Studies 68 Acknowledgements 72 Endnotes 73 5 CREATIVE CREDITS A randomized controlled industrial policy experiment FOREWORD When it started Nesta’s Creative Credits project looked like a brave but isolated experiment. There had been remarkably little use of rigorous experiment design in industrial policy, innovation policy and the arts. At worst some governments evaluated their policies by surveying grant recipients to ask whether they had found the grants useful. Now things look rather different. Since the project was carried out there has been a resurgence of interest in the rigorous use of evidence and in the use of properly constituted control groups. At Nesta we have launched the Alliance for Useful Evidence which has both benefited from, and contributed to, renewed enthusiasm for measurement and rigour in public policy. The UK government has started to raise its game, building on the success of NICE in healthcare with a batch of new ‘What Works’ centres (though so far these have been mainly focused on social policy rather than business). There’s also growing interest in international organisations, and the European Commission. Vast amounts of public money are spent supporting businesses around the world. Much of this may do good – helping firms to adopt new technologies or to sharpen up their strategies or marketing. But the truth is that nobody knows whether it’s having any real impact. Officials don’t know. Ministers don’t know. And the businesses themselves don’t know. They don’t know because, in stark contrast with fields like medicine, new approaches are introduced without testing. The Creative Credits project should be seen as a pioneer. We will be following it up with new initiatives to support intelligent experimentation and measurement in innovation policy. My hope is that before long it will be obvious that any new programme should be tested before it’s taken to scale. That way we’ll get smarter, more effective policies. But we may also save a great deal of money too... Geoff Mulgan Chief Executive of Nesta. 6 CREATIVE CREDITS A randomized controlled industrial policy experiment EXECUTIVE SUMMARY Very little policy supporting business is subjected to rigorous evaluation. As a result we have no reliable way of knowing if large sums of public money are being wasted. Creative Credits – the focus of this report – is an important innovation in itself: it helps smaller firms become more competitive by connecting them with creative businesses. But the evaluation method used may be even more important, since it not only uses randomization to establish the scheme’s additional impact, but also links that to longitudinal data collection which is vital in helping to assess the longer term effectiveness of different policy tools. This is an approach that should be applied much more widely. At a time when public resources are scarce it’s more important than ever that we find out what works and what doesn’t – and that principle should apply as much to business support as it does to programmes in healthcare or schools, where the UK Government has so far shown much greater interest. This report uses a randomized controlled trial methodology to evaluate a business support scheme called Creative Credits. Creative Credits is a business–to–business voucher mechanism designed to encourage small and medium–sized enterprises (SMEs) to innovate in partnership with creative service providers. In the pilot in the Manchester City Region in the North West of England in 2009 and 2010, SMEs received Creative Credits worth £4,000, which they could use to purchase a variety of creative services from local creative businesses. The SMEs were required to contribute at least a further £1,000 of their own towards their projects. This report evaluates the impact of the pilot. Previous research has suggested that creative businesses, as a source of new ideas and knowledge, exert positive influences on innovation in other firms they are transacting with. For a wide range of innovation measures, firms with stronger links to the creative industries appear to have a superior innovation performance. A UK study combining survey measures of innovation with the input–output tables that quantify supply chain links between sectors finds that firms that spend double the average amount on creative inputs – 6 per cent compared with three per cent of their gross output – are 25 per cent more likely to introduce product innovations. Such findings are indicative at best, as they cannot rule out that the causality runs in the opposite direction. That is, that more innovative businesses have a tendency to make greater use of creative services. In the Creative Credits pilot, the random assignment of SMEs to the treatment and control groups meant that we were able to explore the causal relationship between the use of creative services and innovation with a much greater degree of statistical robustness than would otherwise have been possible. We found that Creative Credits created genuinely new relationships between SMEs and creative businesses, with the award of a Creative Credit increasing the likelihood that firms would undertake an innovation project with a creative business they had not previously worked with by at least 84 per cent (Table A, where the dependent variable takes the value of one for firms that proceeded with their project and zero otherwise, and the ‘Creative Credit’ variable takes the value of one for firms that received a Creative Credit and zero if they did not). 7 CREATIVE CREDITS A randomized controlled industrial policy experiment Table A: Likelihood of firms undertaking their innovation project Dependent variable: Whether or not firms undertook their project Number of observations 451 Adjusted R–squared 0.653 Variable Coefficient Std. Err. t–statistic Signif. Creative Credit 0.840 0.028 29.11*** 0.000 Constant term 0.119 0.017 7.18*** 0.000 Notes: Analysis is based on respondents to the initial baseline survey undertaken immediately after the allocation of Creative Credits. *** denotes significance at the 1 per cent level. The evidence also supports the view that working with creative businesses can lead firms to be more innovative: in the six months following completion of their creative projects, SMEs were significantly more likely than those that were not assigned Creative Credits to have introduced product and process innovations (Table B, Part A). The use of creative services also had a statistically significant positive effect on the sales growth of SMEs over the same period. These findings provide striking evidence that working with creative businesses can generate commercial benefits for SMEs. Twelve months after the completion of the Creative Credits project, however, there was no longer a statistically significant difference between the treatment and control groups in the proportion of firms innovating, nor in their sales growth, in the previous six months. (Table B, Part B). Though only a minority of SMEs at this point reported having received the full benefits from their projects. 8 CREATIVE CREDITS A randomized controlled industrial policy experiment Table B: Output additionality in terms of the probability of innovation: 6 months and 12 months after the completion of the Creative Credits projects Control % firms Treatment % firms t–statistic Signif. N Product or service innovation 145/105 55.9 72.4 2.740*** 0.007 New to the market innovation 126/92 23.0 35.9 2.089** 0.038 Process innovation 142/105 47.2 63.8 2.618*** 0.009 Treatment % firms χ2(6) Signif. N Control % firms 146/107 6.4 7.5 11.5* 0.075 Control % firms Treatment % firms t–statistic Signif. N Product or service innovation 154/113 63.0 70.8 1.345 0.180 New to the market innovation 135/97 32.6 40.2 1.192 0.235 Process innovation 153/111 51.0 47.7 0.517 0.606 Treatment % firms χ2(6) Signif. N Control % firms 155/114 4.7 7.8 7.7 0.261 A. After 6 months Average sales growth B. After 12 months Average sales growth Notes: Table is based on the longitudinal sample. See Annex 1 for details. After six months, firms were asked to select a band within which their sales had grown/fallen in the previous six months and whether they had introduced any innovations in the previous six months. After twelve months, firms were again asked to choose a band within which their sales had grown/fallen in the previous six months and whether they had introduced any innovations in the previous six months. Differences in response numbers between questions and between 6 and 12 months reflect those respondents not answering particular questions and those selecting the (‘Don’t know’) response. * denotes significance at the 10 per cent level; ** at 5 per cent and *** at the 1 per cent level. The χ2(6) test statistic is based on the difference in distribution of sales growth rates, not the average sales growth rates. The number of degrees of freedom in this test is determined by the banding of the sales growth data. 9 CREATIVE CREDITS A randomized controlled industrial policy experiment There was no evidence either that Creative Credits had had a permanent effect on the behaviour of SMEs: for example, those receiving Creative Credits appeared no more likely to work with creative service businesses in the longer term. The qualitative research suggests that for many SMEs where the short-term benefits were not sustained, the creative projects had been too ‘transactional’ in nature, and for others there had been communication difficulties with their creative partners. This suggests that future versions of the scheme which seek to sustain longer term benefits should explore opportunities for targeted brokerage to support the relationship between SMEs and their creative suppliers. Any such modifications should be similarly rigorously evaluated. The evaluation approach that we adopted in this project combined three elements – randomized allocation of Creative Credits, longitudinal data collection, and the use of mixed methods. This has proven to be a powerful methodology, and we argue that it should be used much more widely by the Government and other agencies in developing new innovation support policies. We note that a number of cities across Europe are now adopting their own Creative Credits–style schemes e.g. Strasbourg, Salzburg, Sligo and the Basque Country, inspired by the successes of the Manchester pilot. The UK Government is more generally showing an appetite for innovation voucher schemes. Concurring with a study of innovation support schemes for manufacturing SMEs published last year by the European Commission, we advise strongly that these (and other innovation support) schemes should adhere wherever possible to a randomized allocation as they are rolled out, in order to establish a cumulative evidence base for the effectiveness of their programme design features. 10 CREATIVE CREDITS A randomized controlled industrial policy experiment Section 1: CREATIVE EXPERIMENTATION This report brings together two very recent developments in policymakers’ understanding of innovation. First, the recognition that the creative industries are an innovative sector that can stimulate innovation in other sectors too, through their supply chain linkages. Second, the argument that innovation policymakers should make use of controlled experimentation methods to trial and test new policy interventions. There is a growing body of evidence that creative businesses in sectors like software, architecture and advertising tend to introduce product and process innovations at a higher frequency than other sectors of the economy.1 Furthermore, as a source of new ideas and knowledge, they may exert positive influences on innovation in other firms they are transacting with. Previous Nesta research suggests that for a wide range of innovation measures, firms with stronger links to the creative industries have a superior innovation performance. For example, firms that spend double the average amount on creative products – 6 per cent compared with 3 per cent of their gross output – are 25 per cent more likely to introduce product innovations. To put this into context, these effects are in quantitative terms equivalent to the impacts that national innovation support policies are estimated to have had on innovation. A separate literature emphasises that uncertainty about future opportunities and constraints can be a great barrier to business innovation.2 Innovation policy, it is argued, should aim to reduce this uncertainty, by enabling experimental learning. That is, the goals of innovation policy need to be led by research and learning priorities. Control groups of businesses should be tracked, in order to identify the impact of support measures on business change as well as the mechanisms by which this change is affected. Policymakers should stand ready to scale up or down the experimental inquiry, perhaps significantly, depending on emerging findings.3 This argument lends itself to a more ‘project–based’ conception of industrial policy. In 2009, researchers at Nesta devised an ‘industrial policy experiment’ to explore these arguments further. This report, and the research on which it is based, makes three original contributions. For the first time, it provides robust quantitative evidence at the firm level on the causal – as opposed to merely correlative – relationship between business use of creative services and innovation (and, as such, contributes to the so–called creative industries ‘spillovers’ agenda4). Second, it employs a randomized controlled trial (RCT) experimental methodology to evaluate the scheme’s additional short and longer term impacts on business innovation – a rare example of the use of the RCT method in industrial policy. Lastly, it embeds the RCT in a wider ranging mixed–method evaluation which makes use of qualitative methods to probe the results from the quantitative analysis. As such, it points to a new way that the impact of policies to support business innovation can be rigorously evaluated, one we term ‘RCT+’. 11 CREATIVE CREDITS A randomized controlled industrial policy experiment Section 2: CREATIVITY, OPEN INNOVATION AND GROWTH Key points: Previous studies suggest that partnering with other organizations – through an ‘open’ innovation strategy – can make firms more innovative, and that innovation in turn contributes to enhanced business performance. Other studies suggest that creativity and design also stimulate business innovation, but that a range of difficulties around communication, goal orientation and aspiration gaps between creative staff and others involved in the innovation process can present barriers. Levels of openness and innovation in small– and medium–sized enterprises (SMEs) may further be hindered by behavioural failures in their managers, linked to inertia, risk aversion and myopia. These behavioural failures are likely to lead to levels of openness and innovation which are below what is socially desirable, a factor which has motivated a range of innovation voucher schemes which have attempted to ‘nudge’ SMEs into being more innovative by connecting them with universities. The Creative Credits scheme intends to overcome behavioural failures and build new partnerships between SMEs and creative service providers and so stimulate innovation and organizational learning. 2.1 Introduction Previous research has established positive relationships between innovation and growth, and, separately, creativity and innovation. In this section, we briefly review this evidence, which provides the broader context for the Creative Credits scheme. We emphasise the challenges that SMEs in particular face in undertaking open innovation, and discuss some of the policy responses which have been adopted. Section 2.2 reviews the evidence on the relationship between open innovation and business performance. Open innovation involves working with partners, whether customers, suppliers or other organizations, to generate new products or services. Recent evidence suggests that open innovation is becoming more widespread amongst SMEs.5 However, as other research has made clear, open innovation poses particular challenges for SMEs because of their lack of capacity to both seek and absorb external knowledge when compared with larger firms.6 12 CREATIVE CREDITS A randomized controlled industrial policy experiment Section 2.3 considers some recent evidence on the relationship between creativity and innovation, with a particular focus on the value to firms’ innovation processes of accessing external creative services through suppliers. Again, we might anticipate that SMEs face particular issues in working with external creative service providers due to limited absorptive capacity and/or internal resources, though the existing evidence base for this is much weaker. These issues are discussed in Section 2.4 along with some possible policy responses. 2.2 Open innovation and business performance A growing body of evidence suggests that collaborating with external partners on innovation (or ‘openness’) may influence the performance of firms in a number of ways, through stimulating creativity, enhancing product quality and providing reputational benefits which signal the quality of firms’ innovation activities.7 External linkages may also provide access to networks which create commercial opportunities, and allow firms to search their technological environment in a more systematic fashion, resulting in improved access to technology developed elsewhere.8 Because the success of any innovation process is uncertain, firms may have an incentive to pursue several such external linkages at the same time: having more linkages or different types of linkage spreads the risks of innovation on the one hand, and increases the probability of obtaining useful knowledge from outside of the firm on the other.9 Furthermore, there is empirical evidence that knowledge gained from different sources tends to be complementary, and also complementary with respect to internal R&D in shaping innovation performance.10 Using external knowledge linkages, however, also has potential disadvantages, both for large and small firms. Firms may, for example, face difficulties in defending their intellectual property rights in relationships with partners, which may limit the commercial benefits from collaborating. Furthermore, sourcing information from, or collaborating with, a larger number or variety of partners raises search, management and monitoring costs. Absorbing knowledge simultaneously from a large number of sources can be challenging too.11 There may, for example, be limits to how much attention managers can give to, and cognitively process, competing sources of information. The returns to adding linkages are therefore likely to be diminishing, and may even turn negative beyond some ‘saturation’ point. As a consequence of these trade–offs, some studies detect an ‘inverted U–shaped’ relationship between the breadth of external knowledge linkages (that is, the number of different types of knowledge linkage) and innovation performance.12 A robust research finding is that innovative firms are more profitable.13 This may be because innovators are able – at least for a time – to enjoy monopoly profits while their intellectual property is protected, or alternatively it may be because innovating firms have a tendency to introduce successive innovations over time, and are therefore able to sustain high profits, even if the profits from any one innovation are transitory. Studies also provide clear evidence of the positive relationship between innovation and firm productivity,14 and innovation and business growth in a range of sectors.15 A recent study from the UK, for example, provides evidence of the positive impact of innovation on growth in a sample of business services firms (as well as the impact of openness on innovation itself16). 13 CREATIVE CREDITS A randomized controlled industrial policy experiment The evidence on the effect of openness on innovation and business performance is important for the Creative Credits experiment, in that it points to the potential impacts we might observe. If Creative Credits succeeds in stimulating firms to develop the extent of their collaborative relationships with innovative businesses (pointing to the notion of ‘network additionality’ in which a policy intervention helps or encourages a firm to extend its innovation networks17) we might expect to see increased innovation and improved business performance (‘output additionality’). However, network and output additionality are not the only beneficial effects which might be anticipated from Creative Credits.18 Undertaking innovation projects can also have other capability or learning benefits for firms, which are best described under the heading of ‘behavioural additionality’. This captures the extent to which the behaviour of a firm’s managers changes as a result of an innovation project.19 Behavioural additionality might relate broadly to technological activities (firms may explore new technological or market opportunities after contact with a new technology or market area, for example), processes (firms may develop new coordinating systems for R&D and innovation, either in–house or for managing or developing external links), and people (firms may develop new competencies, ranging from project management skills, through to various acquired technological and market capabilities).20 These three notions of additionality – network, output and behavioural – provide the structure for our analysis in this report of the impacts of the Creative Credits experiment (Figure 1.1). Figure 1.1: Additionality and the Creative Credits scheme Network Output Behavioural 14 CREATIVE CREDITS A randomized controlled industrial policy experiment 2.3 Creativity and innovation The ‘design intensity’ of an increasingly wide range of products suggests that design plays an important role in innovation success.21 Designers enhance the functional, emotional and symbolic value of products in the eyes of consumers and businesses.22 Design–driven, or design–led, innovation processes which adopt novel perspectives on particular innovation problems may also contribute to the development of more radical innovations with the potential to disrupt markets and create large new profit streams.23 There are now a large number of studies providing empirical evidence for these claims.24 A more recent development is evidence that the benefits of a design–oriented approach are reflective of the wider business value of engaging with creative services which include, but are not restricted to, design.25 Managing creativity and design in the innovation process is not always easy, however. Case studies of the involvement of designers in innovation highlight issues relating to: cultural barriers associated with language and designers’ self–image; and work process barriers related to the different working practices of designers and others involved in innovation. So, for example, “the goal of good industrial design [is] perceived by designers to be the creation of an ‘iconic’ product – one that [will] become famous and instantly recognizable. By contrast, managers [perceive] design as a means to build brand and achieve the right price.”26 Similar tensions have also been observed between designers and marketing staff involved in innovation: “This frequently [leads] to design–marketing conflict. Designers [are] compelled to express performance parameters in marketing terms, of which they [have] no experience and [are] unable to understand”.27 Goal incongruity between marketing staff and others involved in innovation is, in turn, held by some to lead to conflicts in firms attempting to integrate design into their innovation activities.28 The evidence on creativity, design and innovation has at least two implications for the Creative Credits experiment. First, the Creative Credits, insofar as they help SMEs access creative services, should have additional output effects on innovation. Second, we might expect that issues around communications between design and creative staff and others involved in the innovation process will give rise to potential conflicts between firms, raising potentially important barriers to the positive effects on innovation. 2.4 Creativity and open innovation in SMEs Recent evidence suggests that SMEs are no different from the general population of firms in innovation being a driver of business performance.29 There are, however, good reasons for thinking that SMEs differ from larger firms in terms of the ability to realise the benefits of open innovation and creative partnerships. Open innovation poses particular challenges for resource–constrained SMEs, associated with the need to: (i) develop mechanisms for identifying useful external knowledge; (ii) build organizational structures to support collaborations, and (iii) absorb externally developed ideas and technologies and make them fit for the purpose of their own businesses. These same challenges can make SMEs less attractive as potential collaborators, presenting further barriers to establishing partnerships.30 Partly as a consequence, it is not surprising that smaller firms are generally found to be less ‘open’ – that is, they have fewer external innovation partnerships on average – than larger firms.31 15 CREATIVE CREDITS A randomized controlled industrial policy experiment This does not mean, however, that the benefits of openness for SMEs are necessarily smaller than for large firms. Indeed, SMEs’ more limited internal resources may mean it is actually more worthwhile for SMEs to engage in open innovation strategies than larger firms. One recent study finds that small firms benefit greatly at the margin from external linkages, but that they reach their cognitive limit to benefitting from openness earlier than larger firms, at around 3–4 different external linkages.32 Arguably, these cognitive limits are perpetuated by behavioural failures on the part of managers running SMEs – such as inertia (the tendency to accept the status quo, no matter how strong the case for change might be), excessive risk aversion (cognitive biases push owners of SMEs to make choices that anticipate more certain outcomes, particularly at the boundaries of their knowledge or experience), and myopia (the tendency to opt for short– term gain at the expense of longer term, strategic decisions).33 These behavioural failures likely contribute to a reluctance on the part of SMEs to undertake open innovation, which requires novelty, risk tolerance and a willingness to make strategic investments. Policymakers have responded with instruments such as innovation vouchers to overcome some of these behavioural failures and encourage SMEs to engage in open innovation with universities.34 Innovation brokers have also been used in some situations to help SMEs identify firms with complementary capabilities and develop new innovation partnerships. Measures to support collaborative innovation have also been a consistent feature of innovation policy in some EU countries leading to the development of strong traditions of open innovation. This type of measure has been less common in the UK and Germany than in some other EU countries, however, most notably Finland.35 16 CREATIVE CREDITS A randomized controlled industrial policy experiment Section 3: CREATIVE CREDITS: AN RCT+ EXPERIMENT Key points: In the scheme’s pilot in the Manchester City Region between October 2009 and November 2010, 150 Creative Credits – vouchers with a face value of £4,000 – were awarded to eligible SMEs to develop collaborative innovation projects with a creative business. The SMEs were required to contribute a further £1,000 of their own towards their project. We profile the anticipated impact of the Creative Credits scheme in a ‘logic model’. This hypothesises that in SMEs, behavioural failures linked to inertia, excessive risk aversion and myopia lead to lower levels of collaboration than is socially desirable. Creative Credits may help firms to overcome these behavioural failures and lead to increased levels of connectivity between firms and therefore innovation. In the longer term, the Creative Credits logic model suggests that this should lead to behavioural changes and stronger business performance. We tested the Creative Credits scheme’s logic model with a controlled experiment, using what we call an ‘RCT+’ evaluation methodology. This has three key features: (i) a randomized allocation of Creative Credits to firms (the ‘treatment’ group), enabling the scheme’s additional impacts to be rigorously evaluated by comparing innovation and business performance in the ‘treatment’ group with a ‘control’ group of firms made up of non–recipients; (ii) a longitudinal data strategy, allowing the longer term as well as short–term impacts to be assessed, and (iii) a mixed–methods empirical approach, combining qualitative interviews with quantitative survey techniques. We note the rarity of randomized controlled trials in industrial policymaking. We argue, however, that their use is important in situations where selection biases are severe – that is, when the businesses that self–select to, or that are selected to, participate in support programmes are already more innovative than the ‘average’ business the policymaker is targeting. A recent evaluation of SME innovation support schemes in the manufacturing sector, funded by the European Commission, suggests that this is not the exception, but the rule. There are arguments for randomization that are related to cost effectiveness too: it is a cheaper, simpler and, arguably, fairer method of allocating constrained business support resources amongst firms than alternative methods based on subjective assessments of worthiness by programme managers. 17 CREATIVE CREDITS A randomized controlled industrial policy experiment 3.1 Introduction The empirical evidence reviewed in Section 2 provides the broad context for exploring interventions to encourage SMEs to work with creative partners as part of their innovation activity. In this section, we focus on the different elements of the Creative Credits experiment: its underlying logic model; the justification for the use of formal experimental methods; the implementation of the Creative Credits pilot in the Manchester City Region; and, the deployment of our evaluation methodology. Section 3.2 focuses on the logic model for Creative Credits. This sets out the rationale for intervention, suggests how Creative Credits should work, and forms hypotheses about its short–term outputs and longer term outcomes. Testing the causal process envisaged in the logic model is the main focus of the evaluation. The traditional approach to evaluating business support programmes is to conduct an ex post evaluation, asking participating businesses to reflect on their experience of the scheme. We adopt an alternative experimental approach which combines three main elements: randomized allocation of Creative Credits, longitudinal data collection, and the use of mixed methods. Section 3.3 outlines the background to the Creative Credits experiment, discussing both the methodological and economic arguments for taking the RCT+ approach. Sections 3.4 and 3.5 briefly describe the implementation of the Creative Credits scheme and its evaluation. 3.2 Creative Credits, how it works and its logic model Whether or not they recognise it in these terms, all firms have innovation processes, some well developed, others not. The quality and the extent to which a firm’s innovation process is embedded in its business practices determine its long–term competitive position. There is a great deal of evidence for market (or system) failures in SMEs’ innovation processes. Most obviously, SMEs face difficulties in accessing external funding sources, owing to information asymmetries between lenders and SMEs about their commercial prospects.36 Such failures may be compounded by behavioural failures on the part of the managers of SMEs: inertia, excessive risk aversion and myopia may form significant barriers to changing attitudes and routines which, it is argued, leads firms to make insufficient investments in innovation.37 Together, these market, system and behavioural failures can act as overwhelming constraints for SMEs when it comes to introducing novelty. Interventions such as Creative Credits aim to overcome these failures, by encouraging collaboration with businesses in the creative industries. The aim is to test whether a one– off injection of creativity can ‘nudge’ SMEs into being more innovative. The idea of nudging innovation this way is prompted by theoretical arguments that creative businesses are less prone to the behavioural failures discussed earlier, and empirical research which suggests that firms which make greater use of services from the creative industries have superior innovation performance.38 The ‘logic model’ for the Creative Credits scheme is shown in Figure 3.1. This links the justification for public intervention in the relationship between SMEs and creative businesses, the scheme’s objectives, the process by which the scheme operates, its 18 CREATIVE CREDITS A randomized controlled industrial policy experiment immediate outputs and intended longer term outcomes.39 It also points to the Key Performance Indicators which provide an indication of the effectiveness of the Creative Credits scheme. Figure 3.1: Logic model for Creative Credits Outputs Situation Modality Low level of interaction between small businesses and creative service providers Outcomes-impact Inputs Funding for Creative Credits Partner commitment Demand–led Creative Credits with random allocation Motivation and search barriers Assumptions Creative Credits will stimulate linkages and innovation Partners Short Manchester City Region SMEs Awareness Behaviour Attitudes Practices Manchester City Region creative service providers Networks Innovation Opinions Innovation collaboration Medium Long Innovation collaboration Commercial impacts Environment Contingency factors may influence outputs in determining outcomes The logic model describes how the award of a Creative Credit should help SMEs overcome the barriers to innovation associated with financial barriers and behavioural failures: SMEs develop new collaborations with a business partner in the creative industries, which lead to innovations and improvements in firms’ longer term competitive positions. For the creative partner, the Creative Credit represents a potentially valuable new business opportunity (the scheme requires that creative businesses and SMEs have not previously worked with each other) certainly in the short– and potentially the longer term too. The final elements of the logic model relate to the specification of the scheme’s outputs and their relationship to longer term outcomes, identifying possible contingent factors. In the short term, the outputs from Creative Credits are measured primarily in terms of increased levels of interactions between SMEs and creative businesses relative to what would have prevailed absent the intervention – or in other words, network additionality. In the longer term, we anticipate that this might generate three types of outcome: output additionality, as innovations result from the Creative Credits projects which impact on sales and growth; behavioural additionality, as organizational learning takes place and SMEs learn to work with, and value more highly, creative inputs to innovation; and, perhaps, further network additionality, as SMEs absorb the value of partnering for innovation. The logic model highlights that such outcomes are contingent on other factors influencing firms’ innovation processes, however, and that these need to be controlled for, along with market and other contextual factors, if the scheme is to be convincingly evaluated. 19 CREATIVE CREDITS A randomized controlled industrial policy experiment 3.3 The Creative Credits experiment – rationale As the evidence reviewed in Section 2 suggests, there are arguments for public intervention to promote connectivity between SMEs and creative service providers to stimulate innovation. Public intervention in this arena is a recent development, however, and goes beyond current experience with innovation vouchers which have largely focussed on linking firms to technical knowledge providers such as universities.40 This raises the question of how the effectiveness of the Creative Credits scheme should be assessed. A standard approach might be to commission a small–scale pilot exercise and undertake monitoring and an ex post evaluation. This would typically involve the allocation of support using some type of peer review or assessment mechanism to explore the appetite for any larger scheme which might follow. Evaluation would then seek to assess the impact of the scheme, potentially allowing for any selection bias, using econometric methods, which might have occurred in the allocation of the business support. This type of evaluation – allowing for potential selection biases – has been emphasised by the OECD as best practice in ex post evaluation and has been widely applied in recent years.41 The ‘fundamental problem of causal inference’ is that the treated outcomes and non– treated outcomes for any single firm are never jointly observed.42 The analytical problem this raises is how to estimate the difference between the actual realised outcomes and the potential outcomes if no treatment had been administered. Ideally, the substitute for the unobserved (un–treated) outcome needs to meet two criteria: (i) it should be observable to the researcher and (ii) it should be an internally ‘valid’ substitute for the set of un–treated outcomes.43 Validity in this sense requires that “the only difference between the member of the control group and the member of the treated group corresponds to the fact that the latter is treated and the first one is not” (Reiner, 2011, p. 18). The standard approach to piloting business support schemes breaches this requirement. The pilot might, for example, be targeted on firms which are ‘better’ than others in some sense, suggesting that the difference between the treated and control group would reflect both the effects of this selection and the treatment itself. One recent study, for example, illustrates how funding allocations in the Norwegian Research Council are based on ex ante project rankings generating a potential selection bias when evaluating the Research Council’s funding decisions.44 Related biases may also arise in cases where firms self– select to apply for a scheme. In the case of an innovation support measure, for example, only firms which are at the point of application interested in innovating in the current period are likely to apply. In its assessment of innovation support schemes for SMEs in the manufacturing sector in seven European regions, the European Commission’s GPrix project finds that businesses that are selected to participate in standard programmes do not actually benefit from them, whereas support allocated to firms on a random basis has positive impacts.45 The prevalence of these selection issues in policy evaluation has led to the development and widespread application of econometric approaches which can ‘control’ for potential selection biases. Typically, such approaches involve a two–step process, modelling first the probability that a firm will be in the treatment rather than the control group, and second the impact of the treatment ‘controlling’ for any selection biases. Implementing this type of approach, however, involves making restrictive structural assumptions about the underlying causal processes and often poses significant statistical issues in terms of identification, requiring for example the use of a variable or group of variables which influence the 20 CREATIVE CREDITS A randomized controlled industrial policy experiment probability of being in the treatment group but which have no influence on subsequent outcomes. These procedures – called instrumental variables techniques – are difficult to validate as tests which attempt to do so have notoriously low statistical power; that is, they have a high tendency to conclude that the identification schemes are valid when in fact they are not. The very great difficulties in controlling for selection bias point to the alternative experimental approach with random allocation of firms into the treatment and control groups. “Random assignment [also] removes any systematic correlation between treatment status and both observed and unobserved participant characteristics. Estimated treatment effects are therefore free from the selection bias that potentially taints all estimates based on non–experimental sources of information.”46 In other words, with random allocation of firms between the treatment and control groups, it is possible to directly infer the impact of the scheme as the difference between outcomes for the two groups. A number of potential issues arise, however, even with this type of RCT approach:47 • It is not clear if the results of any single experiment can be generalised to the target population and to what extent they can be generalised to other populations – that is, whether the results are ‘externally’ valid. • It is possible that substitution biases occur if a firm in the control group, through either applying to Creative Credits or participating in the scheme’s evaluation, accesses other types of public support for their proposed project, leading to an under–estimation of the additional impact of the scheme. • Adverse selection biases may occur if firms that are not representative of the wider business population apply to Creative Credits e.g. firms that apply may have more of a history of applying to public support schemes or perhaps be more ‘innovation– engaged’ (and Annex 2 of this paper confirms this is indeed what we found in the pilot). • A further source of possible bias is if there are patterns in the types of business that drop out from the scheme after the random assignment. Depending on the scale of any drop–out, additionality estimates compiled on the basis of the original randomized treatment group may not reflect those of the group actually treated. In the Manchester pilot, we attempted to minimise the possibility of such a bias by randomly selecting eligible firms to be on a reserve list to take the place of any businesses that did not take up their Creative Credits after they had been assigned. While these issues are potentially important and have implications for the statistical robustness of what we can infer from an RCT, we concur with the European Commission’s GPrix project48 in believing that the statistical case for using an RCT methodology to evaluate a scheme like Creative Credits, where selection biases would otherwise potentially be severe, is overwhelming. Our case is compounded by economic arguments for random assignment: unlike alternative approaches based on expert opinion on which firms are worthier of support than others, the selection process is cheap to administer, a not insignificant consideration when policymakers are facing a tight financial environment. We characterise the evaluation approach we developed in the Creative Credits experiment as ‘RCT+’, in that it combines an RCT–based quantitative assessment with longitudinal data collection and qualitative follow–up to help test the logic model in Figure 3.1. The 21 CREATIVE CREDITS A randomized controlled industrial policy experiment evaluation focuses on the ‘causative’ elements of the programme theory underlying the scheme. In other words, the evaluation objective is not just to assess the additionality of programme outcomes but also to consider whether these outcomes are being achieved through the mechanisms envisaged in the underlying programme theory.49 This requires a theoretically grounded analysis of process and causal mechanisms alongside the evaluation of outcomes.50 In empirical terms, this underlines the value of our methodological approach: that of a mixed–methods evaluation combining a qualitatively structured examination of underlying processes and decisions with a more quantitative assessment of causal process and outcomes.51 3.4 The Creative Credits experiment – implementation52 Creative Credits was piloted in the Manchester City Region in the North–West of England between September 2009 and October 2010.53 150 Creative Credits were distributed in two waves roughly six months apart. Each Creative Credit had a face value of £4,000 with recipient firms also required to contribute a minimum of £1,000 towards the cost of the project with their creative partner. In fact, the average budget for the Creative Credits projects across the two waves was £5,400, with SMEs contributing in practice an average of £1,400 over and above the value of the Creative Credit. The first wave of Creative Credits opened for applications in September 2009 and the second in February 2010. Applications from SMEs were received and each of these were checked by Nesta staff for eligibility. The Creative Credits scheme was open to SMEs (excluding sole traders) in almost any sector of the economy with the exceptions of primary industries and the creative industries. Creative Credits was marketed through various channels in the Manchester City Region. The regional partners played a crucial role in promoting the opportunity to relevant SMEs and creative businesses through Business Link Advisors, North West Development Agency networks, Manchester City Council networks, and other business and financial advisors to SMEs. Promotional efforts also included a regional media campaign, a launch event, email and telemarketing, as well as an online presence developed through Facebook and LinkedIn. More than 2,000 firms made an enquiry about the scheme through its two waves of operation. In the event, a total of 672 SMEs made eligible applications for the Creative Credits scheme: 312 in the first wave and 501 in the second, 141 applying in both waves. Amongst applicants, the sector with greatest representation was services, and in particular: Consultancy, Professional Services, General Business Services and Retail. No applications were received from the Aerospace or Medical sectors. Applications were received from all boroughs included in the pilot. Figure 3.2 illustrates the geographical breakdown of the group of SMEs which received a Creative Credit compared with that for all applicants. The turnover of the group of firms applying for the scheme also varied relatively widely (Figure 3.3), although over one–half of all applicants had an annual turnover of less than £500,000. And within this group, a fifth of all applicants reported that their turnover was less than £100,000 per year. 22 CREATIVE CREDITS A randomized controlled industrial policy experiment Figure 3.2: Percentages of applicants and recipients by borough 30% 25% 20% Percentage of applicants 15% and recipients 10% 5% M an B ol B to n ur y C i c t C h ity es y o of ter f Sa lfo rd C on gl et M on ac cl es fie ld O ld ha m R oc hd al e St oc kp or t Ta m es id e Tr af fo rd Va le R oy W al ar rin gt on W ig an 0% Borough Recipients percentage Applicants percentage Figure 3.2: Percentages of applicants and recipients by business size (turnover) 60% 50% 40% Percentage of applicants 30% and recipients 20% 10% Recipients percentage Applicants percentage £4 . sw er + o an £5 M N M 99 5M ‐£ 4. 49 M M ‐£ 4. 99 3. ‐£ Turnover £4 M 3. £3 . 5M ‐£ M £3 M .5 £2 49 M 99 2. ‐£ £2 to M £2 M .4 9M 9M .9 £1 to M .5 £1 M ‐£ 1.4 9M £1 9K ‐£ 99 K 0 0 £5 £0 ‐£ 49 9 K 0% 23 CREATIVE CREDITS A randomized controlled industrial policy experiment Comparing the group of Creative Credits applicants to the eligible population of companies in the MCR provides an indication of the penetration of the scheme. Data from Companies House suggests that there were perhaps around 4,200 eligible firms in the MCR of which 672, or around 1:6, applied for the Creative Credits scheme. Applications were not, however, evenly spread across the size distribution of firms, with micro businesses (those with less than ten employees) and firms with more than fifty employees under–represented in the group of Creative Credits applicants, and firms with between ten and fifty employees significantly over–represented. Compared with the overall application proportion of 1:6, this meant that for the individual size bands the scheme penetration was: 1:8 for micro firms with less than 10 employees; 1:2 for firms with 10–50 employees and 1:16 for SMEs with more than 50 employees. Once applications had been received, a lottery of the eligible applications was held in each of the two waves to allocate Creative Credits, with 75 firms in each group being notified that they had been ‘awarded’ a Creative Credit. In addition, a small number of ‘reserve’ firms were also selected by lottery to replace those where the awarded Creative Credit was declined by the SME.54 In total, 22 per cent of the eligible businesses that applied were awarded a Creative Credit. For firms in the first wave these awards were made in October 2009, and in the second wave in March 2010. Subsequent to the award of their Creative Credits, SMEs were encouraged to identify a creative partner and develop a collaborative project proposal. To help with this process, a web–based marketplace – a ‘Creatives Gallery’ – of creative firms55 was designed and made available by Nesta to all eligible SMEs in both the treatment and control groups. The majority of creative businesses on the Gallery classified themselves as offering ‘Design or Web Design’ (79 per cent), with 63 per cent of businesses listing this as their primary offering. Over 50 per cent of businesses offered ‘Advertising or PR’ as a service (20 per cent of businesses listed this as their primary service). Twenty–four per cent of businesses offered ‘Film and Video’ services; 9 per cent listed this as their primary service. Very few businesses listed any other service as their primary offering. ‘Software’ was not listed by any businesses as their primary service but it was offered as a secondary service by 13 per cent of creative businesses. ‘Publishing’ and ‘TV and Radio’ were also significant secondary services offered. The aim of creating the online Gallery was to explore the potential for a minimal brokerage model and reduce the burden of administrative costs of the pilot – for the programme managers and the SMEs alike. SMEs awarded a Creative Credit were encouraged to navigate the Gallery on their own to select their preferred provider, and creative firms on the Creatives Gallery were also able to contact prospective SME partners, as per normal business practice. In the event, the 150 Creative Credits were ‘spent’ with 79 creative service providers, with one service provider working with 13 Creative Credit recipients. Once a partnership was formed, all Creative Credits projects were required to be completed within five months, i.e. by end–March 2010 for those in the first wave and mid–September 2010 for those in the second wave. As discussed earlier, random allocation was used to avoid any systematic bias in the characteristics of treated firms and to help provide a more robust indication of the extent of additionality of Creative Credits.56 Care was taken wherever possible to minimise selection biases in the experiment: so, for example, in the promotion of the scheme, 24 CREATIVE CREDITS A randomized controlled industrial policy experiment randomized business lists were used in the marketing calls and care was taken to minimise locational or sectoral biases. Details of the implementation of the randomization, and the RCT+ methodology within which it was embedded, are provided in Annex 1. The evaluation of the Creative Credits experiment began in September 2009 and aimed to consider both the scheme’s short–term additionality and its potential longer term outcomes. Qualitative and quantitative data collection comprised of two quantitative surveys at the start and finish of the Creative Credits project and two further surveys conducted six months and twelve months after its completion. Qualitative data collection was similarly conducted in four stages, in each case after the quantitative survey to avoid influencing the survey responses. The initial quantitative baseline survey (Survey 1) was conducted immediately after the lottery to award the Creative Credits. 451 responses were received – 150 from the treatment group (those awarded a Creative Credit), 301 from the control group (those who applied and were unsuccessful). Over the next three surveys attrition was, as expected, evident in each survey (Table A1.1 and Figure A1.1), and cash incentives were used to maximise response rates (though, as we discuss later, this does not appear to have induced any biases in our results57). By Survey 4, a year after the Creative Credits projects finished, respondent numbers had fallen to 157 among the control group (52.2 per cent of those firms initially responding to Survey 1) and 117 among the treatment group (78.0 per cent of Survey 1 respondents). Comparison between the characteristics of respondents to all of the four surveys – the longitudinal sample – and those firms which dropped out confirm, however, that there are no systematic differences between firms who responded to all four surveys and those who dropped out during the study (Table A1.2). Stages 1, 2 and 4 of the qualitative data collection consisted of semi–structured interviews (106 in total), while Stage 3 comprised two workshops using the Journey Making58 technique from Operational Research. SMEs in the treatment group and creative businesses working with them were both included in the process. As with the quantitative surveys, attrition was evident, especially amongst the creative businesses: the stage 1 interviews covered 24 pairings (SMEs and their creative partners), while by stage 4 this had fallen to 11 pairings, plus the ‘surviving’ partner of 9 further pairings in cases where the other business had closed or had withdrawn from the interview process. Further details of the samples and the qualitative research methodology are given in Annex 1. 25 CREATIVE CREDITS A randomized controlled industrial policy experiment Section 4: THE CREATIVE CREDITS EXPERIMENT: RESULTS Key points: Creative Credits generated significant short–term network additionality, with the award of a Creative Credit increasing the likelihood that firms would undertake their innovation project (with a creative business they had not previously worked with) by at least 84 per cent. The evidence also suggests there was statistically significant output additionality in the form of product and process innovations: in the six months immediately following the Creative Credits projects, firms were significantly more likely to have introduced product and process innovations than those that were not assigned Creative Credits. Creative Credits also had a (weakly) significantly positive effect on the sales growth of the treatment group after 6 months. Twelve months after the completion of the Creative Credits projects, however, there was no longer a statistically significant difference between the treatment and control groups in the proportion of SMEs innovating, nor in their sales growth, in the previous six months. Though we note that only a minority of firms in the treatment group at the twelve–month stage reported having received the whole benefit from their Creative Credits projects. We go some way to validate the robustness of this particular finding by establishing a significant positive correlation between firms who said shortly after completing their projects (in Survey 2) that they were expecting peak benefits in a year’s time and those who said that they had in fact done so after twelve months (in Survey 4). There was no evidence of network additionality after twelve months, or of behavioural additionality: SMEs receiving Creative Credits were no more likely to have worked with creative service providers or other innovation partners than other firms. The qualitative research suggests two main reasons for this. First, in some cases SMEs felt that their Creative Credits project had been ‘transactional’ in nature – self– contained projects which could in principle have been undertaken with any number of innovation partners, and, second, in other cases there were clear incompatibilities between the SMEs and their creative partners, something which – according to some SMEs – Nesta might have addressed if it had offered opportunities for brokerage in the scheme. 26 CREATIVE CREDITS A randomized controlled industrial policy experiment 4.1 Introduction In this section we focus on the results from implementing the RCT+ evaluation of Creative Credits. We test the different elements of the logic model outlined in Section 3.2 and the longer term contribution of Creative Credits to strengthening SMEs’ open innovation partnerships. We begin in Section 4.2 by assessing the short–term network additionality of the scheme, and in particular whether the award of a Creative Credit actually increased the probability that an SME went ahead with its project (with a creative partner with which they had not previously done business). Section 4.3 explores the scheme’s output additionality insofar as it impacted on levels of innovation activity and sales growth. Data from Surveys 3 and 4 provide an indication of levels of business innovation and sales growth in the period six months and twelve months respectively after the completion of the Creative Credits project. Section 4.3 also provides an indication of the scheme’s longer term output additionality, as we also have information on the longer term (three–year) intentions of Creative Credits recipients. We use the information gleaned from the qualitative interviews to highlight some of the mechanisms shaping the outcomes and firms’ longer term intentions. Most of the quantitative results presented in this section are based on what we label the ‘longitudinal sample’, i.e. the group of firms in the treatment and control groups which responded to each of the four surveys covering the Creative Credits project period and the subsequent 12 months. This longitudinal sample comprises 274 firms (117 in the treatment group and 157 in the control group) or 60.8 per cent of all firms which responded to our initial baseline survey. Attrition during the study period – drop out from the sample – occurred for a number of reasons, including firm closure and staff changes within SMEs; in addition, a number of firms were simply unwilling to complete all the surveys. Annex 1 profiles the longitudinal sample in some detail and discusses the problem of attrition and how we have managed it. There were few systematic patterns among firms dropping out of the sample – in other words, firms completing all four surveys appeared to be representative of the wider group of SMEs participating in the scheme. Section 4.4 focuses on the behavioural and network additionality of Creative Credits relating to innovation cooperation – and particularly the likelihood of partnering with creative service providers – in the twelve months following completion of the projects. The findings of the qualitative research again provide deeper insights into the causal mechanisms shaping behavioural and network additionality. Where results from the qualitative data are reported in the form of quotations or conclusions from interviews, each business has been allocated a letter and number to signify SMEs in the treatment group (S), their creative partners (C) and their participation in wave 1 or wave 2 of the experiment. So, for example, W1S1 relates to wave 1, SME 1. Any unattributed quotes come from the ‘free format comments’ boxes in the quantitative surveys and therefore cannot be attributed under the non–disclosure terms of those surveys. Annex 6 presents one–page case studies of three Creative Credits projects. 27 CREATIVE CREDITS A randomized controlled industrial policy experiment 4.2 Network additionality of Creative Credits In this section we exploit the randomized control trial structure of the Creative Credits experiment to consider its network additionality, i.e. by how much did the award of a Creative Credit increase the probability that an SME entered into a new relationship with a creative business? All SMEs were able to access creative partners from the online Gallery and complete their innovation projects, whether or not they had been allocated a Creative Credit. Of the 301 firms in the control group which responded to our baseline survey (Survey 1), 36 firms (11.9 per cent) went ahead anyway with their projects within the scheme’s four to five–month project timeline. Among the group of 150 firms which were assigned Creative Credits, 144 (96.0 per cent) actually commissioned projects. This group of 451 firms, of which 150 were Creative Credit recipients and 301 non–recipients, forms the basis for our estimate of network additionality. The dependent variable in our analysis then takes a value of 1 if a firm commissioned the project within the four– to five–month Creative Credits timescale, and zero otherwise. Table 4.1 reports the results from estimating this simple model using Ordinary Least Squares (OLS). The coefficients have the following interpretation: without a Creative Credit a firm has a probability of 11.9 per cent of commissioning the innovation project (from a creative business with which it has not previously worked). This is (at least)59 84 per cent more likely when a firm receives a Creative Credit.60 This suggests strong network additionality of the receipt of a Creative Credit, with the 90 per cent confidence interval of the additionality lying between 78.2 and 89.8 per cent. This interval includes the estimated additionality of 79.0 per cent from the analysis undertaken for the pilot for the Dutch national innovation vouchers scheme.61 The short–term network additionality of the Creative Credits scheme was strongly supported by the interviews conducted with SMEs immediately following the four to five month window within which the Creative Credits projects were completed, with 15 SMEs reporting that the Creative Credit had brought their innovation forward because of the cost saving (W1S3, W1S6, W1S8, W1S10, W1S12, W1S13, W2S1, W2S2, W2S4, W2S6, W2S7, W2S8, W2S10, W2S11, W2S12). “I’d have waited. I’d have waited until we’d accumulated more money. And then probably programmed it in for sort of the back end of this year. And then launched in probably February next year.” (W1S10) “Maybe it’s allowed us to accelerate things and it’s perhaps made me focus on it rather than put things off. So it’s got things moving quicker than if I’d been left to my own devices.” (W1S8) “It’s something which I guess if we didn’t have the Credit it wouldn’t be the first thing on the top of my head. I wouldn’t say, ‘Oh I’m gonna make a video before doing say conferences’, and actually even now I couldn’t honestly say that, because I think in terms of importance I think probably getting out to conferences is more important at this stage. But, interestingly enough, if we ever get to the point where we were more comfortable and we did have a bit of spare cash, then actually I would think about another video because I do like the media and I think it’s quite powerful”. (W1S3) 28 CREATIVE CREDITS A randomized controlled industrial policy experiment Table 4.1: Simple OLS treatment model regression Number of observations 451 Adjusted R–squared 0.653 Variable Coefficient Std. Err. t–statistic Signif. Creative Credit 0.840 0.028 29.11*** 0.000 Constant term 0.119 0.017 7.18*** 0.000 Notes: Analysis is based on respondents to the initial baseline survey undertaken immediately after the allocation of Creative Credits. *** denotes significance at the 1 per cent level. The OLS estimates of additionality in Table 4.1 implicitly assume that there are no systematic differences between Creative Credits recipients and the control group of firms, other than in their receipt of a Creative Credit. This would of course follow by design of the RCT, but in small samples it is always possible such differences may arise, and there is some evidence that this is indeed the case for the pilot (see Table A1.3). Although the differences are very small, they may conceivably have introduced biases in the estimated coefficient on the treatment term, and it is important to investigate this. To do this, we followed standard econometric procedures in estimating two–stage multivariate Probit models to test the robustness of the OLS results: in stage one, we modelled the probability of a firm receiving a Creative Credit as a function of firm characteristics (Table A4.1); and in stage 2 we estimated the impact of the Creative Credit on the probability of a firm undertaking its creative project (Table A4.2). This analysis suggested that there was no selection bias in the estimates and therefore supports the validity of the simple OLS results reported in Table 4.1.62 4.3 Output additionality of Creative Credits – Innovation and sales impacts Table 4.2 explores whether firms that received Creative Credits were any more likely to have innovated in the period following the end of the Creative Credits projects than those firms that had not received Creative Credits. In the six months following completion of the Creative Credits projects, we saw a marked difference between the treatment and control groups, with firms in the former significantly more likely to have introduced new products or services, more likely to have introduced new to the market products or services and more likely to have introduced new process innovations (Table 4.2, part A). A further six months on, however, the differences were no longer statistically significant (and were slightly negative in the case of process innovations) (Table 4.2, part B). 29 CREATIVE CREDITS A randomized controlled industrial policy experiment These results suggest that there was significant output additionality from the Creative Credits scheme, but that this additionality was short–lived. Similar results are found in the multivariate analysis (Tables A4.4 and A4.5), though there is weak evidence that firms receiving Creative Credits were more likely to have introduced new to market innovations even twelve months out. Table 4.2: Output additionality in terms of the probability of innovation: 6 months and 12 months after the completion of the Creative Credits projects Treatment % firms t–statistic Signif. N Control % firms Product or service innovation 145/105 55.9 72.4 2.740*** 0.007 New to the market innovation 126/92 23.0 35.9 2.089** 0.038 Process innovation 142/105 47.2 63.8 2.618*** 0.009 Control % firms Treatment % firms t–statistic Signif. N Product or service innovation 154/113 63.0 70.8 1.345 0.180 New to the market innovation 135/97 32.6 40.2 1.192 0.235 Process innovation 153/111 51.0 47.7 –0.517 0.606 A. After 6 months B. After 12 months Notes: Table is based on the longitudinal sample. See Annex 1 for details, after six months, firms were asked whether they had produced more or fewer production innovations in the previous six months. After twelve months, firms were again asked about their innovation performance in the previous six months. Differences in response numbers between questions and between 6 and 12 months reflect those respondents not answering particular questions and those selecting the (‘Don’t know’) response. * denotes significance at the 10 per cent level; ** at 5 per cent and *** at the 1 per cent level. We might have expected this pattern of significant, but not sustained, impacts from Creative Credits on innovation to have been echoed in the sales performance of firms, and this is indeed what we found. Table 4.3 summarizes the distribution of sales growth of SMEs in the longitudinal sample receiving Creative Credits and those not, six and twelve months after the completion of the Creative Credits projects. Although this period coincided with a period of deep economic recession, and therefore challenging times for all the SMEs in our study, the comparison of performance between treatment and control group firms is still statistically valid. 30 CREATIVE CREDITS A randomized controlled industrial policy experiment After six months (Survey 3) we see a (weakly) significant difference between the distribution of growth rates of the control and treatment groups, with the award of the Creative Credit lowering the proportion of firms with declining sales (the p value of 0.075 means that there is only a 7.5 per cent probability of incorrectly rejecting the hypothesis that the growth rates of the two groups of firms were identical). After twelve months, although the difference in average growth rates was even bigger – at 3.1 per cent compared with 1.1 per cent after six months – it was no longer statistically significant. These results are confirmed by the multivariate analysis, where although sales growth of those SMEs receiving Creative Credits was higher on average than those in the control group, the differences were not statistically significant (Table A4.6). Table 4.3: Output additionality in terms of sales growth: 6 and 12 months after completion of Creative Credits projects Impacts after 6 months* Impacts after 12 months Control N=146 Treatment All N=107 N=253 Control N=155 Treatment All N=114 N=269 Sales have fallen 20% or more 6.8 5.6 6.3 7.1 7.0 7.1 Sales have fallen by 10–19% 6.8 4.7 5.9 9.0 6.1 7.8 Sales have fallen by 1–9% 9.6 2.8 6.7 10.3 7.9 9.3 Sales have remained similar 28.1 36.4 31.6 29.0 23.7 26.8 Sales have risen by 1–9% 13.0 23.4 17.4 13.5 16.7 14.9 Sales have risen by 10–19% 17.8 13.1 15.8 11.0 21.9 15.6 Sales have risen 20% or more 17.8 14.0 16.2 20.0 16.7 18.6 Average growth rates 6.445 7.53 6.9 4.71 7.81 6.02 Notes: Table is based on the longitudinal sample and responses. See Annex 1 for details. After six months, firms were asked to select a band within which their sales had grown/fallen in the previous six months. After twelve months, firms were again asked to choose a band within which their sales had grown/fallen in the previous six months. For impact after six months, difference in distribution of growth rates between control and treatment groups χ2(6)=11.464, p=0.075; after 12 months χ2(6)=7.701, p=0.261. Average growth rates derived using group means. Differences in response numbers between different questions and between 6 and 12 months reflect those respondents not answering particular questions in the survey and those selecting the (‘Don’t know’) response. * denotes significance at the 10 per cent level; ** at 5 per cent and *** at the 1 per cent level. CREATIVE CREDITS A randomized controlled industrial policy experiment 31 These additional sales impacts do, however, need to be treated with some caution, as they may under–estimate the longer term sales effects of the Creative Credits projects. Consistent with this, Table 4.4 summarises the anticipated duration of firms’ sales benefits at the time of Survey 4, with only 17.6 per cent of firms reporting that they had already by this time received all of the benefits of their Creative Credits project (and a further 5.6 per cent saying that they had experienced no benefits at all). A further 23.1 per cent of firms anticipated obtaining the remaining benefits from their Creative Credits project over the next year, and well over 50 per cent of SMEs said the benefits would fully accrue over the next two years and beyond. Comparing the reported sales growth figures for SMEs which had either obtained all of the benefits of their Creative Credits project, or expected to have done so before Survey 4, with those anticipating longer term benefits suggests some marked differences. Average growth during the year after the Creative Credits project for firms anticipating all the benefits within one year of completion was 7.7 per cent, compared with only 2.0 per cent for firms anticipating longer term benefits from their projects (t=2.141, p=0.035).63 This difference underlies the great heterogeneity in the time profile of benefits enjoyed by different firms, and, in general, the difficulty of making a complete assessment of the growth impacts within the timescale of our study. Table 4.4: Anticipated duration of sales benefits from Creative Credits: treatment firms N % of firms No benefits experienced from the Creative Credits project 6 5.6 You have already got all the benefits 19 17.6 You expect to get all the benefits in the next year 25 23.1 In the next two years 46 42.6 In the next 3–5 years 10 9.3 It will take more than 5 years to get all of the benefits 2 1.9 Total 108 100.0 Notes: Table is based on the longitudinal sample. See Annex 1 for details. To summarise, we find evidence of statistically significant output additionality in the form of more product and process innovations within six months of completing their Creative Credits projects. However, the differences in innovation were not statistically significant twelve months after the projects had been completed. This pattern is broadly mirrored in the observed sales impacts too. Considering the evidence that most of the SMEs still expected further benefits to materialise in the future from their projects, however, it is perhaps best to think of the estimated additional sales benefits as a lower bound for the true figure. 32 CREATIVE CREDITS A randomized controlled industrial policy experiment 4.4 Behavioural additionality Along with their potential to stimulate specific innovations it was also anticipated that Creative Credits might encourage SMEs to become generally ‘more innovative’ firms. Table 4.5 compares the probability that SMEs in the control and treatment groups said they were likely to engage in innovation in a number of different areas over the next three years. There were no significant differences at either six or twelve months following the completion of the Creative Credits projects – a finding that was confirmed in multivariate analysis (Tables A4.7 and A4.8). Table 4.5: Future innovation intentions for control and treatment firms: 6 months and 12 months after completion of Creative Credits projects Control Treatment N Not Likely Quite Likely Very Likely Not Likely Quite Likely Very Likely χ2(2) Signif. Goods or services 248 4.3 35.5 60.3 5.6 27.1 67.3 2.036 0.361 Processes 245 7.9 34.5 57.6 8.5 33.0 58.5 .074 0.964 Strategy 235 15.7 41.8 42.5 12.9 38.6 48.5 .912 0.634 New technologies 204 38.5 36.8 24.8 41.4 42.5 16.1 2.321 0.313 Organization 216 39.5 28.7 31.8 44.8 32.2 23.0 1.984 0.371 Marketing 231 10.4 40.7 48.9 11.5 45.8 42.7 .864 0.649 Goods or services 257 6.7 40.9 52.3 7.4 35.2 57.4 .876 0.645 Processes 260 9.3 42.4 48.3 11.9 36.7 51.4 1.059 0.589 Strategy 249 14.6 45.8 39.6 18.1 38.1 43.8 1.583 0.453 New technologies 206 47.6 29.8 22.6 45.1 32.9 22.0 .224 0.894 Organization 221 49.3 36.6 14.2 47.1 32.2 20.7 1.676 0.433 Marketing 235 18.4 38.2 43.4 15.2 48.5 36.4 2.464 0.292 A. After 6 months B. After 12 months Notes: Table is based on the longitudinal sample. See Annex 1 for details. Differences in response numbers between different questions and between 6 and 12 months reflect those respondents not answering particular questions and those selecting the (‘Don’t know’) response. * denotes significance at the 10 per cent level; ** at 5 per cent and *** at the 1 per cent level. 33 CREATIVE CREDITS A randomized controlled industrial policy experiment One other anticipated outcome from the Creative Credits scheme was its potential to overcome the reticence of SMEs to partner with other businesses, and creative businesses in particular, as part of their innovation strategies. Table 4.6 compares the percentage of innovating firms in the control and treatment groups working with creative service providers and other types of innovation partner six and twelve months after the completion of the Creative Credits projects. Again, there is no evidence of behavioural additionality, as it relates to the collaborative behaviour of innovating firms in the treatment group either 6 or 12 months after the completion of the projects. Table 4.6: Additionality in terms of probability of innovation cooperation: 6 months and 12 months after completion of Creative Credits projects Control % firms Treatment % firms t–statistic Signif. N Other group companies 79/73 27.8 31.5 0.490 0.625 Suppliers 81/74 58.0 52.7 –0.662 0.509 Creative service suppliers 80/75 53.8 58.7 0.613 0.540 Customers 81/74 58.0 55.4 –0.327 0.744 Competitors 79/73 24.1 9.6 –2.429 0.016 Higher Education Institutes 81/75 22.2 20.0 –0.338 0.736 Public Laboratories 81/74 12.3 10.8 –0.297 0.767 Other group companies 93/75 33.3 37.3 0.536 0.593 Suppliers 95/77 52.6 63.6 1.458 0.147 Creative service suppliers 95/77 52.6 64.9 1.637 0.104 Customers 95/77 65.3 71.4 0.864 0.389 Competitors 96/77 24.0 32.5 1.228 0.221 Higher Education Institutes 93/78 29.0 26.9 –0.305 0.761 Public Laboratories 94/76 11.7 18.4 1.204 0.231 A. After 6 months A. After 12 months Notes: Table is based on the longitudinal sample. See Annex 1 for details. Respondents numbers are given as ‘control group/treatment group’. Differences in response numbers between different questions and between 6 and 12 months reflect those respondents not answering particular questions and those selecting the (‘Don’t know’) response. * denotes significance at the 10 per cent level; ** at 5 per cent and *** at the 1 per cent level. 34 CREATIVE CREDITS A randomized controlled industrial policy experiment One further possibility considered in the evaluation was whether firms in the treatment and control groups were intending in the future to partner with external organizations on innovation projects. Specifically, firms were asked whether as part of their innovation activity over the next three years they expected to work with creative businesses. The results in Table 4.7 echo those in Table 4.6: there was no evidence that Creative Credits had stimulated sustained new innovation partnerships between SMEs and creative businesses. Table 4.7: Additionality in terms of intended future cooperation with creative service providers: 6 months and 12 months after completion of Creative Credits projects Control N Treatment Not Likely Quite Likely Very Likely Not Likely Quite Likely Very Likely χ2(2) Signif. 10.1 43.9 46.0 4.0 39.6 56.4 4.421 0.110 16.3 44.2 39.5 12.3 45.3 42.5 .847 0.655 A. After 6 months Creative services 240 A. After 12 months Creative services 253 Notes: Table is based on the longitudinal sample. See Annex 1 for details. Differences in response numbers between different questions and between 6 and 12 months reflect those respondents not answering particular questions and those selecting the (‘Don’t know’) response. * denotes significance between the control and treatment groups at the 10 per cent level; ** at 5 per cent and *** at the 1 per cent level. In our interviews, we probed SMEs for the reasons why their Creative Credits projects had not had changed their intentions to work with creative service providers. These clearly pointed to two considerations. First, for a number of SMEs there was a sense that their Creative Credits project had been based on a ‘transactional’ relationship with their creative service providers. Indeed, fifteen of the nineteen SMEs interviewed at the time of Survey 4 referred to having already worked with other creative businesses prior to applying to the Creative Credits scheme. For these firms, Creative Credits gave them an opportunity to work with a new creative partner, but generated little new organizational learning about partnering with creative businesses in general. As a result, most of the SMEs ended their working relationship with their Creative Credits partner when their projects had completed, consistent with their previous working relationships with creative businesses (W1S7, W1S9, W1S10, W1S12, W2S4, W2S6). The second apparent reason for the lack of sustained network additionality suggested by some SMEs in the interviews was a sense of dissatisfaction on their part with their creative partners (W1S7, W1S8, W1S9, W1S11, W2S2, W2S4). For several SMEs, the dissatisfaction related to the attitude of their creative partner: “I can’t say that we learnt much in terms of what they did. It still felt like we were going cap in hand for any amends or any corrections or anything like that, it did feel that they were in a sense had bigger fish to fry than our project” (W2S4). One SME felt they “were not big enough for the website people. They always want to try and push you into something that 35 CREATIVE CREDITS A randomized controlled industrial policy experiment is far bigger than you really are I didn’t actually learn anything from them” (W2S2). Another spoke of how, “I sort of trusted in them to do it all really... I wish I’d sort of prepared for it better, then I probably would have got more out of them...or certainly it would have gone in the direction I wanted it to go in” (W1S7). Such comments were validated in the interviews with creative service providers themselves which tended also to emphasise communication difficulties, different interpretations of what the project objectives were and how great a priority the different partners had attached to the projects.64 So, one creative service provider commented of their SME partner that “they didn’t seem to want to go in some of the directions we were suggesting to them” (W2C12). Others felt that the SMEs had very set ideas about what they wanted, seeing the Creative Credits project as a ‘quick fix’ and treating their creative services provider as if they were selling a fixed product, rather than developing a working relationship in which “there is a trust that can build” (W1C7). In other cases, SMEs seemingly did not provide the information requested by their creative partner, thus impeding progress on the commission, “We had to hold their hand all the way through it” (W2C12). This may have been due to the SME undervaluing what the creative company could offer because it was seen as “a one–off hit where they got something for free…They may not need what you’re selling” (W1C7). By contrast, successful projects were distinguished by good communication between partners leading to a shared understanding of the project objectives. For example, “Because my relationship has grown with them on a personal level … they’ll back me up or help me out … in what I’m trying to do” (W1S5); and “I feel I can work with them on projects and I’ve got a better perception of what they can contribute and I can value that contribution” (W1S13). The same sentiments applied even where projects were seen as transactional, as mentioned earlier: “rather than just give them a brief, I think we will sit down with them and create the brief based on some of their ideas as well” (W2S8). This good communication also appeared to be related to the likelihood of sustained benefits from the project. Of the nine SMEs interviewed in Stage 4 who reported continuing benefits, seven specifically made positive comments about communication and shared understanding, while only two made negative comments. A number of SMEs, (W1S1, W1S4, W1S5, W1S9, W1S10, W1S11, W1S12, W1S13, W2S6, W2S7, W2S11) suggested that an element of brokerage in the scheme from Nesta would have helped to reconcile the tensions with their creative partner. These varied from the passive, such as changes in the design of the online Creatives Gallery, to the more active, including Nesta’s involvement in the matching process and the provision of more networking opportunities with other SMEs participating in the scheme. 36 CREATIVE CREDITS A randomized controlled industrial policy experiment Section 5: IMPLICATIONS FOR POLICY DESIGN AND EVALUATION Key points: Very little policy supporting business is subjected to rigorous evaluation. As a result we have no reliable way of knowing if large sums of public money are being wasted. Creative Credits is an important innovation in itself: it helps SMEs more competitive by connecting them with creative businesses. But the research method used may be even more important, since it not only uses randomization to establish the scheme’s additional impact, but also links that to longitudinal data collection which is vital in helping to assess the longer term effectiveness of different policy tools. This is an approach that should be applied much more widely. At a time when public resources are scarce it’s more important than ever that we find out what works and what doesn’t – and that principle should apply as much to business support as it does to programmes in healthcare or schools. This research has demonstrated that Creative Credits significantly boosted innovation and sales growth in SMEs in the six months following completion of their Creative Credits projects, but that these impacts were no longer statistically significant by the time a further six months had passed. The RCT+ evaluation approach that we adopted combining three elements – randomized allocation of Creative Credits, longitudinal data collection and use of mixed methods – has proven to be a powerful methodology that we recommend is used more widely in developing innovation support schemes. We note that a number of cities across Europe are now adopting their own Creative Credits–style schemes, inspired by the successes of the Manchester pilot. In the UK, the Government has shown interest in innovation voucher schemes more generally, including a new Growth Vouchers scheme announced in the 2013 Budget. We advise strongly that these schemes should adhere, wherever possible, to a randomized allocation as they are rolled out, in order to establish a cumulative evidence base for the effectiveness of their progamme design features. Examination of the RCT dimension of the Creative Credits pilot confirms its high degree of internal validity, with strong evidence of both network and output additionality. But, arguably, it also demonstrates its limited external validity, with significant differences, for example, in the characteristics of SMEs applying to Creative Credits and the general population of SMEs in the Manchester City Region (This is only arguably, because the target for such schemes may not in fact be the wider population of SMEs in Manchester). As with all pilots – controlled experiments or otherwise – this suggests that drawing precise inferences from the Creative Credits experiment for other firms and regions should always be done with care. 37 CREATIVE CREDITS A randomized controlled industrial policy experiment 5.1 Introduction In this section, we consider the implications of the findings of the Creative Credits experiment for the design and evaluation of SME support initiatives. Section 5.2 focuses on the key issues arising when implementing the three planks of the RCT+ approach – random assignment, longitudinal data and the use of mixed methods. Section 5.3 considers the implications for future business–to–business voucher schemes inspired by Creative Credits, such as those currently being piloted across Europe. Our emphasis here is on the positive impacts of Creative Credits on innovation and the approaches which might be explored to strengthen the longer term benefits of the partnerships enabled by Creative Credits. 5.2 Implementing the RCT+ evaluation approach As outlined in Section 3, the RCT+ approach we have used to evaluate Creative Credits combines three elements: randomized assignment of firms to the treatment and control groups, longitudinal data collection, and the use of mixed methods. In this section we discuss each of these elements, and comment on their contribution to the evaluation. Randomization The advantages of randomization are self–evident. It has enabled us to evaluate the additional impacts of the scheme with a much greater degree of statistical confidence than is typically the case with SME support measures. This no doubt helps explain the renewed interest that governments in countries like the UK and US have shown in RCTs. https:// www.gov.uk/government/publications/what-works-evidence-centres-for-social-policy In particular, random allocation of the Creative Credits has given the results of the experiment a high degree of internal validity: the SMEs awarded Creative Credits were broadly representative of the group that had applied, meaning that there were few if any systematic differences between the treatment and control groups, other than in their award of a Creative Credit (Table A1.2). This was actually far from guaranteed. There were three particular challenges. First, respondents in both groups were contacted regularly by telephone and email by members of the research team to encourage participation in the various surveys, which may conceivably have induced self–selection biases if there were differences between those firms that responded and those that did not. Second, respondents in the control group, who were inevitably less motivated to participate in the evaluation of Creative Credits, were provided with small cash incentives of £30 to encourage their participation, which again may have induced self–selection biases.65 Third, the more personal relationships established with SMEs that had participated in the qualitative interviews may arguably have distorted their responses to the surveys. Yet, despite all these challenges, the similarity of the initial characteristics of the treatment and control groups achieved through randomization (Table A1.2), and also the sustained representativeness of the longitudinal sample of respondents for all four surveys (Table A1.3) both support the internal validity of the experiment. The external validity of an experiment relates to the generalizability of its results to the wider population of firms. The first question therefore is whether the group of Creative Credits applicants were typical of the wider population of SMEs in the Manchester City Region (MCR) and therefore whether the results of the experiment were generalizable at this level. 38 CREATIVE CREDITS A randomized controlled industrial policy experiment In fact, comparing the characteristics of those SMEs applying to Creative Credits with those that did not suggests that the former differed from the general MCR population of SMEs in several different ways:66 they were more focused on innovation than non– applicants, at least as indicated by our surveys, more likely to have previously worked with external partners, and to have higher internal skill levels than non–applicants (Annex 2). This suggests that, as with experiments more generally, inferences about the likely impact of the scheme on the wider population (including SMEs in the Manchester City Region), should always be drawn with caution.67 One other element of validity relates to the particular economic conditions which provided the backdrop to the Creative Credits experiment. That is, the treatments took place while the UK economy, including the Manchester City Region, was in deep recession. One SME said of their Creative Credits project that “it was useful at the time, but because of the recession and all the rest of it, I don’t believe we’ve really seen the benefit yet, but I suspect we will do” (W1S7). Another SME could not fully implement the outcome of their project because it had not been able to secure bank funding, saying that “We are working our guts out, I have to try and raise finance” (W1S3). Other SMEs highlighted issues with public procurement with one SME selling into the health service saying that “there are not many big jobs around” and “one of my competitors has just made 50 per cent of people redundant” (W1S4). Even where market conditions were perceived as difficult, however, some SMEs did highlight their gains from the Creative Credits projects: “… commerce in the last months has been affected by the recession so it’s difficult to know whether the Creative Credit has helped deflect further shrinkage in trade than would have been suffered and what impact it would have had in normal circumstances. All in all though, a fabulous opportunity for small business that had a big impact and was delivered in a smooth and helpful manner”. (Survey 4 SME, comments box). Although the findings’ internal validity were arguably robust to the impact of recession – the treatment and control group of SMEs had been exposed to the same macroeconomic conditions – it is nonetheless possible that the severity of the recession may have acted to ‘swamp’ any differences between the SMEs receiving Creative Credits and those that did not. Reflecting on the impact of their Creative Credit, a number of firms in the treatment group clearly felt that the recession had impacted on their ability to benefit fully from the scheme. One SME commented: “We are still experiencing a depressed market due to the recession, but our short term future looks better and I expect to see significant up–turn in 2012. As such, our new products have not had an easy start in life, but we are happy with their development and remain optimistic for their future” (Survey 4 SME, comments box). Still another consideration which may raise questions about the experiment’s validity is if the innovation activities of SMEs in the control group were positively impacted simply by applying to Creative Credits and/or being asked to participate in the research68 (these are called ‘Hawthorne’ effects in the literature69). Such effects might have biased upwards the performance of SMEs in the control group leading to an under–estimate of the impacts of Creative Credits. 39 CREATIVE CREDITS A randomized controlled industrial policy experiment To shed at least some suggestive light on this possibility, we tested if there were any differences in the use of other external sources of support for innovation between the control and treatment group of firms following the completion of the Creative Credits projects; no such differences were identified.70 Longitudinal data collection One of the key benefits of the longitudinal data collection is that we have been able to examine the evolution of the scheme’s impacts over time. This has revealed one of the most important findings of the experiment, namely that there were significant impacts on innovation and sales growth six months after the end of the Creative Credits projects, but that the impacts were not significant after a further six months (Tables 4.2 and 4.3). These results highlight the limitations of traditional one–off policy evaluations: if we had only measured the scheme’s impacts after six months we would have had a skewed view of the scheme’s longer term impacts. We have shown that adopting a longitudinal approach to evaluating business support schemes insures against potentially misleading inference and captures the sustainability of outcomes.71 This argument also highlights a significant limitation of our own analysis, however. In particular, twelve months after the end of the treatment, when asked about whether they had derived all of the anticipated sales benefits from the Creative Credit, only 23.2 per cent of firms responding said that they had, including the 5.6 per cent who replied they had achieved no benefits. The remaining 76.8 per cent suggested that additional benefits would accrue over subsequent years (Table 4.4). The anticipated duration of these potential benefits, and the uncertainty which inevitably attaches to firms’ responses, suggests the potential value of extending our analysis beyond a 12–month impact period to obtain more accurate outcome data. The trade–off here is clear, however: in the obvious delays in obtaining additional research results, and the challenges of collecting survey data from SMEs relating to a support measure which fades further away in their memories over time. More broadly, and particularly where interventions are more significant in scale than those considered here, this suggests the potential value of a longitudinal approach covering relatively long impact periods, of perhaps 2–3 years, or even longer in some cases. Combining the findings at 6 months and 12 months after project completion, with the fact that the great majority of SMEs were still expecting to see benefits from their Creative Credits projects is nonetheless important, as it suggests that the estimated commercial benefits are likely to be highly sensitive to when they are measured. Mixed methods The quantitative survey findings that Creative Credits had strong network and output additionality, at least out to six months, was strongly supported by the qualitative evidence gathered through our interviews and the Journey Making workshop. Reassuringly, triangulation of qualitative and quantitative findings showed a large measure of agreement. For example, 19 SMEs were interviewed in Stage 4. If we compare their qualitative reports of sales benefits with their quantitative survey responses in Survey 4, we find agreement in 17 out of 18 cases. One SME had not completed the quantitative survey: in the interview, it revealed that it had experienced sustained sales growth, but it did not attribute this to the Creative Credits project. Interestingly, this SME’s creative partner pointed out in its Stage 4 interview that the Creative Credits project had first introduced 40 CREATIVE CREDITS A randomized controlled industrial policy experiment this SME to the use of social media, and the SME had cited use of social media as the reason for its growth in sales. It had attributed this success to the skills of one of its own staff members, subsequent to the Creative Credits project. This case is an example of the additional detail that the mixed methods approach makes available. In a good many cases – unsurprisingly, given the web–based nature of the majority of the projects – SMEs felt that their creative partners had portrayed them in a more positive and engaging way online to their customers. One firm commented: “certainly it has improved the image and it’s improved the response we’re getting, people are getting in touch with us which they weren’t before” (W1S13). Another SME commented: “it’s changed the way that people perceive [us] because instead of looking like a clacky old website that I was slightly embarrassed by, it looked like I wanted it to look, because the person that designed it originally, the two guys that were involved from the very start five years ago, and did the update two and a half years ago, were not the ones I should have used. I should have been using a professional organization” (W1S10) For other SMEs, the Creative Credits projects had resulted in a more unified marketing message: “The main impact we’ve had is the little mascot man, the little robot, because he also appears on the hard copy material that we produce as well” (W2S8). Where the qualitative work really came into its own was in helping to provide a deeper understanding of the quantitative results. First, the qualitative analysis uncovered some of the communication and co–ordination issues which had arisen between the SMEs and their creative partners, and which helped to explain the dissipation of the network and behavioural impacts of the creative projects twelve months out. One consequence of these communication and coordination difficulties – and the tensions they created in some of the SME–creative relationships – was a lack of organizational learning on the part of some SMEs, which was again identified in the interviews. Second, the qualitative analysis also helped to provide some suggestions as to how these communication and coordination issues might be addressed in the design of future versions of the Creative Credits scheme. Economising on these (costly) brokerage elements had been one of the conscious design decisions made by Nesta and its funding partners when developing the pilot. Their goal had been to test the additional impact on innovation of the vouchers mechanism, not additional services provided to broker the relationship between SMEs and their creative suppliers. Yet, a number of SMEs made suggestions as to how Nesta could perhaps have provided a more active, yet still light touch, matching and brokerage function between SMEs and their creative partners. As one SME put it (W1S9): “…we would have got loads more value if they went down a completely different route that was more online focused. Now, I think I would have made better decisions if I had have had almost somebody from NESTA to bounce ideas off first rather than me ... because it was quite limited and I think, perhaps, even if there was some sort of almost a network, an open day so – that everyone that was accepted onto the programme could actually go and meet a few of these other companies … I think that would have been a lot better. It would have meant I would get heaps more out of the project”. 41 CREATIVE CREDITS A randomized controlled industrial policy experiment Another SME posed the challenge of negotiating the Creatives Gallery as follows: “There’s just so many of them, so you would almost not know where to start. So, I don’t know if there’s anything else that could be done to make that selection process a little bit easier or help guide people if there was some kind of filtering system”. Other firms highlighted the value of more networking events and the potential value of providing SMEs with examplar case studies (W1S13). 5.3 Implications for Innovation Voucher Schemes We have learned from the surveys that the Creative Credits mechanism was a highly effective way of connecting SMEs with creative suppliers with which they had not previously worked. And that the creative projects appeared to have successfully boosted innovation and sales growth in SMEs in the near term, although these benefits appeared to have attenuated by the time twelve months had passed. Even then, most SMEs said that they expected further impacts from their projects on sales growth to be felt in future. The interview evidence corroborated these findings, and also shed some light on why there was no network or behavioural additionality after twelve months. In particular, the engagement with creative businesses appears in many cases to have been viewed by SMEs as a transactional relationship; Creative Credits does not seem to have succeeded in nudging SMEs into making persistent changes in their behaviour, at least insofar as it relates to innovation. Arguably, the £5,000 face value of the Creative Credits (small, even by the standards of SMEs) may not have been large enough to ‘nudge’ SMEs into the type of behavioural changes envisaged in the scheme’s logic model. We note that in other EU cities and regions which are piloting B2B voucher schemes drawing inspiration from Creative Credits (eg Strasbourg, Salzburg, Sligo and the Basque Country), the value of innovation vouchers appears also to be small, and future implementations of such measures may want to introduce an element of random variation in the value of the voucher to investigate its importance. Aside from the value of the Creative Credits themselves, a number of SMEs were also conscious of having made a poor choice of creative partner. The suggestion is that SMEs might have benefitted more from Creative Credits if they had been advised by Nesta when selecting their creative partner. In other cases, once projects had started, we might speculate that some brokerage aimed at establishing that both the SMEs and their creative partners still had a shared understanding of what the project was trying to achieve might have reduced the number of unsuccessful projects. Future implementations of innovation voucher schemes would be well advised to adhere as closely as possible to an RCT+ approach, in order to establish a cumulative evidence base for the effectiveness of such schemes. The high levels of interest there has been in Creative Credits in other regions, and the interest in innovation vouchers mechanisms in general in countries like the UK, presents an excellent opportunity for testing further where the obstacles to sustained innovation impacts lie, and how the scheme’s design can be refined to address these.72 42 CREATIVE CREDITS A randomized controlled industrial policy experiment Despite the increasing maturity – and widespread adoption – of innovation voucher schemes, there remains relatively little robust evaluation of their effectiveness in the public domain. The evidence which is available suggests, however, that the profile of impacts and outcomes from Creative Credits is similar to that identified in other voucher studies. The first point of similarity relates to the popularity of the scheme with SMEs. Creative Credits was over–subscribed by a factor of five. In the original 2004 pilot programme in the Netherlands, for example, there were 1,044 applications for 100 vouchers.73 Likewise the first three waves of the NWDA voucher scheme attracted 727 applications for 500 vouchers.74 The second point of similarity relates to network additionality and the strong catalytic influence of Creative Credits on the formation of new innovation partnerships with creative service providers. Creative Credits generated significant short–term network additionality, with the award of a Creative Credit increasing the likelihood that firms would undertake their innovation project by 84 per cent. As noted in Section 4 this level of short–term network additionality is marginally higher than that (79.0 per cent) achieved in the pilot innovation voucher scheme in the Netherlands.75 Evaluations of the Swiss and Austrian innovation voucher schemes suggest similarly high levels of network additionality.76 The third point of similarity between the results of Creative Credits and other innovation voucher schemes relates to the weakness of the longer–term behavioural outcomes from the scheme. A year after the end of the Creative Credits projects, we were unable to identify statistically significant differences in SMEs’ future intentions to collaborate with creative firms or undertake innovation. This echoes the findings in the Dutch innovation vouchers scheme, where follow–up with participating SMEs after 18 months had suggested no evidence of persistent effects, either in the formation of new partnerships or in the development of new products or processes.77 One possible reason for the lack of longer term effects suggested in relation to the Austrian and Swiss innovation voucher evaluations, was the small size of many voucher recipients and the difficulty which they had in establishing follow–up partnerships. This is consistent with a key suggestion from our qualitative research, namely that the benefits might be more sustained if some form of brokerage can be provided which supports the relationship between SMEs and their creative suppliers. The absence of an existing rigorous evidence base for innovation voucher schemes is indicative of a much wider problem. Which is that very little policy supporting business is subject to robust evaluation. As a result we have no reliable way of knowing if large sums of public money are being wasted. Creative Credits - the focus of this report - is a policy innovation in itself, in using the innovative strengths of creative businesses to enhance innovation in SMEs. But the research method we have used may be even more important, in using randomization to establish the scheme’s additionality and longitudinal data to identify its long-term impact. We believe this approach should be used much more widely. At a time when public resources are scarce it’s more important than ever that we find out what works and what doesn’t - and that principle should apply as much to business support as it does to programmes in healthcare or schools where the Government has so far shown much greater interest.78 43 CREATIVE CREDITS A randomized controlled industrial policy experiment Annex 1: IMPLEMENTATION AND EVALUATION A1.1 Implementation Creative Credits was piloted as a regional experiment in the Manchester City Region (MCR). The Manchester Independent Economic Review (MIER) had previously investigated creativity and innovation in the MCR and found that large numbers of creative firms, while well connected to firms outside the City Region, were poorly integrated into local supply chain networks.79 The MIER had therefore argued that there may be large and immediate payoffs to innovation if creative businesses could be better integrated into its supply chain networks. In the MCR, Creative Credits operated alongside the North West Development Agency’s own established Innovation Voucher scheme which, like traditional innovation voucher schemes, focused primarily on knowledge transfer between universities and SMEs. 150 Creative Credits were available within the Manchester Pilot, with an intention to distribute these equally between two waves. The first wave opened for applications in September 2009 and the second in February 2010. The scheme was promoted and marketed through a number of channels including a media public relations campaign, business networks and direct telemarketing to potential applicants. In promoting and advertising the scheme, care was taken to minimise self–selection biases. For example, the telesales companies who were telemarketing the scheme were encouraged to use a strictly random method in identifying which SMEs to call. More than two thousand firms made some form of enquiry about the scheme over the two waves. Online applications from SMEs were checked for eligibility by a Nesta project manager before firms were randomly assigned to the ‘treatment’ group. The barriers to entry were low: Creative Credits was open to SMEs in almost any sector of the economy with the exceptions of primary industries and the creative industries. The eligibility criteria for Creative Credits had a number of further dimensions: • Geographical coverage – SMEs and creative firms had to have their main office located in either the City of Manchester, the City of Salford, Stockport, Tameside and Trafford (Greater Manchester South), Bolton, Bury, Oldham, Rochdale and Wigan (Greater Manchester North), Congleton, Macclesfield, Vale Royal or Warrington; • Size range – SMEs and creative firms had to have fewer than 250 employees and turnover of less than £46 million at the time of application; • Legal status – both SMEs and creative firms had to be either limited liability companies, limited liability partnerships, general partnerships (added in Wave Two) or industrial or provident societies; • VAT–registered – SME applicant firms had to be registered for VAT. 44 CREATIVE CREDITS A randomized controlled industrial policy experiment SMEs awarded a Creative Credit in the first wave were required to submit their final project proposal for approval by late November 2009 (Wave 2, April 2010) together with the name of their ‘servicer’ – i.e. the creative business who they had chosen to service their Creative Credit. (Other creative companies on the Gallery who were not selected are referred to in this report as ‘non–servicers’).80 Project eligibility criteria were changed slightly between Waves 1 and 2 to tighten the definitions of ‘innovation projects’. SME– creative partnerships were then given five months to complete their project and the SME could claim the Creative Credit once they had been invoiced by their creative partner. SMEs in the first wave were required to complete their project by end–March 2010 and those in the second wave by mid–September 2010. A1.2 Evaluation A strictly random allocation was used to avoid any systematic bias in the allocation of Creative Credits and provide a more robust indication of the extent of additionality of the instrument. The approach had previously been used to evaluate the pilot phase of the Dutch national innovation voucher scheme.81 Mixed methods and a longitudinal data strategy (four surveys and a rolling programme of qualitative interviews) were used to highlight temporal patterns in outputs and outcomes and provide deeper insights into causal mechanisms. Figure A1.1 illustrates the timeline for one of the two waves of Creative Credits in the Manchester City Region pilot and the pattern of surveys and interviews. The four surveys of SMEs in the treatment and control groups were as follows: • Survey 1 – a baseline survey undertaken at the time of the award of the Creative Credits and covering firm characteristics along with information about prior innovation and cooperation; • Survey 2 – undertaken around six months after Survey 1 at a point just after the SME had completed its Creative Credits project. Questions related to the short–term impacts of the Creative Credits project and its learning effects. • Survey 3 – six months later related to firms’ innovation activities in the six months after the Creative Credits project along with their patterns of cooperation. • Survey 4 – undertaken around a year after the end of the Creative Credits project related to the perceived impact of the project and outcomes at that point. 45 CREATIVE CREDITS A randomized controlled industrial policy experiment Figure A1.1: Timeline for Creative Credits projects and their evaluation Project Timeline Qualitative Analysis Project Approval Partnership Agreed 1 month Survey 1 Treatment and control groups Project undertaken 5 months Sign-off Impact period 1 6 months Impact period 2 6 months Survey 1 Survey 2 Survey 3 Survey 4 Creative servicers Treatment and control groups and creative servicers Treatment and control groups Treatment and control groups Qualitative Analysis Qualitative Data Collection The sample consisted of 25 SMEs and their 17 Creative Servicers, with 13 of the pairings from the first wave and 12 from the second. With the respondents’ permission, the interviews were recorded and subsequently transcribed for analysis. • In Stage 1, semi–structured interviews were conducted with 25 SMEs and 16 Creative Services between March and August 2010. (One creative service opted not to participate in the first round.) • In Stage 2, telephone interviews were conducted with 22 SMEs and 14 Creative Services between November 2010 and January 2011. • In Stage 3, 7 SMEs from the qualitative sample participated in two ‘Journey Making’ workshops that took place on 23rd March 2011. • In Stage 4, semi–structured interviews were conducted with 19 SMEs and 8 Creative Servicers between June and October 2011. Of the 19 SMEs interviewed at Stage 4, 11 intended to continue working with their paired creative servicer in the future and 8 did not. The initial baseline survey (Survey 1) was conducted immediately after the award of the Creative Credits. Taking figures for both Wave 1 and Wave 2 together, 451 responses were received to the baseline survey (150 treatment, 301 control). Attrition was evident in each survey thereafter, with response rates given in Table A1.1 and Figure A1.1. As Figure A1.1 suggests, attrition was, unsurprisingly, worse among the control group as these firms had relatively little commitment to the Creative Credits initiative with respondent numbers by Survey 4 falling to 157, 52.2 per cent of those firms initially responding to Survey 1 (despite the use of cash incentives). In the treatment group, attrition was less severe, with 78.0 per cent of Survey 1 respondents also responding to Survey 4. Little difference was evident in the attrition rates of the control group between the two waves of Creative Credits, but attrition was slightly higher among the Wave 2 recipients of Creative Credits. CREATIVE CREDITS A randomized controlled industrial policy experiment 46 Table A1.1: Survey response and attrition by survey, wave and recipient Survey Survey 1 No. 2 No. 3 No. 4 No. 1 % 2 % 3 % 4 % Treatment 150 136 129 117 100.0 90.7 86.0 78.0 Wave 1 70 65 65 58 100.0 92.9 92.9 82.9 Wave 2 80 71 64 59 100.0 88.8 80.0 73.8 Control 301 247 175 157 100.0 82.1 58.1 52.2 Wave 1 153 119 89 79 100.0 77.8 58.2 51.6 Wave 2 148 128 86 78 100.0 86.5 58.1 52.7 Total 451 383 304 274 100.0 84.9 67.4 60.8 Wave 1 223 184 154 137 100.0 82.5 69.1 61.4 Wave 2 228 199 150 137 100.0 87.3 65.8 60.1 During the course of the study, considerable efforts were made to reduce attrition and ensure adequate response during the later surveys. Surveys were conducted online using Survey Monkey with firms being contacted initially by email and then (often repeatedly) by telephone. Small cash incentives (£30) were also paid to firms in the control group to encourage their continued participation in the project. In Survey 2, for example, 167 of the 247 control businesses responding to the survey were incentivized in this way. 47 CREATIVE CREDITS A randomized controlled industrial policy experiment Figure A1.1: Survey responses and attrition by survey and type of respondent 100% 90% 80% 70% Percentage of respondents to Survey 1 60% 50% 40% 30% 20% 1 2 3 4 Survey Treatment group All firms Control group Given the attrition in the proportion of respondents to each of the four surveys, it is of considerable interest to consider whether the longitudinal sample – i.e. those firms which responded to all four surveys – was typical of the whole group of initial respondents. Table A1.2 therefore compares the characteristics at the time of Survey 1 of firms responding to Survey 1 (and possibly also Surveys 2 and/or 3) but not Survey 4 and firms responding to the whole series of four surveys. Respondent numbers are given in the Table as well as t–statistics for the equality of means between the two groups. Reassuringly, for both the treatment and control groups, the basic characteristics and innovative behaviours of the longitudinal sample and firms dropping out of data collection were not statistically different (though for the control group there is weak evidence that firms in the longitudinal sample were significantly more likely to be engaging in new to the market innovation). 48 CREATIVE CREDITS A randomized controlled industrial policy experiment Table A1.2: Characteristics of all respondents and the longitudinal sample N Responded to Survey 1 not 4 Responded to all Four Surveys t–statistic Signif. Limited company 117/33 84.8 90.6 0.83 0.41 Family business 68/17 41.2 57.4 1.18 0.25 Exporting firm 116/33 30.3 39.7 1.00 0.32 Internal R&D 113/31 54.8 69.9 1.50 0.14 External R&D 114/28 25.0 21.9 –0.33 0.74 Innovation 113/32 78.1 79.6 0.18 0.86 Radical innovation 83/20 55.0 55.4 0.03 0.97 Limited company 157/144 91.0 94.9 1.32 0.19 Family business 78/74 36.5 46.2 1.21 0.23 Exporting firm 150/134 37.3 31.3 –1.06 0.29 Internal R&D 151/138 75.4 79.5 0.83 0.41 External R&D 150/138 32.6 36.0 0.60 0.55 Innovation 148/135 74.8 75.0 0.04 0.97 Radical innovation 106/98 50.0 62.3 1.77* 0.08 A. Treatment B. Control Notes: Sample numbers in first column relate to respondents to Survey 1 and 4/Survey 1 and not Survey 4. Differences in response numbers reflect those respondents not answering particular questions and those selecting the (‘Don’t know’) response. * denotes significance at the 10 per cent level; ** at 5 per cent and *** at the 1 per cent level. One other comparison we consider relates to the starting characteristics of the control and treatment groups. As allocation between these groups was the result of a lottery, we would expect no systematic differences between the characteristics of the two groups, other than in whether or not they received a Creative Credit. But it is important to test for this when there are small samples of firms. We do this in two ways: by comparing the characteristics of the control and treatment firms responding to Survey 1; and, restricting this comparison to the longitudinal sample of SMEs (Table A1.3). 49 CREATIVE CREDITS A randomized controlled industrial policy experiment Based on the group of Survey 1 respondents, we do in fact see three significant differences between respondents in the treatment and control groups: treatment firms were more likely to be family firms, and were less likely to be engaged in internal and external R&D (Table A1.3 Part A). For the longitudinal sample, we see a similar picture in terms of firms’ R&D activity (Table A1.3 Part B). Table A1.3: Characteristics of all treatment and control firms–all survey 1 respondents and longitudinal sample. N Treatment Control t–statistic Signif. A. Whole group of respondents Limited company 150/301 89.33 93.02 –1.26 0.21 Family business 85/152 54.12 41.45 1.88* 0.06 Exporting firm 149/284 37.58 34.15 .70 0.48 Internal R&D 144/289 66.67 77.51 –2.33** 0.02 External R&D 142/288 22.54 34.38 –2.63*** 0.01 Innovation 145/283 79.31 74.91 1.04 0.30 Radical innovation 103/204 55.34 56.37 –.17 0.86 B. Longitudinal sample Limited company 117/157 90.60 94.90 –1.33 0.18 Family business 68/78 57.35 46.15 1.35 0.18 Exporting firm 116/150 39.66 31.33 1.40 0.16 Internal R&D 113/151 69.91 79.47 –1.76* 0.08 External R&D 114/150 21.93 36.00 –2.54*** 0.01 Innovation 113/148 79.65 75.00 .89 0.37 Radical innovation 83/106 55.42 62.26 –.94 0.35 Notes: Sample numbers in first column relate to Treatment/Control. Differences in response numbers reflect those respondents not answering particular questions and those selecting the (‘Don’t know’) response. * denotes significance at the 10 per cent level; ** at 5 per cent and *** at the 1 per cent level. 50 CREATIVE CREDITS A randomized controlled industrial policy experiment As the treatment group was less likely to have engaged in R&D than the control group, and as R&D should be positively linked to innovation, we might anticipate that future innovation is also lower among the treatment group. Other things equal, comparisons of treatment and control groups are therefore, if anything, likely to understate the true impact of the Creative Credits scheme. A1.3 The qualitative research process Data analysis in the qualitative elements of the study was guided by a grounded theory methodology.82 Within this methodology, substantive concepts emerged from the data following the constant comparison of questionnaires, interviews and observations aligned to the research questions. Abstraction from these data was done using an emergent approach where the data were interrogated for themes and trends. The identification of themes from each round of analysis was then used to direct and focus the next round of data collection providing theoretical sampling. The longitudinal aspect of the qualitative interviews, informed by the surveys, provided opportunities to confront the emerging theory with further data, and move towards the theoretical saturation that the grounded theory method requires. Our justification for using a mixed–methods approach was that it enabled triangulation, complementarity and facilitation of the research findings.83 By triangulation, we mean the corroboration and cross–checking of results between the RCT and the qualitative research. For example, by Stage 4 of the evaluation, both the qualitative and quantitative findings pointed to the transactional nature of the impact on innovations in many SMEs. Two types of triangulation, by data and by investigator,84 supported the emerging findings by corroborating amongst different sources of data.85 Data triangulation was applied to the different sources of data collected from the SMEs and the Creative Servicers. This was achieved by looking for corroboration between what each SME said in their interviews compared with what the Creative Servicers had reported. When conflict in the data arose, attention was given to establish why there was a disconnect between what the parties were saying. In the investigator triangulation, the results from analyzing the interview transcripts were compared between two researchers to ensure themes being identified were not corrupted by investigator bias.86 Complementarity allowed us to draw on insights from the qualitative work that reinforced the quantitative findings as these emerged. Facilitation was used in the ‘Journey Making’ focus group to support further development of the surveys to ask respondents what they thought that we should be probing in the survey questions. The grounded theory method provided us with the techniques to develop new theory through inductive enquiry. In–depth interviews allowed us to explore perceptions and meanings, and to probe firms in more detail than would have been possible in quantitative surveys alone.87 Topics covered included: company background, strategy, the nature of the Creative Credits project, the experience of working with a creative company, communication, innovation, creativity, learning and Intellectual Property. All respondents agreed to be 51 CREATIVE CREDITS A randomized controlled industrial policy experiment taped and their interviews were transcribed. This allowed us to analyse the data in its totality several times after the interviews.88 In our data analysis we combined a manual approach and a software–supported approach using NVivo 7.0. We initially conducted some content analysis by coding for predefined topics.89 Moreover, our use of grounded theory enabled codes to ‘emerge’ from the data, which helped prevent the researchers from missing issues of importance through having a predefined structure. A concern with NVivo is that the way it structures analysis could lead to imposing fixed hierarchical conceptualisations on the data, which may not be appropriate for structuring the analysis.90 Whilst the NVivo 7.0 software was used because of its data management efficiency, the researchers immersed themselves in the transcripts and manually coded prior to comparing the coding to check if anything had been missed through this hierarchical structuring process. This helps to address Bryman’s91 critique of grounded theory that coding involves taking small fragments of text from the data which may lead to the loss of contextual information. Through a process of open coding, the researchers went through each transcript to identify concepts and properties. Here, anything that appeared relevant (e.g. concepts such as innovation, creativity, strategy, processes, communication, creative project experiences/issues, intellectual property, etc.) was coded into common categories. By way of illustration, anything relating to scheduling problems was coded under the heading ‘time issues’ and manually as ‘time lag in delivery’. In a separate stage of axial coding, the higher–level categories were built from groups of several categories from the open coding. This enabled relationships between the categories to emerge. For example, the ‘time issues’ code (from above) was grouped under a high–level category called ‘project problems’. Through this process, for example, the code identified as ‘time lag in delivery’ was related to ‘breakdown in trust’. In the final stage, selective coding, categories were refined until clear relationships between them were identified, leading to the development of a theory about the data. This stage merged similar/overlapping categories together and removed duplication. This led to the production of a refined tree structure of categories, which enabled the identification of important themes. For example, the relationship category of valuing creativity was selected as ‘creativity transfer’. 52 CREATIVE CREDITS A randomized controlled industrial policy experiment Annex 2: PROFILING CREATIVE CREDITS APPLICANTS A2.1 Introduction The external validity of the Creative Credits experiment depends in part on the representativeness of the control and treatment samples compared with the target population. To compare Creative Credits applicants to, for example, the wider population of SMEs in the Manchester City Region (MCR) we conducted an external benchmark survey. The intention was to construct a sampling frame which matched Creative Credits’ eligibility criteria and applicant profile, and then (randomly) select a sample of around 500 firms for a postal survey. A2.2 Non–applicants survey The sampling frame for the group of non–applicants to the Creative Credits scheme was taken from the list of registered companies which forms the FAME database (accessed on 14th April 2010).92 The list of potential companies to survey was developed to match as closely as possible the eligibility criteria for Creative Credits by location, size, sector and legal status. One element of the eligibility criteria for Creative Credits which was less easy to match within FAME related to the required VAT registration of eligible businesses. Here, we limit the sampling frame to firms which employ 3 or more people to minimise the probability of sampling non–VAT registered firms, though we could not guarantee that some non VAT–registered SMEs were surveyed. This gave an in–scope sampling frame of 4,200 companies, of which a number were excluded due to incomplete contact details, from which a random sample of 500 firms was selected. Firms in the sample were sent a postal questionnaire at broadly the same time as the baseline survey of Wave Two. A response rate of around 13 per cent was finally obtained suggesting a sampling error of around ±6 per cent. Some care is therefore necessary in the interpretation of comparisons between the applicant group and survey results for the wider population of firms. A2.3 Characteristics of Creative Credits Applicants Amongst Creative Credits applicants the sectors with greatest representation were services businesses and, in particular: Consultancy, Professional Services, General Business Services and Retail. No applications were received from the Medical sectors. Applicants were widely dispersed geographically.93 The turnover of the applicant group varied relatively widely, although over half of all applicants had an annual turnover of less than £500,000. Within this group, one–fifth of all applicants reported that their turnover was less than £100,000 per year.94 53 CREATIVE CREDITS A randomized controlled industrial policy experiment Comparing the group of Creative Credits applicants to the eligible population of companies in MCR provides an indication of the penetration of the scheme. Using data from Companies House, we estimate that there were around 4,200 eligible firms in the MCR of which 672 or around 1:6 control for Creative Credits. Applications were not, however, evenly spread across the firm size range with micro firms and firms with more than 50 employees under–represented in the group of Creative Credits applicants and firms with 10–50 employees significantly over–represented. Compared to the overall application proportion of 1:6, this meant that for the individual sizebands the scheme penetration was: 1:8 for micro firms with fewer than 10 employees; 1:2 for firms with 10–50 employees and 1:16 for SMEs with more than 50 employees. Comparing the characteristics of Creative Credits applicants using the baseline survey and non–applicants from the non–applicants survey suggests: • Creative Credits applicants were less likely to be exporting than the broader population of eligible firms. • Creative Credits applicants were significantly more likely to have a high proportion of graduate employees (more than 40 per cent of the workforce) than the broader population of eligible firms. This partly reflects the disproportionate number of applicant firms in the (graduate–intensive) business services sector. • Creative Credits applicants were much more active users of a wide range of external business support organizations than the broader population of eligible firms. Despite the broad–based nature of the scheme’s marketing, it seems likely that these linkages encouraged firms to apply for Creative Credits. • Creative Credits applicants were significantly more likely to have engaged in prior innovation than firms in the broader eligible population. In particular, around three–quarters of Creative Credits applicants reported having introduced product innovations in the previous three years compared with only 42.9 per cent of non– applicants. • Forty–two per cent of Creative Credits applicants who had undertaken prior innovation had previous experience of working with creative service providers, compared with only 20 per cent of non–applicants. In general terms, this suggests that Creative Credits applicants were more focused on innovation than non–applicants, more likely to have worked with external partners (including Creative Servicers) and to have had higher skill levels. 54 CREATIVE CREDITS A randomized controlled industrial policy experiment Annex 3: PROFILING CREATIVE CREDITS PROJECTS In this annex we profile the projects which were supported by the Creative Credits pilot. The main characteristics of the Creative Credits projects are summarized in Table A3.1 based on the main focus of the project undertaken. By far the most common theme was the upgrading and development of firms’ websites, with marketing and video production significantly less common. Fairly typical among the website development projects were the following project outlines taken from the original project proposals: “Bring site from Web 1.0 to Web 2.0. Accessibility for all users. Enhance the capability for SEO on the site. CMS system to manage content and images, giving a better level of control than previously. To create an outstanding website by delivering a cutting– edge design to encourage the viewer to remain on the site and purchase services as well as view the site as a valuable source of information, encouraging repeat visits to the site and grow the strong reputation the company already enjoys”. “To rebrand the company, to design new fonts and design for the name …. To design a new logo that can be used on the website and other items. To redesign the website completely incorporating the new redesigned logo”. “It is proposed to create a suite of flexible video tools to: have a brand dialogue with potential opinion leaders, the wider media industry and potential customers. Reinforce the positive outcomes of the service. Highlight [our company] as leaders in this industry. The resulting video content would be used at trade shows, in proposals to clients and on the company website”. “Produce informative and educational video targeting professionals and laymen … What, How, Why format using state–of–the–art mixed media techniques. Create fresh, dynamic content demystifying the science and technologies”. “Four or five short funny videos set for viral release to raise awareness of [our company]. The videos will form a series that will be seeded according to the target market and sent out at regular intervals as part of a creative marketing campaign. There is scope for the series to be continued, potentially encouraging viewers/ customers to submit their own videos or ideas with the best ones being made”. In Survey 1, the creative businesses servicing Creative Credits were asked how the Creative Credits projects had come about, and how these projects related to their other business. In the majority of cases (66 per cent), the initial contact that had led to their creative commission was made by the SME. In around a third of cases, the initial approach was made by the creative business. Almost 55 per cent of Creative Servicers reported working with an SME which was in a different sector to their usual clients, and over 41 per cent described the SME as being outside their usual business networks.95 55 CREATIVE CREDITS A randomized controlled industrial policy experiment Table A3.1: Breakdown of all Creative Credits projects Number of projects including this as primary or secondary goal Percentage of projects Web 81 60% Marketing 14 11% Video 13 10% Brand Development 10 9% Logo 8 8% Publication 8 15% PR Campaign 6 5% Market Research 2 1% New Media (iPhone App) 1 1% Product Design 1 1% Total 144 Note: In six of the projects there were a range of objectives making it difficult to identify clear primary or secondary goals. 56 CREATIVE CREDITS A randomized controlled industrial policy experiment Annex 4: ECONOMETRIC TABLES In this annex, we include a series of regression analyses underlying the findings reported in different parts of the main text. In each case models relate output and outcome variables to a set of independent variables measured in the baseline survey at the time the Creative Credits were awarded. The tables are as follows: Table A4.1 reports probit models for the probability of receiving a Creative Credit. As expected given the random allocation of Creative Credits there are few variables that are statistically significant, and where there are any these simply reflect small sample biases and have no economic significance. Model 2 in this table is used to derive the sample selection correction (Inverse Mills ratio) in subsequent models. Tables A4.2 and A4.3 report probit models of the probability of a project going ahead allowing for any selection bias. The (anticipated) insignificance of the Inverse Mills Ratio in these models suggests that there is no selection bias. Models in Table A4.3 test the impact of incentives provided to encourage survey response. Again these are statistically insignificant. Tables A4.4 and A4.5 report probit models of the probability of undertaking product, process and new to the market innovation after six and twelve months respectively. Table A4.6 reports tobit models of the impact of the Creative Credits on the sales growth distribution of the treatment and control groups six and twelve months after the end of the project. (Tobit models are used here as the dependent variable is upper and lower bounded due to survey questions which collected data on growth rates using a categorical question). Tables A4.7 and A4.8 report probit models of the probability of undertaking other forms of innovation over the next three years measured after six and twelve months respectively. Table A4.9 reports probit models of the probability of future cooperation with designers six and twelve months after the completion of the Creative Credits project. All of these models allow for potential sample selection biases. 57 CREATIVE CREDITS A randomized controlled industrial policy experiment Table A4.1: Probit models of the probability of receiving a Creative Credit Probit models of the probability that a firm received a Creative Credit Model 1 dy/dx Model 2 dy/dx Micro or new firm –0.066 –0.091* Small firm (11–50 employees) 0.027 Firm vintage (years) –0.007*** Firm member of wider group 0.027 Family–owned company 0.103 Business services company –0.044 Manufacturing company 0.052 Transport services company 0.089 Retail or wholesale business 0.087 Non–executive directors 0.053 Firm has formal business plan –0.071 Firm is an exporter 0.012 Firm has more than 20% graduates 0.028 Firm was innovator in last 3 years 0.039 Number of observations 383 432 Equation χ2 19.17 13.448 Pseudo R2 0.041 0.025 –0.006*** –0.092* Notes: Marginal effects are reported. Sample sizes differ due to missing responses to some questions in the baseline survey from which the independent variables are derived. * denotes significance at the 10 per cent level; ** at 5 per cent and *** at the 1 per cent level. 58 CREATIVE CREDITS A randomized controlled industrial policy experiment Table A4.2: Probit models of project continuation with and without selection controls Probit models of the probability that the project would have gone ahead with and without sample selection term Model 1 dy/dx Model 2 dy/dx Model 3 dy/dx Model 4 dy/dx Employment one year ago –0.002 –0.002 –0.002 –0.002 Firm member of wider group –0.172 –0.170 –0.173 –0.170 Business services company –0.091 –0.078 –0.087 –0.074 Manufacturing company –0.112 –0.077 –0.112 –0.078 Other services company 0.089 0.121 0.086 0.118 Transport services company –0.058 –0.048 –0.036 –0.029 Retail or wholesale business 0.142 0.178 0.142 0.177 Non–executive directors 0.148 0.166 0.137 0.155 Family–owned company –0.052 –0.061 –0.067 –0.074 Firm has formal business plan –0.103 –0.099 –0.122 –0.116 Firm is an exporter 0.090 Firm has more than 20% graduates 0.079 0.085 0.086 0.09 Firm was innovator in last 3 years –0.089 –0.084 –0.079 –0.075 Creative Credit recipient 0.862*** 0.860*** 0.864*** 0.861*** Inverse Mills Ratio –0.181 –0.161 Number of observations 382 382 382 382 Equation χ2 287.72 286.414 287.008 285.833 Pseudo R2 0.570 0.567 0.568 0.566 0.085 Notes: Marginal effects are reported. Sample sizes differ due to missing responses to some questions in the baseline survey from which the independent variables are derived. * denotes significance at the 10 per cent level; ** at 5 per cent and *** at the 1 per cent level. 59 CREATIVE CREDITS A randomized controlled industrial policy experiment Table A4.3: Probit models of project continuation with and without selection allowing for incentive payments Probit models of the probability that the project would have gone ahead with and without sample selection term and allowing for incentive payments Model 1 dy/dx Model 2 dy/dx Model 3 dy/dx Model 4 dy/dx Employment one year ago –0.002 –0.002 –0.002 –0.002 Firm member of wider group –0.173 –0.170 –0.173 –0.170 Business services company –0.091 –0.078 –0.087 –0.074 Manufacturing company –0.109 –0.075 –0.109 –0.075 Other services company 0.090 0.121 0.086 0.118 Transport services company –0.052 –0.043 –0.031 –0.024 Retail or wholesale business 0.142 0.178 0.141 0.177 Non–executive directors 0.146 0.164 0.134 0.153 Family–owned company –0.054 –0.063 –0.07 –0.077 Firm has formal business plan –0.101 –0.097 –0.12 –0.115 Firm is an exporter 0.091 Firm has more than 20% graduates 0.080 0.086 0.087 0.091 Firm was innovator in last 3 years –0.089 –0.084 –0.079 –0.075 Creative Credit recipient 0.865*** 0.862*** 0.867*** 0.864*** Inverse Mills Ratio –0.180 –0.159 Incentive payment to firm 0.020 0.017 0.022 0.019 Number of observations 382 382 382 382 Equation χ2 287.782 286.457 287.082 285.889 Pseudo R2 0.570 0.567 0.568 0.566 0.086 Notes: Marginal effects are reported. Sample sizes differ due to missing responses to some questions in the baseline survey from which the independent variables are derived. * denotes significance at the 10 per cent level; ** at 5 per cent and *** at the 1 per cent level. 60 CREATIVE CREDITS A randomized controlled industrial policy experiment Table A4.4: Probit models of the probability of innovation after six months Probit models of the probability of innovation six months after the end of the Creative Credits project allowing for sample selection Product/ service innovation New to the market innovation Process Innovation b/se b/se b/se Employment one year ago 0.000 –0.001 0.001 Firm member of wider group 0229*** 0.119 0.063 Business services company –0.057 0.167 0.006 Manufacturing company 0.122 0.253 0.117 Other services company -0.046 0.029 0.040 Transport services company 0.047 Retail or wholesale business 0.195** 0.134 0.090 Non–executive directors 0.084 0.103 0.049 Family–owned company -0.165** -0.182** -0.017 Firm has formal business plan 0.028 -0.018 0.107 Firm is an exporter 0.019 -0.091 -0.066 Firm has more than 20% graduates 0.106 0.002 0.186** Firm was innovator in last 3 years 0.219** 0.112 0.101 Creative Credit recipient 0.132** 0.140* 0.195*** Inverse Mills Ratio 0.314* -0.016 0.100 Number of observations 228 175 209 Equation χ2 34.472 16.739 19.95 Pseudo R2 0.119 0.076 0.069 0.093 Notes: Marginal effects are reported. Sample sizes differ due to missing responses to some questions in the baseline survey from which the independent variables are derived. * denotes significance at the 10 per cent level; ** at 5 per cent and *** at the 1 per cent level. 61 CREATIVE CREDITS A randomized controlled industrial policy experiment Table A4.5: Probit models of the probability of innovation after 12 months Probit models of the probability of innovation six months after the end of the Creative Credits project allowing for sample selection Product/ service innovation New to the market innovation Process Innovation b/se b/se b/se Employment one year ago –0.001 0.001 0.000 Firm member of wider group 0.058 0.156 0.092 Business services company -0.028 0.073 -0.120 Manufacturing company 0.016 0.070 -0.115 Other services company 0.111 0.106 -0.034 Transport services company -0.06 -0.269** 0.183 Retail or wholesale business -0.035 0.119 0.089 Non–executive directors 0.035 0.016 -0.086 Family–owned company -0.046 -0.11 -0.105 Firm has formal business plan 0.220*** 0.102 0.073 Firm has more than 20% graduates 0.167** 0.072 0.153** Firm was innovator in last 3 years 0.054 0.150* 0.271*** Creative Credit recipient 0.097 0.127* -0.025 Inverse Mills Ratio -0.103 0.07 -0.032 Number of observations 230 199 222 Equation χ2 27.213 25.124 23.261 Pseudo R2 0.092 0.097 0.076 Notes: Marginal effects are reported. Sample sizes differ due to missing responses to some questions in the baseline survey from which the independent variables are derived. * denotes significance at the 10 per cent level; ** at 5 per cent and *** at the 1 per cent level. 62 CREATIVE CREDITS A randomized controlled industrial policy experiment Table A4.6: Tobit models of the impact on sales growth after 6 and 12 months Tobit models of the impact of Creative Credits on the distribution of sales growth rates after 6 and 12 months allowing for sample selection After six months Coefficients After 12 months Coefficients Employment one year ago –0.053 0.003 Firm member of wider group –4.870 –1.156 Business services company 2.498 –0.352 Manufacturing company 1.857 –3.260 Other services company –3.959 –5.274 Transport services company 6.556 1.753 Retail or wholesale business 3.410 1.495 Non–executive directors 2.075 –4.180 Family–owned company –0.879 1.582 Firm has formal business plan 2.087 7.164** Firm is an exporter 0.684 0.949 Firm has more than 20% graduates 4.269* 6.955*** Firm was innovator in last 3 years 2.076 –0.171 Creative Credit recipient 0.370 3.530 Inverse Mills Ratio –2.598 –3.612 Constant term 1.602 0.286 Number of observations 211 225 Equation χ2 14.662 18.757 Pseudo R2 0.010 0.012 Notes: Coefficients are reported. Sample sizes differ due to missing responses to some questions in the baseline survey from which the independent variables are derived. * denotes significance at the 10 per cent level; ** at 5 per cent and *** at the 1 per cent level. 63 CREATIVE CREDITS A randomized controlled industrial policy experiment Table A4.7: Probit models of likelihood of innovation over the next three years after 6 months with selection Probit models of the likelihood that the firm would undertake innovation in the next three years, measured six months after the end of the Creative Credits projects, with sample selection term. Goods or services Processes Strategy New tech– Organi– nologies sation Marketing Model 1 dy/dx Model 2 dy/dx Model 3 dy/dx Model 4 dy/dx Model 5 dy/dx Model 6 dy/dx Employment one year ago 0.001 0.001 0.003** 0.003 –0.001 0.000 Firm member of wider group –0.168 –0.073 –0.265* –0.078 –0.061 Business services company 0.060 0.046 0.043 –0.055 –0.071 –0.021 Manufacturing company 0.029 0.042* 0.069 0.149 0.035 –0.040 –0.011 –0.154 –0.028 –0.127 –0.264 0.071 0.148 –0.032 Other services company Transport services company Retail or wholesale business 0.048** 0.061 –0.058 0.024 –0.010 Non–executive directors 0.028 0.045 0.004 0.033 –0.009 Family–owned company 0.014 0.045** 0.031 0.022 0.092 0.009 Firm has formal business plan –0.001 0.036 –0.029 0.084 0.162* 0.057 Firm is an exporter –0.005 –0.023 –0.057 0.013 –0.053 –0.006 Firm has more than 20% graduates 0.051 0.022 0.032 –0.048 0.038 0.057 Firm was innovator in last 3 years 0.064 0.129* 0.115 0.107 0.033 0.075 Creative Credit recipient 0.026 –0.012 0.004 –0.060 –0.072 0.001 Inverse Mills Ratio 0.095 0.014 –0.044 –0.278 –0.217 0.068 Number of observations 126 193 185 171 182 176 Equation χ2 11.263 22.748 20.31 15.148 10.315 11.564 Pseudo R2 0.208 0.227 0.15 0.066 0.042 0.118 Notes: Marginal effects are reported. Sample sizes differ due to missing responses to some questions in the baseline survey from which the independent variables are derived. * denotes significance at the 10 per cent level; ** at 5 per cent and *** at the 1 per cent level. 64 CREATIVE CREDITS A randomized controlled industrial policy experiment Table A4.8: Probit models of firms’ likelihood of innovation over the next three years after 12 months with selection Probit models of the likelihood that the firm would undertake innovation in the next three years, measured twelve months after the end of the Creative Credits projects, with sample selection term. Employment one year ago Goods or services Processes Strategy New tech– Organi– nologies sation Marketing Model 1 dy/dx b/se Model 2 dy/dx b/se Model 3 dy/dx b/se Model 4 dy/dx Model 5 dy/dx Model 6 dy/dx 0.000 0.000 0.002 0.000 –0.001 0.001 0.052 0.014 0.148 0.126 –0.052 –0.042 –0.015 0.023 0.043 –0.025 0.047 0.211 0.082 –0.003 Firm member of wider group Business services company –0.004 Manufacturing company Other services company –0.053 –0.136 –0.183 –0.039 –0.111 0.022 Transport services company –0.047 –0.095 –0.061 0.184 0.269 –0.022 Retail or wholesale business 0.044 0.033 –0.036 –0.072 0.125 0.033 Non–executive directors –0.024 0.011 0.050 0.188* 0.189* –0.045 Family–owned company –0.021 0.031 0.060 –0.056 0.047 0.107* Firm has formal business plan 0.061 0.017 0.113* 0.074 0.116 0.165** Firm is an exporter 0.024 0.001 –0.039 0.015 0.124 0.118* Firm has more than 20% graduates 0.051 0.036 0.037 0.072 0.043 0.064 Firm was innovator in last 3 years 0.149* 0.164** 0.064 0.117 0.131 0.081 Creative Credit recipient –0.020 –0.016 –0.052 0.115 0.000 0.054 Inverse Mills Ratio 0.016 –0.133 –0.082 0.152 0.040 0.150 Number of observations 155 216 207 170 185 176 Equation χ2 12.161 17.246 16.847 20.769 19.834 18.058 Pseudo R2 0.129 0.121 0.088 0.088 0.077 0.103 Notes: Marginal effects are reported. Sample sizes differ due to missing responses to some questions in the baseline survey from which the independent variables are derived. * denotes significance at the 10 per cent level; ** at 5 per cent and *** at the 1 per cent level. 65 CREATIVE CREDITS A randomized controlled industrial policy experiment Table A4.9: Modelling the probability of future cooperation with designers Probit models of the probability of future innovation cooperation with designers allowing for sample selection After six months dy/dx After 12 months dy/dx Employment one year ago 0.002 0.001 Firm member of wider group 0.058 0.053 Business services company –0.081 0.046 Manufacturing company 0.032 0.067 Other services company –0.042 –0.018 Transport services company 0.035 Retail or wholesale business 0.088 0.018 Non–executive directors –0.172* –0.009 Family owned company 0.128*** 0.057 Firm has formal business plan –0.028 0.035 Firm is an exporter –0.017 0.009 Firm has more than 20% graduates 0.048 0.035 Firm was innovator in last 3 years 0.056 0.005 Creative Credit recipient –0.029 0.049 Inverse Mills Ratio 0.439** 0.149 Number of observations 193 210 Equation χ2 24.09 5.46 Pseudo R2 0.13 0.03 Notes: Marginal effects are reported. Sample sizes differ due to missing responses to some questions in the baseline survey from which the independent variables are derived. * denotes significance at the 10 per cent level; ** at 5 per cent and *** at the 1 per cent level. 66 CREATIVE CREDITS A randomized controlled industrial policy experiment Annex 5: ESTIMATING THE BENEFIT–COST RATIO FOR CREATIVE CREDITS As part of Survey 2 – immediately at the end of the Creative Credits project – and in Survey 4, 12 months after the end of the Creative Credits project, firms in the treatment group were asked to provide their own estimate of the impact of their Creative Credits project on their sales. These should be viewed as an indirect estimate of the sales impacts at best, because they are based on self–reported impacts, as compared with the difference between reported overall sales in the treatment and control group of SMEs. Nonetheless, insofar as identifying the separate impact of Creative Credits from other influences on overall sales is necessarily challenging, these impacts may contain some information. SMEs put their mean impact on sales at £2,900 at the end of their projects and a further £7,740 after 12 months. These estimates were subject to considerable variation, however, with standard deviations of £5,970 at the end of the project (implying a coefficient of variation of 2.03) and £8,880 after 12 months (coefficient of variation of 1.14). We make two observations. First, after 12 months the average addition to sales reported stemming from the Creative Credits project was substantially greater than the actual cost of the Creative Credits themselves. Second, although the coefficients of variation suggest that additions to sales varied substantially at the end of the projects themselves this variation had declined markedly after 12 months. In other words, the variability in outcomes had declined over this period. Marked variation across SMEs is also evident in the anticipated duration of the benefits of the Creative Credits. In Survey 4 – a year after project completion – only 17.6 per cent of firms in the treatment group reported having already got the whole benefit from their Creative Credits project. The larger proportion of firms continued to see future benefits with some even seeing the benefits running beyond five years. On average – using mid– points in the calculation – the average duration of benefits from Creative Credits was 2.5 years. Using this benefit duration and the benefit profile, we are able to derive an estimate of the Creative Credits’ benefit–cost ratio. On the cost side of the equation, a Creative Credit had face value £4,000 with each recipient required to add a minimum of an additional £1,000, totalling £5,000. In the event, however, the average project cost was slightly higher – £5,400 with firms contributing an average of £1,400 each. In terms of benefits, projects’ additional sales averaged £2,900 in year 1 and in the next twelve months added a further £7,740 to sales. This benefit was expected to continue for 2.5 years. Adopting a standard annual discount rate of 3.5 per cent as suggested in the Integrated Evaluation Framework96 the benefit stream therefore had total sales impact of £2,900+£7,469+£7,207+ £3,478=£21,054. To derive a gross value added (GVA) impact figure requires a turnover/GVA ratio. For the whole economy in 2010 this ratio was 0.300 (ABI 2010, all industries, release date 17/11/2011), implying a discounted total GVA of £6,316. Applying the mean regional displacement value suggested for R&D and innovation support measures (24.5 per cent)97 67 CREATIVE CREDITS A randomized controlled industrial policy experiment and the average suggested regional multiplier of 1.5698 for similar schemes suggests a total GVA of £6,316 x 0.755 x 1.56 =£7,438. This suggests a benefit–cost ratio of 1.37. Note, however, that this benefit–cost ratio is based on an assumed persistence of benefits of 2.5 years derived from SME survey responses. Government best practice guidelines for project evaluation (the Integrated Evaluation Framework Guidelines) suggest instead a 3–year benefit duration. Making this assumption and maintaining all others suggests a higher discounted GVA impact of £7,360. Applying displacement and multiplier ratios to this number suggests a total regional GVA impact per case of £8,670, and a benefit– cost ratio of 1.60. There are great uncertainties around the value of all the assumed parameters, so (in addition to the uncertainties surrounding impact estimates based on firms’ subjective assessments of project sales benefits) these impact estimates should be treated with especially great caution. 68 CREATIVE CREDITS A randomized controlled industrial policy experiment Annex 6: CASE STUDIES W1S10 /W1C9 CASE STUDY ONE The project objective outlined by W1S10 in Stage 1 was to improve the delivery and design of an e–commerce site. Output additionality of the project was suggested by a sales increase of 75 per cent by the Stage 2 qualitative interview, around nine months after the completion of the project. There were some signs of behavioural additionality, in that the SME had clearly recognised the business benefits of creative input (learning ‘to listen to’ rather than direct the process), and was more open to new ways of doing business as a result. Network additionality was also evident through the SME’s expanded creative and supplier networks. The relationship between W1S10 and W1C9 itself proved short–term, as W1S10 felt ‘they’re too expensive for us.’ W1S10 stated that, despite the positive outcomes, the final cost of the work with W1C9, which amounted to £15,000, was excessive. W1C9 was not available for interview at Stage 4. Output Additionality By Stage 2, W1S10 had experienced a jump in sales of 75 per cent. W1S10 attributed this directly to the improved website. However, W1S10 was unsure of the extent to which this increase in turnover had translated into ‘a sizeable profit’, but believed ‘we will make a sizeable profit next year’. Behavioural Additionality Prior to the Creative Credits project, W1S10 had had minimal engagement with the creative sector, preferring to work in–house. W1C9 had won the confidence of W1S10: As the latter said, “I realised that I shouldn’t be leading the design…You tell him the product that’s going to be sold, let him decide how it looks, because otherwise, you’re not getting any value out of it.” By Stage 4, W1S10 had outsourced all of its creative work (to creative companies that were less costly than W1C9) to ensure that its marketing materials were visually consistent. W1S10 spoke of their initial disappointment that the website had not been more ‘funky’, but later recognised W1C9’s skills in presenting “something that actually is totally right.” and how “The value came in the depth and the polished look of the design.” W1S10 believed that the confidence gained from the improved website had encouraged them to consider “more ways of developing, ways of taking the business forward’ such as trading in Europe.” While W1S10 felt it was more creative and innovative as a result of the project, it was still careful to evaluate new ventures “…if you’re gonna jump into the void you’ve gotta know what the bottom line is.” Network Additionality W1S10 reported how exposure to the Creatives Gallery had vastly expanded their creative networks and introduced them to new ideas such as social media “[the area] is awash with creative talent, which I was not aware of.” They have since worked with several of 69 CREATIVE CREDITS A randomized controlled industrial policy experiment these creative companies. At Stage 4, W1S10 spoke of how the new website had changed suppliers’ perceptions “…[previously inaccessible] brands that had come on as a result of the way the website looks”, expanding and improving the quality of their supplier network. Advice for SMEs working with creative suppliers W1S10’s advice for an SME included: immerse yourself in design throughout; examine numerous websites, reviewing multiple creative companies and check that they have both design and technical capabilities. W1S10 also spoke of the need to trust the creative’s work as “...you can’t dictate, you have to allow them to function, to breathe and to tell you. That’s what they do every day.” W1S6 /W1C6 CASE STUDY TWO W1S6 closed for business at the end of 2011. The project objective outlined by W1S6 in Stage 1 was to create a brochure–style website for a new division that would allow a live portfolio to be maintained and used by future and current clients. The account that they gave of their Creative Credits project was positive. However, there was no evidence of additional sales impacts. Behavioural additionality was evident in the SME, through improved brand coherence and recognition of the value of creative input. There was some evidence that network additionality was evident in terms of a deepening relationship with the Creative Servicer. Output Additionality W1S6 felt unable to assess its Creative Credits project impact in terms of sales by Stage 4. Behavioural Additionality Organizational learning was evident during the project. In Stage 1, W1S6 viewed the website in purely functional terms: “we finalised the brief and got the structure of the site, because before they designed anything the structure was laid down.” This point was echoed in Stage 2, when W1S6 spoke of applying their learning about the multiple functions of a website. By Stage 4, W1S6 spoke about “the image we wanted to portray...” and making their marketing materials visually consistent: “...they’ve created a leaflet that’s based on the images in the website and the look of the website. So it’s a consistent marketed image...[before] They would have been separate.” W1S6 also changed its definition of innovation, from product–oriented in Stage 1, to include the value of presentation in engaging new markets, and spoke of the value of using an expert to engage in creativity. This new visual awareness of W1S6 was also noted by W1C6 at Stage 4. W1C6, in turn, gained experience in a new industry sector and created “a new content management...system (which) streamlined our project timelines” specifically for the project, which they continued to use and develop afterwards. Network Additionality W1S6 and W1C6 developed a positive business relationship through their Creative Credits project. This was apparent in W1S6’s confidence in W1C6’s expertise. By Stage 4, W1S6 stated “…they can see the business picture and they understand the business that we’re in”. Plans for a website for other divisions discussed by W1S6 at Stages 1, 2 and 4, did not materialise as “There are just other priorities to be honest.” (Stage 4). W1S6 closed for business shortly after the Stage 4 qualitative interview. 70 CREATIVE CREDITS A randomized controlled industrial policy experiment Advice for SMEs working with creative suppliers At Stage 4, W1S6’s advice on selecting a creative provider was “Talk to a lot of them, pick as many ideas as you can”, then “pick a company that when you’ve been talking to them they’ve been listening to what you do, not telling you what they can do”. W1C6 felt the key issue was for the SME to be clear what they wanted to achieve by having a website in order to guide the creative company and to consider “...is it actively doing anything for your business?” W1S11 /W1C10 CASE STUDY THREE The project objective outlined by W1S11 in Stage 1 was to create a website to introduce a new product to the marketplace. The relationship with the creative partner W1C10 got off to a bad start, deteriorated further, but subsequently improved. At the end of the Creative Credits project, the relationship appeared to have failed both because of a mismatch of expectations and mistakes by W1C10’s original project manager meant the website design proved unsuitable. However, W1C10 changed the website to address W1S11’s concern so that by Stage 4, W1C10 continued to host the site. Output additionality was evident through increased sales and expanded markets. There appears to have been some behavioural additionality in W1S11 through improved communication skills and an increased confidence in marketing. Network additionality was evident through the ongoing, if shaky, relationship with the Creative Servicer and with other creative companies, customer referrals to W1S11 by W1C10, and a growing group of global innovation partnerships. Output Additionality The manager of W1S11 explained how he had challenged W1C10 about the website: “I was feeling sick”. In response, W1C10 had revamped the website. By Stage 2, W1S11 stated that the website had already “made a huge difference to our business”, increasing exposure and allowing global expansion. By Stage 4, sales had demonstrably improved and W1S11 attributed the increase largely to the website. Behavioural Additionality At Stage 1, W1S11 had been unfamiliar with the creative sector, perceiving little value in it. By Stage 4, their opinion had completely changed, and the value in the project was seen in it having improved communication skills and increasing W1S11’s confidence in marketing. W1S11 had commissioned another creative company to work on the website prior to the Stage 4 interview. Future work with W1C10, in addition to website hosting, remained a possibility. W1S11 had also become highly active in social media, one of W1C10’s key skill sets. For their part, W1C10 had now put in place a project management system as well as having introduced a more formalized approach to working with clients. Network Additionality At Stage 4, the website and social media were cited by W1S11 as having been important for creating new innovation development partnerships both locally and internationally. By Stage 4, W1S11 had worked with another new creative supplier, and was also considering moving some creative work in–house. Advice for SMEs working with creative suppliers W1S11 was concerned that the creative companies had not been vetted prior to going on the Creatives Gallery. W1S11’s advice on selecting a creative provider was to check out their 71 CREATIVE CREDITS A randomized controlled industrial policy experiment references then select on ‘that kind of instinct where it tells you, “I’m talking to the right person here”.’ W1C10 felt that most of the SMEs did not understand that within the creative industries, work is paid mostly up front and that fees are exclusive of VAT. This confusion had led to cash– flow issues and delayed payments, and thus a poor initial working relationship. W1C10 suggested that in any future scheme the creative companies be paid directly by Nesta. 72 CREATIVE CREDITS A randomized controlled industrial policy experiment ACKNOWLEDGEMENTS This report was written by Hasan Bakhshi from Nesta, John Edwards, Judy Scully and Nicola Rathbone from Aston Business School and Stephen Roper, Duncan Shaw and Lorraine Morley from Warwick Business School. Thanks go to Albert Bravo–Biosca, Haidee Bell, Karen Brayne, David Guy, Adam Hewitt, Jon Kingsbury, Brian MacAulay, Juan Mateos–Garcia and Angela Pugh for their help throughout the Creative Credits project. We are also grateful to members of the Creative Credits Steering Board for their input and to the businesses in the Manchester City Region who so generously gave their time to participate in the research. Geoff Mulgan at Nesta and Geoff Pugh at Staffordshire University Business School provided helpful comments on an earlier draft of this report. Creative Credits was funded jointly by Nesta, the Economic and Social Research Council, the Arts and Humanities Research Council, the North West Development Agency and Manchester City Council. 73 CREATIVE CREDITS A randomized controlled industrial policy experiment ENDNOTES 1. Bakhshi, H., McVittie, E. and Simmie, J. (2008) ‘Creating Innovation: Do the Creative Industries Support Innovation in the Wider Economy?’ London: NESTA. NESTA Research Report in the case of the UK; Stam, E., de Jong, J.P. and Marlet, G. (2008), Creative Industries in the Netherlands: Structure, development, innovativeness and effects on urban growth. ‘Geografiska Annaler.’ Series B, Human Geography 90, 119-132, for the Netherlands, and Muller, C., Rammer, C. and Truby, J. (2009), The Role of Creative Industries in Industrial Innovation. ‘Innovation: Management, Policy and Practice.’ 11(2), 148-168, for Austria. 2. Bakhshi, H., Freeman, A. and Potts, J. (2011) ‘State of Uncertainty.’ Nesta Provocation 14. London: NESTA. 3. Stoker, G. and John, P. (2008) Design experiments: engaging policy makers in the search for evidence about what works. ‘Political Studies.’ 57 (2), 356-373. 4. Bakhshi, H., Hargreaves, I. and Mateos–Garcia, J. (2013) ‘A Manifesto for the Creative Economy.’ London: Nesta 5. Chesbrough, H.W. (2010) ‘Open Innovation: A Key to Achieving Socioeconomic Evolution. How Smaller Companies Can Benefit from Open Innovation.’ Japan Economic Foundation, mimeo; Brunswicker, S. and Vanhaverbeke, W. (2011) ‘Beyond Open Innovation in Large Enterprises: How Do Small and Medium-Sized Enterprises (SMEs) Open Up to External Innovation Sources?’ Available at SSRN: http://ssrn.com/abstract=1925185 van de Vrande, V, de Jong, J.P.J., Vanhaverbeke, W. and de Rochemont, M. (2009) Open innovation in SMEs: trends, motives and management challenges. ‘Technovation.’ 29 (6-7), pp. 423–437. 6. van de Vrande, V, de Jong, J.P.J., Vanhaverbeke, W. and de Rochemont, M. (2009) Open innovation in SMEs: trends, motives and management challenges. ‘Technovation.’ 29 (6-7), pp. 423–437. 7. Powell, W.W. (1998) Learning from collaboration: knowledge and networks in the biotechnology and pharmaceutical industries. ‘California Management Review.’ 40, pp. 228-240. 8. Niosi, J. (2003) Alliances are not enough for explaining rapid growth in biotechnology firms. ‘Research Policy.’ 32 (5), pp. 737-750. 9. Leiponen, A. and Helfat, C. E. (2010) Innovation objectives, knowledge sources, and the benefits of breadth. ‘Strategic Management Journal.’ 31, pp. 224-236. 10. Roper, S., Du, J. and Love, J. H. (2008) Modelling the Innovation Value Chain. ‘Research Policy.’ 37, pp. 961-977. 11. Sieg, J.H., Wallin, M.W. and von Krogh, G. (2010) Managerial challenges in open innovation: a study of innovation intermediation in the chemical industry. ‘R&D Management.’ 40 (3), pp. 281–291. 12. Laursen, K. and Salter, A. (2006) Open for Innovation: The role of openness in explaining innovation performance among UK manufacturing firms. Strategic Management Journal 27, pp. 131-150. Leiponen A. and Helfat C. E. (2010). Innovation objectives, knowledge sources, and the benefits of breadth. ‘Strategic Management Journal.’ 31, pp. 224-236. 13. Geroski, P., Machin, S. and Van Reenen, J. (1993). The Profitability of Innovating Firms. ‘Rand Journal of Economics.’ 24, pp. 198-211. Leiponen, A. (2000). Competencies, innovation and profitability of firms. ‘Economics of Innovation and New Technology.’ 9, pp. 1-24. 14. Crepon, A., Hughes, A., Lee, P. and Mairesse, J. (1998) Research, Innovation and Productivity: An econometric analysis at the firm level. ‘Economics of Innovation and New Technology.’ 7, pp. 115-158. Loof, H. and Heshmati, A. (2002) Knowledge capital and performance heterogeneity: A firm level innovation study. ‘International Journal of Production Economics.’ 76, pp. 61-85. 15. Anyadike-Danes, M., Bonner, K., Hart, M. and Mason, C. (2009) ‘Measuring business growth: high growth firms and their contribution to employment in the UK.’ London: NESTA. 16. Love, J. H., Roper, S. and Bryson, J. (2011) Knowledge, Openness, Innovation and Growth in UK Business Services. ‘Research Policy.’ 40, pp. 1438-1452. 17. OECD (2006) Evaluating Government Financing of Business R&D: Measuring Behavioural Additionality – Introduction and Synthesis, 23-24 March 2006, Sydney; DSTI/STP(2006). 18. Hewitt-Dundas, N. and Roper, S. (2009) Output Additionality of Public Support for Innovation: Evidence for Irish Manufacturing Plants. ‘European Planning Studies.’ 18, pp. 107-122. 19. Buiseret, T., Cameron, H. M. and Georghiou, L. (1995). What differences does it make? Additionality in the public support of R&D in large firms. ‘International Journal of Technology Management.’ 10, pp. 587-600. Georghiou, L. (2004) ‘Evaluation of Behavioural Additionality.’ Concept Paper. European Conference on Good Practice in Research and Evaluation and Indicators. 20. Falk, R. (2004) ‘Behavioural additionality effects of R&D subsidies.’ Vienna: Austrian Institute of Economic Research. 21. Gemser, G. and Leenders, M. (2001) How integrating industrial design in the product development process impacts on company performance. ‘The Journal of Innovation Management.’ 18, pp. 28-38. Stoneman, P. (2010) ‘Soft Innovation.’ Cambridge: Cambridge University Press. 22. Verganti, R. (2009) ‘ Design-driven innovation.’ Boston, MA: Harvard Business Press. 23. Verganti, R. (2008) Design, meanings, and radical innovation: A meta-model and a research agenda. ‘Journal of Product Innovation Management.’ 25, pp. 436-456. 24. For the Netherlands see: Marsili, O. and Salter, A. (2006) The Dark Matter of Innovation: Design and Innovative Performance in Dutch Manufacturing. ‘Technology Analysis & Strategic Management.’ 18, pp. 515-534. For the UK see: Cereda, M., Crespi, G., Criscuolo, C. and Haskel, J. (2005) Design and Company Performance: Evidence from the Community Innovation Survey in: REPORT, D. (ed.). London. For Flanders see: Czarnitzki, D. and Thorwarth, S. (2012) The design paradox: the contribution of in-house and external design activities on product market performance. ‘Journal of Product Innovation Management.’ Forthcoming. 25. Bakhshi H., McVittie E. and Simmie J. op. cit.; Muller, K., Rammer, C., and Truby, J. op. cit., and Stam, E., de Jong, J.P. and Marlet, G. op. cit. 26. Source: page 32, Goffin, K. & Micheli, P. (2010). Maximizing the value of industrial design in new product development. Research-Technology Management, 53, pp. 29-37. 27. Source: pages 119-120, Perks, H., Cooper, R. and Jones, C. (2005). Characterizing the role of design in new product development: An empirically derived taxonomy. ‘Journal of Product Innovation Management.’ 22, pp. 111-127. 28. Song, X. M., Montoya-Weiss, M. M. and Schmidt, J. B. (1997) Antecedents and consequences of cross functional co-operation: a comparison of R&D, manufacturing and marketing perspectives. ‘Journal of Product Innovation Management.’ 14, pp. 35-47. 29. Rosenbusch, N., Brinckmann, J. and Bausch, A. (2011) Is innovation always beneficial? A meta-analysis of the relationship between innovation and performance in SMEs. ‘Journal of Business Venturing.’ 26, pp. 441-457. 30. Chesbrough, H.W. (2010) ‘Open Innovation: A Key to Achieving Socioeconomic Evolution. How Smaller Companies Can Benefit from Open Innovation.’ Japan Economic Foundation, mimeo. 74 CREATIVE CREDITS A randomized controlled industrial policy experiment 31. van de Vrande, V., de Jong, J.P.J., Vanhaverbeke, W. and de Rochemont, M. (2009) Open innovation in SMEs: trends, motives and management challenges. ‘Technovation.’ 29 (6-7), pp. 423–437. 32. Vahter P., Love, J. H. and Roper, S. (2012) ‘Openness and innovation performance: are small firms different?’ CSME Working Paper 113, January. 33. Potts, J. and Morrison, K. (2009) ‘Nudging Innovation: Fifth generation innovation, behavioural constraints, and the role of creative business – considerations for the NESTA innovation vouchers pilot.’ NESTA Working Paper, London: NESTA. 34. DG ENTR-Unit D2 (2009) ‘Availability and Focus on Innovation Voucher Schemes in European Regions.’ Brussels: European Commission. 35. Czarnitzki, D., Ebersberger, B. and Fier, A. (2007) The relationship between R&D collaboration, subsidies and R&D performance: Empirical evidence from Finland and Germany. ‘Journal of Applied Econometrics.’ 22, 1347-1366. 36. Hall, B. and Lerner, J. (2009) ‘The financing of R&D and innovation.’ NBER working paper 15325. Woolthuis, R. K., Lankhuizen, M. and Gilsing, V. (2005). A system failure framework for innovation policy design. ‘Technovation.’ 25, pp. 609-619. Carlsson, B. and Jacobsson, S. (1997) In search of useful public policies: key lessons and issues for policy makers. In ‘Technological systems and industrial dynamics.’ edited by Carlsson, B. Dordrecht: Kluwer Academic Publishers. 37. Potts and Morrison op. cit. 38. Bakhshi, H., McVittie, E. and Simmie, J. op. cit.; Muller, K., Rammer, C., and Truby, J. (2009) The role of creative industries in industrial innovation. ‘Innovation: Management, Policy and Practice.’ Vol. 11, pp.148-168. 39. Donaldson, S. I. and Gooler, L.E. (2003) Theory-driven evaluation in action: lessons from and $20 million statewide Work and Health Initiative. ‘Evaluation and Program Planning.’ 26, pp. 355-366. 40.DG ENTR-Unit D2 (2009) ‘Availability and Focus on Innovation Voucher Schemes in European Regions.’ Brussels: European Commission. 41. Potter, J. and Storey, D. (2007) ‘OECD Framework for the Evaluation of SME and Entrepreneurship Policy and Programmes.’ Paris: OECD. Mole, K. F., Hart, M., Roper, S. and Saal, D. (2008) Differential Gains from Business Link Support and Advice: A Treatment Effects Approach. ‘Environment and Planning.’ C, 26, pp. 316-334. 42. Holland, P. (1986) Statistics and causal inference. ‘Journal of the American Statistical Association.’ 81, pp. 945-960. 43. Reiner, C. (2011) Evaluating innovation policies by chance? The case for randomized R&D-programme evaluation. ‘Plattform Forschungs- und Technologieevaluierung.’ 37, pp. 15-28. 44. Bremnes, H. Bergem, B. and Nesset, E. (2011) ‘Coherence between policy formulation and implementation of public research support? An examination of project selection mechanisms in the Norwegian Research Council.’ Molde University College, Working Paper 2011:3. 45. GPrix (2012) Good Practices in Innovation Support Measures for SMEs: facilitating transition from the traditional to the knowledge economy. Recommendations report by Staffordshire University and Fraunhofer IFF on behalf of the GPrix consortium, FP7. 46. Burtless, G. (1995) The case for randomized field trials in economic and policy research.’ ‘Journal of Economic Perspectives.’ 2, 63-84, p. 68. 47. Reiner, C. (2011) Op. cit., p. 20-21. Duflo, E., Glennerster, R. and Kremer, M. (2006) ‘Using Randomization in Development Economics Research: A Toolkit.’ NBER Technical Working Paper 333. Cambridge, MA: NBER. 48. GPrix (2012) Op. cit. 49. Chen, H-T. (1990) ‘Theory-Driven Evaluations.’ London: Sage. 50. Ludwig, J., Kling, J. and Mullainathan, S. (2011) ‘Mechanism Experiments and Policy Evaluations.’ NBER working paper 17062. Cambridge, MA: NBER. 51. Jackson, A. (2001) An evaluation of evaluation: problems with performance measurement in small business loan and grant schemes. ‘Progress in Planning.’ 55, pp. 1-64. White, H. (2008) Of probits and participation: The use of mixed methods in quantitative impact evaluation. ‘Ids Bulletin-Institute Of Development Studies.’ 39 (1), pp.98-109. 52. The mechanics of implementing the Creative Credits scheme are described in detail in (2011) ‘A Guide to Creative Credits.’ London: NESTA. Available at: http://www.nesta.org.uk/library/documents/Creative_Credits_Guide.pdf 53. The Creative Credits scheme was implemented as a partnership between NESTA, Manchester City Council, North West Development Agency (NWDA), the Economic and Social Research Council (ESRC) and the Arts and Humanities Research Council (AHRC). 54. This applied to around 5 per cent of the awarded Creative Credits which were re-allocated to firms on the reserve list. This re-allocation will have imparted a small upwards bias to our estimates of additionality. 55. To be approved, creative firms had to have been a going concern for at least one year with supporting account documentation, and also have professional indemnity insurance. 56. Technically, however, the randomized control trial was ‘unblinded’ in that both the ‘treatment’ and ‘control’ groups clearly knew whether or not they had received a Credit. 57. See also Bakhshi, H., Edwards, J., Roper, S., Scully, J. and Shaw, D. (2011) ‘Creating Innovation: Evaluating the Short-Term Effects of the Creative Credits Pilot.’ NESTA Working Paper. London: NESTA. 58. Eden, C. and Ackermann, F. (1998) ‘Making Strategy: The Journey of Strategic Management.’ London: Sage. 59. This is ‘at least’ rather than exactly 84 per cent, because SMEs in the control group were asked whether or not they had proceeded with their creative project, not whether they had proceeded with their creative project specifically with a business they had not previously worked with. 60. This is higher than the 78.1 per cent estimate reported in Bakhshi et al. (2011), which did not include Creative Credits that were reallocated to SMEs. Bakhshi et al. (2011) op. cit. 61. Cornet, M., Vroomen, B. and van der Steeg, M. (2006) ‘Do innovation vouchers help SMEs to cross the bridge towards science?’ CBP Discussion Paper. Like other innovation voucher schemes, the Dutch scheme aimed to stimulate linkages between universities and SMEs, so it is not strictly comparable with Creative Credits. 62. As part of the same exercise we also investigated whether the small cash incentives provided to firms to help encourage survey response had biased these results. No evidence of any significant bias could be identified (Table A4.3). 63. Similar differences are evident after six months, with firms anticipating all the benefits from their Creative Credits projects to fall within six months having an average growth rate of 5.8 per cent, compared with 1.9 per cent for those anticipating longer-term benefits (t=1.663, ρ=0.099). 64. Goffin, K. and Micheli, P. (2010) Maximizing the value of industrial design in new product development. ‘Research-Technology Management.’ 53, pp. 29-37. Song, X. M., Montoya-Weiss, M. M. and Schmidt, J. B. (1997) Antecedents and consequences of cross functional co-operation: a comparison of R&D, manufacturing and marketing perspectives. ‘Journal of Product Innovation Management.’ 14, pp. 35-47. 65. The results in Table A4.3 suggest that there were no statistically significant biases. 66. Bakhshi, H., Edwards, J., Roper, S., Scully, J. and Shaw, D. (2011), Creating Innovation in SMEs: Evaluating the Short-Term 75 CREATIVE CREDITS A randomized controlled industrial policy experiment Effects of the Creative Credits Pilot.’ NESTA working paper. London: NESTA. 67. Duflo, E., Glennerster, R. and Kremer, M. (2006) ‘Using Randomization in Development Economics Research: A Toolkit.’ NBER Technical Working Paper 333. Cambridge, MA: NBER. 68. Pages 20-21, Reiner, C. (2011) Evaluating innovation policies by chance? The case for randomized R&D-programme evaluation. ‘Plattform - Forschungs- und Technologieevaluierung.’ 37, 15-28. 69. The term was coined in 1950 by Henry Landsberger when analysing older experiments from 1924-1932 at the Hawthorn Works (a Western Electric factory outside Chicago). 70. Comparing the treatment and control groups’ use of different types of business support in Survey 4 yields the following: for ‘other’ business support χ2(2)= 0.040, p=0.98; for accountants, bankers and lawyers χ2(2)= 1.080, p=0.583; for private consultants, χ2(2)= 0.456, p=0.796, and for other innovation vouchers, χ2(2)= 0.393, p=0.822. In all cases the differences between the two groups were not statistically significant. 71. See also: Hewitt-Dundas, N. and Roper, S. (2011) Creating advantage in peripheral regions: the role of publicly funded R&D centres. ‘Research Policy.’ 40, 6, 832-841. 72. Stoker, G. and John, P. (2008) Design experiments: engaging policy makers in the search for evidence about what works. ‘Political Studies.’ 57 (2), 356-373. 73. Cornet, M., Vroomen, B. and van der Steeg, M. (2006) ‘Do innovation vouchers help SMEs to cross the bridge towards science?’ No 58 CBP Discussion Paper CBP, p. 7. 74. Biggar Economics (2011) ‘Evaluation of the first phase of the Scottish funding council innovation voucher scheme.’ April 2010, p. 31. 75. Cornet et al. 2006, op. cit. 76. Good, B. and Tiefenthaler, B. (2011) Innovation voucher- small is beautiful. ‘Plattform.’ 37, December 2011, p. 45-53. 77. Van der Steeg, M. (2008) ‘The Dutch innovation voucher and evaluation issues.’ Presentation October 2008. Available at: http://www.nesta.org.uk/assets/documents/marc_van_der_steeg_presentation 78. http://www.nesta.org.uk/areas_of_work/public_services_lab/alliance_for_useful_evidence 79. (2009) ‘Innovation, Trade and Connectivity.’ Manchester: Manchester Independent Economic Review. Available at: http:// www.manchester-review.org.uk/projects/view/?id=719 80. In the event that a treatment SME proposed to use their Credit for a project that did not satisfy the eligibility criteria, the SME was asked to go back and rework their proposal. If the SME could not identify an eligible project within the deadlines the Credit was allocated, again on a random basis, to a reserve business. 81. Cornet, M., Vroomen, B. and van der Steeg, M. (2006) ‘Do innovation vouchers help SMEs to cross the bridge towards science?’ No 58 CBP Discussion Paper. 82. Glaser, B. and Strauss, A. (1967) ‘The Discovery of Grounded Theory: Strategies for Qualitative Research.’ Chicago: Aldine Publishing Company. Glaser, B. (1978) ‘Theoretical Sensitivity: Advances in the Methodology of Grounded Theory.’ Mill Valley: The Sociology Press. Glaser, B. (1998) ‘Doing Grounded Theory: Issues and Discussions.’ Mill Valley: The Sociology Press. Strauss, A. (1987) ‘Qualitative Analysis for Social Scientists.’ Cambridge: Cambridge University Press. Katz, J. (1983) ‘A Theory of Qualitative Methodology: The Social System of Analytical Fieldwork.’ In Emerson, R. (Ed) ‘Contemporary Field Research: A Collection of Readings.’ Boston: Little Brown Company. 83. Lee, N. and Lings, I. (2008) ‘Doing Business Research: a guide to theory and practice.’ SAGE Publications Ltd. 84. Denzin, N. (1970) Introduction. In Denzin, N. (Ed.) ‘Sociological Methods.’ (pp. 471-475). London: Butterworth & Co. Ltd. 85. Yin, R. (2003) ‘Case Study Research Design and Methods.’ California: SAGE Publications Inc. 86. Bryman, A. (2004) Triangulation. In Lewis-Beck, M. Bryman, A. and Liao, T. (Eds.) (2004) ‘The Sage Encyclopedia of Social Science Research Methods.’ California: Sage Publications Inc.; Mays, N. and Pope, C. (1995) Rigour and qualitative research. ‘British Medical Journal.’ 311, 109-112. 87. Miller, R. and Brewer, J. (Eds.) (2003) ‘The A-Z of Social Research.’ London: Sage Publications Ltd. 88. Gillham, B. (2000) ‘Case Study Research Methods.’ London: Continuum. 89. Bryman A., and Bell, E. (2007) ‘Business Research Methods.’ New York: Oxford University Press. 90. Crowley, C., Harré, R. and Tagg, C. (2002) Qualitative research and computing: Methodological issues and practices in using QSR NVivo and NUD*IST. ‘International Journal of Social Research Methodology.’ 5(3), 193-197. 91. Bryman, A. (2001) ‘Social Research Methods.’ Oxford: Oxford University Press. 92. FAME or Financial Analysis Made Easy is an online version of the UK Companies House data sold commercially by Bureau Van Dyke. It contains information on all UK-registered firms. 93. See Bakhshi et al. (2011), op. cit. for more details. 94. Bakhshi et al. (2011) op. cit. 95. Bakhshi et al. (2011), op. cit. 96. Department for Business Innovation and Skills (2009) ‘Research to improve the assessment of additionality.’ Occasional Paper 1, October 2009. 97. Source: op cit., Table 4.2. 98. Source: op cit., Table 7.2. 76 THE VENTURE CROWD Crowdfunding equity investment into business Nesta 1 Plough Place London EC4A 1DE research@nesta.org.uk www.twitter.com/nesta_uk www.facebook.com/nesta.uk www.nesta.org.uk June 2013 Nesta Operating Company. English charity no. 7706036. Scottish charity no. SC042833.