Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

Lunchtime Talk: Evaluating design enabled innovations

•Download as PPTX, PDF•
•
The Tavistock Institute of Human Relations
The Tavistock Institute of Human Relations

Special session: evaluating design enabled innovations – towards a common impact evaluation methodology

1 of 33
Evaluating design enabled innovations. Lessons from Designscapes for a common impact evaluation methodology Dr Joe Cullen and Dr Kerstin Junge Tavistock Institute of Human Relations 22.9.2021
Designscapes
• Funded by the European Union’s research excellence programme Horizon 2020 (2017-2021) • 12 organisations, in 10 European countries – academia, consulting, public sector • To encourage the uptake and scaling of Design enabled Innovations by enterprises, start-up companies, public authorities, and other urban stakeholders Design Innovation Urban Designscapes
Desigscnapes Evaluation Designscapes research activities Designscapes funding Policy recommenda- tions Designscapes capacity building Designgscapes communication Designscapes activities
Designscapes evaluation • To support the Designscapes’ overall vision, mission, objectives and outcomes • To develop a scalable impact evaluation methodology Developmental evaluation Contributing to the project design and its development. Process evaluation Monitoring project progress and assessing the effectiveness and efficiency of the project delivery. Outcome evaluation Assessing project outcomes and impacts.
Designscapes evaluation 1. Does Designscapes support inclusiveness, and reduce inequalities in citizens’ access to innovation? 2. Does Designscapes support co-creation, and does co-creation in turn lead to successful innovation? 3. Does design-led innovation improve performance and efficiency in the commercial and public sector - by addressing the challenges cities face now and in the future - and hence improve competitiveness? 4. What additional contribution does the urban context make? • Theory based evaluation design (theory of change and contribution analysis) • Mixed method design, e.g.: • KPI design and monitoring • Partner interviews, online surveys and workshops • Case studies (multiple case study design) • Longitudinal stakeholder survey • Cost consequence analysis • Regression modelling • Systems mapping
Emerging findings: Does design-led innovation improve performance and efficiency in the commercial and public sector - by addressing the challenges cities face now and in the future - and hence improve competitiveness?
Problem / context Activites Outputs Outcomes Impacts DEI is potentially an excluding process, so those who might benefit or have something to contribute don’t participate. [ inclusion is not a key element of design generally. Always the same people involved] The urban environment presents a specific manifestation of wicked problems and is at a scale where you can develop solutions. Top down perspective: “Many public sector organisations (role of citizens) are missing out on DEI.” Things like organisational structure, role and purpose of public sector organisations and attitudes towards new ways of working can get in the way of using DEI. The public sector is not as agile as the private sector and has different challenges and roles in society. There is fear of experimentation and therefore a lot of the more innovative or creative methods are not taken up. Lack know how and experience in certain areas of DEI. Bad experiences with using DEI and challenges around adopting new solutions on policy side. Currently not many lcoal policy documents that include DEI. Design capabilities are not diffused in the urban context (ie many different actors in the urban environment know about DEI and can use it) and cannot capture the potential of urban contexts to generate innovation. Problem to not utilise design has a geographic dimension: much more evolved in society and business of northern Europe than in other parts Businesses, especially SMEs, miss out on the potential to utilise DEI in the urban context as a source for improving efficiency and stimulating growth”. Maybe business is less competent at involving people with disadvantages. Concepts of design agency and design enabled innovation are used differently by different types of organisations. Design capabilities are different in different geographic and organisational contexts. There is a lack of evidence on the effectiveness of service or strategic design and innovation (with some exceptions, e.g DK). There is little use of evaluation. Policy forums and mid-term conference Application form design / content and application process for pilots Development and offering of a training programme to round 2 and round 3 pilots, online and face to face. (ultimately opened to everybody at the end of the project) An increase in public sector effectiveness and private sector competitiveness, and sustaining innovation capabilities and processes Integration of design and user-driven innovation into innovation policies and support mechanisms, including funding programme Global problems as manifested at urban level improve Improvements to global priorities at urban level: Contribution to Design enabled Innovation sustainable urban quality, economic and social values; sustainable innovation capability at urban level. An improvement in evaluation tools, skills and capacity to evaluate the effectiveness of design in the innovation process Top down: local or national policies are changing to include DEI and / or DEI are introduced (more widely) into policy at local level. Bottom up: design capabilities more diffused in urban environments: mabny different actors’ capabilities are raised. An open-ended and tranferrable, global ecosystem of design-enabled innovators in urban environments that can be copied and leveraged by other cities. Cities have increased their design capabilities. Bottom up design policy making and project initiatition encouraged Increasing political awareness and interest on DEI at urban level, plus effectiveness - how can you show use / relevance of DEI in local contexts? Emergence of a strategy / policy to support the transformation of urban ecosystems into DEI ecosystems in other European (not Designscapes) cities. Different facebook groups and communities linked together, building a large community of DEI practitioners. 1 open Designscapes facebook group 1 closed Designscapes facebook group X number of participants X number of facebook and other groups linked together Citizens engaged in policy making processes; greater awareness among policy makers about opportunities and challenges of doing this Greater confidence among urban / regional policy makers to experiment with innovate DEI tools Integration of Green and White paper contents into programmes and / or policies by policy makers at urban level Pilot teams advance new solutions to alleviate wicked problems in the urban space Pilot teams improve and / or sustain their innovation capacity /capability at urban level Increase in awareness about diffuse design skills as a result of completing the application form especially among those that did not start from a design process perspective. Greater familiarity among designscapes audiences with state of the art then hopefully they will be more willing to adopt innovative tools. Increased awareness of benefits of ToC and evaluation in the sector Better evidence on benefits DEI in urban environments Applicants create their own ideas / solutions specific to their urban environments. Designscapes Green paper Designscapes White paper Designscapes funds 100 pilots: 50 feasibility studies, 40 prototypes, 10 scalability proofed projects Designscapes developes a public Design toolbox describing 30 tools, instruments and methods to enhance DEI in cities X number of Designscapes events X number of participants Evaluation framework (D2.1) At the end: a tested and “universal” evaluation approach Community of DE innovators through facebook group consisting of those funded by the project initiatlly. This might expand after the project. Solutions and policy recommendations. Policy examples implementing DEI in practice at city level Evaluation reports Designscapes mentors allocated to round 2+3 funded pilots Co-creation and co-construction of events and strategies with other player Facebook group for funded pilots as first stage for community building Other activities needed to create ecosystem, connecting projects and initiatives. Thinking ongoing on how to do this, possibly by creating a network of mentors. Creating physical spaces for DEI practitioners / Designscapes stakeholders to connect. Designscapes develops 3 rounds of an open call call, addressing everyone in Europe and stressing user co-creation and social inclusion Dissemination activities: toolbox, blog, journal articles Implementing evaluation at Designscapes and pilot level Blogs on evaluation and DEI Evaluation / ToC training Researching policy landscape feeding into agenda for Valencia policy forum Connecting Design innovators/projects (online - e.g. facebook- and / or through networking events / activities?); also providing opportunities to connect and showcase their work Invite pilots to policy fora Capacity building activities and working collaboratively (with policy makers and others) to understand gaps in capacities Iterative toolbox matching needs with tools (WP4), making sure needs are identified Match initiatives: bringing together organisations and matching the ‘right’ Designscapes partner to support ‘skilling up’ Assumptions: Public and private sector organisations use DEI differently and have different capabilities. The project focuses on the urban system not specific (private) businesses Are we able to provide enough funding or support or resources to sustain a community or availability of financial incentives ? Are we able to build a community? Are we able to fund diversity (e.g. geographic) and involve disadvantaged people? Are we able to exploit the innovation potential of urban environments? Are we able to develop consensus among policy makers? Assumptions: Designscapes is able to fund geographically diverse projects. We are funding the ‘right’ projects to provide data and epxerience helping to achieve the outcomes. Projects and stakeholders access the training, support and tools provided Risk: reality of funded pilots might not help us achieve outcomes; capacities in peripheral regions limited. Assumption: we are able to ‘attract’ or develop interest among policy audiences. Risk: challenge of communicating Designscapes vision due to the multitude of terminology used. Webinars and coffee mornings
Global problems as manifested at urban level improve. Design enabled Innovation creates economic and social value The urban environment presents a specific manifestation of wicked problems and is at a scale where you can develop solutions. Design capabilities are not diffused in the urban context (ie many different actors in the urban environment know about DEI and can use it) and cannot capture the potential of urban contexts to generate innovation. Pilot teams advance these solutions alleviate wicked problems in the urban space Applicants create their own solutions specific to their urban environments. Designscapes funds 100 pilots: 50 feasibility studies, 40 prototypes, 10 scalability proofed projects Designscapes develops 3 rounds of an open call call, addressing everyone in Europe and stressing user co- creation and social inclusion DEI creates sustainable innovation capability at urban level. Effectiveness and efficiency causal strand Assumptions: • Designscapes is able to fund geographically diverse projects. • We are funding the ‘right’ projects to provide data and experience helping to achieve the outcomes. • Projects have the potential to be further developed Risks: • Reality of funded pilots might not help us achieve Outcomes; • Capacities in peripheral regions limited M1: Designscapes funding instrument M2: Performance and efficiency
Global problems as manifested at urban level improve. Design enabled Innovation creates economic and social value The urban environment presents a specific manifestation of wicked problems and is at a scale where you can develop solutions. Design capabilities are not diffused in the urban context (ie many different actors in the urban environment know about DEI and can use it) and cannot capture the potential of urban contexts to generate innovation. Pilot teams advance these solutions alleviate wicked problems in the urban space Applicants create their own solutions specific to their urban environments. Designscapes funds 100 pilots: 50 feasibility studies, 40 prototypes, 10 scalability proofed projects Designscapes develops 3 rounds of an open call call, addressing everyone in Europe and stressing user co- creation and social inclusion DEI creates sustainable innovation capability at urban level.
3 rounds of an open call • Applicants to focus on: • Urban dimension conducive to use of design • Using design to find more effective responses to big challenges • DEI projects that can be transferred to other contexts (including to less innovative contexts) • Feasibility studies, prototypes, scalability pilots • Relatively modest funding: • between EUR 5,000 (feasibility studies) and • EUR 25,000 (prototypes and scalability pilots) “We liked the approach in terms of design and innovation and urban, and we think it was also made for creative people (not super scientific and bureaucratic as some other calls), we liked the process approach where the outcome was not the main focus – that fitted well in a prototytpe call; the division of the call was intriguing – to have the option to proceed further, feels understood for a creative process” [project interview, round 2]. “When it comes to Swedish funding, any calls so clearly focused on using design for urban sustainability is unique. That is what we’re trying to do, design something that fits into what society today, (…).” [project interview, round 3]
99 pilots funded 0 10 20 30 40 South West North East Location of Designscapes pilots • 50 % of successful pilots would not have gone ahead without Designscapes funding at feasibility stage • 3% of successful applicants would not have gone ahead without Designscapes funding at prototype stage 11 projects received funding in two rounds 3 projects received funding in all three rounds Majority of survey respondents not applying to round 3 had continued to develop their project in some form (as designed, small changes, large changes)
Solutions to urban problems 46% 65% 40% 37% 62% 59% 0% 20% 40% 60% 80% Climate Change Social Exclusion Economic Crisis Crisis of Democracy Low Quality of Life Crisis of Values Average priority areas across all stages by success Successful •Lived experience •Users as content experts •Professional experience / existing product •Shared knowledge between users and project leads •Users as providers of information •Research: •Users as providers of information 36 % of successful applicants at feasibility stage were looking for funding for a new idea Self-reported perceptions that funded design enabled innovations unique in their country or in their focus (e.g. social dimension)
Mechanism M1: Designscapes funding instrument. Designscapes provides a flexible funding instrument to which applicants respond by applying design thinking and tools to generating and developing new and innovative ideas to tackle problems linked to their urban environment.
Using design methods 40% 50% 60% 80% 90% 90% 66% 78% 56% 90% 63% 68% 78% 68% 46% 84% 74% 74% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Design methods to generate ideas Prototyping methods Usability Evaluation methods Participatory design Use of design thinking methodologies Participatory process to validate ideas Design methods proposed by successful projects by stage Stage 1 Stage 2 Stage 3 2578 citizens 54 public administrations 103 not for profits 47 for profit enterprises
• 81% ‘quite likely’ they will take prototype to market (round 2) • 30% greater readiness to introduce innovation to new markets (round 3) Benefits of co-creation using design thinking Bringing different people and perspectives together enriches final output and outcomes Working through a problem with stakeholders so the solution works in a given context Understanding user needs and market interest in the innovation Refining the innovation and creating ownership by users “It wasn’t, ‘Take this project and see what you think of it and how you would improve it’; it was, ‘Here you have the square – what would you do to it, what would you remove, what would you keep, and what would you add?’ In this sense, it was more enriching.” [citizen interview, scalability pilot]
Value of Designscapes grant “It has lowered our thresholds to incorporating design methodology in our work significantly, so we will now use it much more frequent and in a more self-confident and relaxed manner” [Case study interview, round 2] We used design tools that allowed us to better understand the calibration of the project according to the different types of people/companies that might be interested in using our technology and led us to diversify the technology we developed according to the needs of those who might use it. [survey, round 3] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Stimulating new ways of working Ability to generate evidence on impact of the innovation Using design thinking and tools To what extent has particiaption in Designscapes enabled you to: round 3 round 2
Sustainable innovation capacity – behavioural additionality effects Professional network gains Round 1: 80%* Round 2: 55% Round 3:77% * Statistically significant difference between funded and non-funded pilots 0 5 10 15 20 Other Desigsncapes project… Potential funders Potential customers Experts in design thinking… Higher education institutions Public administrations Maintaining links after end of Designscapes funding “It has lowered our thresholds to incorporating design methodology in our work significantly, so we will now use it much more frequent and in a more self- confident and relaxed manner” [Case study interview, round 2] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Stimulating new ways of working Ability to generate evidence on impact of the innovation Using design thinking and tools To what extent has particiaption in Designscapes enabled you to: round 3 round 2
Economic value – business impact Additional funding generated by successful pilots • 21 % of round 2 pilots: • EUR 74,428 (average) • higher than unsuccessful applicants • 23% of round 2 pilots • EUR 16,170 (average) • lower than unsuccessful round 3 pilots (though fewer of these were able to secure this additional funding in the first place)
Social impact • Projects more confident in achieving social inclusion and quality of life impacts. • Less confident in economic and climate change impacts. 0 1 2 3 4 5 6 7 8 9 10 Climate impacts Social inclusion impacts Economic impacts Quality of life impacts Crisis of democracy impacts Crisis of values impacts Not at all To a small extent To a moderate extent To a great extent Completely
Mechanism M2: Performance and efficiency. By locating their pilots in the urban context and using design thinking and tools, end-users creativity is mobilised and combined with pilot teams’ technical and professional expertise. This leads to new solutions or the adaptation of existing ones which meet user needs better and hence get adopted. This generates business and social value.
Towards a common impact methodology
The Co-Creation Call Theory of Change user needs responsiveness design thinking increased performance & efficiency competitiveness Presenting problem: Europe needs to become more competitive – by identifying and capitalising on untapped sources of growth and employment User involvement central to innovation process Design thinking and co-creation encourages user involvement Leads to service & business model innovation, then increased performance & efficiency Leads to increased competitiveness of public, private and third sectors
Application areas and evaluation questions addressed by CIM Application area Evaluation question Examples from the Designscapes evaluation 1. User benefits and business impacts What user benefits and business impacts are associated with design- related policies and programmes? Calculating the economic and social benefits of the funding provided to projects by Designscapes using Cost Consequence Analysis (CCA) 2.Relationship between design and innovation What contribution does design thinking make to innovation? Using Regression Analysis to predict the likely effects of the application of design thinking methods and tools to increase innovation 3.Value-creating networks and efficiency and competition How does DEI support value-creating networks and in what ways do they increase efficiency and competitiveness? The contribution of the Designscapes financial instrument to creating value-driven networks and their impact on the efficiency and competitiveness of funded projects
CIM Framework: Theory of Change
Why use Theory of Change in DEI Evaluation? Not only does Theory of Change provide a holistic view of the vision of the intervention and the interconnections that make up this vision, it situates that vision within the wider social context. This helps to better articulate the contribution of the intervention to the broader societal impacts of DEI.
Applicable at different scales Scale Presenting Problem Activities Outputs Outcomes Impacts Organisational (single actor) Small atelier making customised designer-wear is experiencing mounting production costs and decreasing profit margins Owners join EU-wide textile and clothing innovation network Installation of laser cutting and 3D printing equipment Production time per output unit decreased by 15% Profit increased by 12% after 1 year Project (partnership) Across Europe, cities and towns are losing their cultural heritage artefacts. 10 partners in 6 EU countries involved in using design thinking to explore factors affecting cultural heritage loss Cultural Academy set up And delivers 10 international workshops and 10 online webinars, attended by 650 people in total Over 75% of educational programme participants report increased awareness of cultural heritage Over 60% of city administrations report increased attendance at heritage events in their cities Community/regional Urban spaces in the region have untapped potential to develop innovative and sustainable farming methods Research on feasibility of urban ‘vertical farms’, including use of AI and Industry 4.0 tools Vertical Farm Prototype in 8 cities across the region 5,000 customers purchase produce from vertical farms in 6 month trial period 5% of derelict urban properties rehabilitated to farming use in region National Rates of unemployment of low skilled young people are increasing across the country in all cities. Develop a national jobs database co-ordinated nationwide 6,000 employment service staff participate in Training programme for employment service staff 55% of users placed in jobs after 1 year of programme operation Reduction in youth unemployed rate of 15% nationally Trans-national The EU needs to become more competitive by capitalising untapped sources of growth through co-creation EC launches 5 year programme to support design-enabled innovation through funding pilot projects Design-enabled Innovation funding instrument (DEIFI) launched through ESF Participating organisations introduce innovations into their practice Behavioural additionality analysis shows net increase in efficiency of 12% across EU
And for different actors Actor Presenting Problem Activities Outputs Outcomes Impacts SME A small specialist furniture maker is experiencing falling demand due to high costs and changing consumer attitudes Apply design thinking to ‘outside the box’ business re-structuring strategies Prototype Circular Economy furniture service for landlords and tenants Reduction in volume of furniture thrown away Contribution to reduction in use of landfill sites Public Administratio n Public administrations have a duty to support lonely but they lack resources Design platform and App to help neighbours connect with each other Anti-isolation network set up involving 80 public administrations Over 2,000 people use the platform and App in 6 month period Self-reported loneliness and isolation levels have reduced by 23%
ToC • Design Theory of Change • Fit to scale and audience Plan • Implementation Plan • Purposes, resources, actions Measure • Design indicators • Select and apply MoV Interpret • Analyse and integrate results • Triangulation, Mechanisms, Contribution analysis 6 Presenting problem Inputsand activities Outputs Outcomes Impacts STEP 7: What are the assumptionsthat link each step? The things you do The changes you wish to see Step 1 Step 2 Steps 4-5 Step 3 STEP 6: IFwe invest these resourcesand deliver these activities, THEN these outputs will be produced, which will trigger these outcomesand THEN lead to our impact. SMART SPICED Methodology Methods Tools 2 BIPOLAR DISORDER REDUCED MOOD SWINGS GSK3B BDNF SEROTONIN LITHIUMONLY WORKS FOR ONE THIRD OF PATIENTS WE DON’T REALLY KNOWWHY OR HOW
Application area 1: User benefits and business impacts SROI = (social impact value – initial investment amount) ÷ initial investment amount x 100% Stage What this involves 1. Mapping Produce an Impacts Map showing the expected impacts of the changes realised by the intervention 2. Measurable financial consequences Select and quantify the impacts and outcomes that have measurable financial consequences attached 3. Non-financial consequences Select and evaluate the impacts and outcomes that have measurable non- financial consequences attached 4. Non-recurrent costs Identify and quantify non-recurrent costs 5. Analysis Compare cost consequences of alternatives and review results
Application area 2: Relationship between design and innovation  Providing an explanation of the causal relationships between variables and how they lead to observed outcomes  Providing a measure of the relative strength of the contribution each variable of interest makes to the observed outcomes  Making predictions about the likely effects on outcomes of interest if the values of the contributing variables are changed  Producing simulation models based on ‘what-if’ scenarios – for example if you increase the level of investment design-enabled innovation, what is the likely effect on public sector competitiveness?
Application Area 3: Value-creating networks and their contribution to efficiency and competition • Social Network Analysis (SNA) maps and evaluates the effects of interconnections between actors in a network, including more complex networks that make up eco- systems • Main focus is on the relationships between network members – but also maps the characteristics of the members • Captures longitudinal evolution of network/ecosystem over time
Questions and discussion

More Related Content

Lunchtime Talk: Evaluating design enabled innovations

  • 1. Evaluating design enabled innovations. Lessons from Designscapes for a common impact evaluation methodology Dr Joe Cullen and Dr Kerstin Junge Tavistock Institute of Human Relations 22.9.2021
  • 3. • Funded by the European Union’s research excellence programme Horizon 2020 (2017-2021) • 12 organisations, in 10 European countries – academia, consulting, public sector • To encourage the uptake and scaling of Design enabled Innovations by enterprises, start-up companies, public authorities, and other urban stakeholders Design Innovation Urban Designscapes
  • 5. Designscapes evaluation • To support the Designscapes’ overall vision, mission, objectives and outcomes • To develop a scalable impact evaluation methodology Developmental evaluation Contributing to the project design and its development. Process evaluation Monitoring project progress and assessing the effectiveness and efficiency of the project delivery. Outcome evaluation Assessing project outcomes and impacts.
  • 6. Designscapes evaluation 1. Does Designscapes support inclusiveness, and reduce inequalities in citizens’ access to innovation? 2. Does Designscapes support co-creation, and does co-creation in turn lead to successful innovation? 3. Does design-led innovation improve performance and efficiency in the commercial and public sector - by addressing the challenges cities face now and in the future - and hence improve competitiveness? 4. What additional contribution does the urban context make? • Theory based evaluation design (theory of change and contribution analysis) • Mixed method design, e.g.: • KPI design and monitoring • Partner interviews, online surveys and workshops • Case studies (multiple case study design) • Longitudinal stakeholder survey • Cost consequence analysis • Regression modelling • Systems mapping
  • 7. Emerging findings: Does design-led innovation improve performance and efficiency in the commercial and public sector - by addressing the challenges cities face now and in the future - and hence improve competitiveness?
  • 8. Problem / context Activites Outputs Outcomes Impacts DEI is potentially an excluding process, so those who might benefit or have something to contribute don’t participate. [ inclusion is not a key element of design generally. Always the same people involved] The urban environment presents a specific manifestation of wicked problems and is at a scale where you can develop solutions. Top down perspective: “Many public sector organisations (role of citizens) are missing out on DEI.” Things like organisational structure, role and purpose of public sector organisations and attitudes towards new ways of working can get in the way of using DEI. The public sector is not as agile as the private sector and has different challenges and roles in society. There is fear of experimentation and therefore a lot of the more innovative or creative methods are not taken up. Lack know how and experience in certain areas of DEI. Bad experiences with using DEI and challenges around adopting new solutions on policy side. Currently not many lcoal policy documents that include DEI. Design capabilities are not diffused in the urban context (ie many different actors in the urban environment know about DEI and can use it) and cannot capture the potential of urban contexts to generate innovation. Problem to not utilise design has a geographic dimension: much more evolved in society and business of northern Europe than in other parts Businesses, especially SMEs, miss out on the potential to utilise DEI in the urban context as a source for improving efficiency and stimulating growth”. Maybe business is less competent at involving people with disadvantages. Concepts of design agency and design enabled innovation are used differently by different types of organisations. Design capabilities are different in different geographic and organisational contexts. There is a lack of evidence on the effectiveness of service or strategic design and innovation (with some exceptions, e.g DK). There is little use of evaluation. Policy forums and mid-term conference Application form design / content and application process for pilots Development and offering of a training programme to round 2 and round 3 pilots, online and face to face. (ultimately opened to everybody at the end of the project) An increase in public sector effectiveness and private sector competitiveness, and sustaining innovation capabilities and processes Integration of design and user-driven innovation into innovation policies and support mechanisms, including funding programme Global problems as manifested at urban level improve Improvements to global priorities at urban level: Contribution to Design enabled Innovation sustainable urban quality, economic and social values; sustainable innovation capability at urban level. An improvement in evaluation tools, skills and capacity to evaluate the effectiveness of design in the innovation process Top down: local or national policies are changing to include DEI and / or DEI are introduced (more widely) into policy at local level. Bottom up: design capabilities more diffused in urban environments: mabny different actors’ capabilities are raised. An open-ended and tranferrable, global ecosystem of design-enabled innovators in urban environments that can be copied and leveraged by other cities. Cities have increased their design capabilities. Bottom up design policy making and project initiatition encouraged Increasing political awareness and interest on DEI at urban level, plus effectiveness - how can you show use / relevance of DEI in local contexts? Emergence of a strategy / policy to support the transformation of urban ecosystems into DEI ecosystems in other European (not Designscapes) cities. Different facebook groups and communities linked together, building a large community of DEI practitioners. 1 open Designscapes facebook group 1 closed Designscapes facebook group X number of participants X number of facebook and other groups linked together Citizens engaged in policy making processes; greater awareness among policy makers about opportunities and challenges of doing this Greater confidence among urban / regional policy makers to experiment with innovate DEI tools Integration of Green and White paper contents into programmes and / or policies by policy makers at urban level Pilot teams advance new solutions to alleviate wicked problems in the urban space Pilot teams improve and / or sustain their innovation capacity /capability at urban level Increase in awareness about diffuse design skills as a result of completing the application form especially among those that did not start from a design process perspective. Greater familiarity among designscapes audiences with state of the art then hopefully they will be more willing to adopt innovative tools. Increased awareness of benefits of ToC and evaluation in the sector Better evidence on benefits DEI in urban environments Applicants create their own ideas / solutions specific to their urban environments. Designscapes Green paper Designscapes White paper Designscapes funds 100 pilots: 50 feasibility studies, 40 prototypes, 10 scalability proofed projects Designscapes developes a public Design toolbox describing 30 tools, instruments and methods to enhance DEI in cities X number of Designscapes events X number of participants Evaluation framework (D2.1) At the end: a tested and “universal” evaluation approach Community of DE innovators through facebook group consisting of those funded by the project initiatlly. This might expand after the project. Solutions and policy recommendations. Policy examples implementing DEI in practice at city level Evaluation reports Designscapes mentors allocated to round 2+3 funded pilots Co-creation and co-construction of events and strategies with other player Facebook group for funded pilots as first stage for community building Other activities needed to create ecosystem, connecting projects and initiatives. Thinking ongoing on how to do this, possibly by creating a network of mentors. Creating physical spaces for DEI practitioners / Designscapes stakeholders to connect. Designscapes develops 3 rounds of an open call call, addressing everyone in Europe and stressing user co-creation and social inclusion Dissemination activities: toolbox, blog, journal articles Implementing evaluation at Designscapes and pilot level Blogs on evaluation and DEI Evaluation / ToC training Researching policy landscape feeding into agenda for Valencia policy forum Connecting Design innovators/projects (online - e.g. facebook- and / or through networking events / activities?); also providing opportunities to connect and showcase their work Invite pilots to policy fora Capacity building activities and working collaboratively (with policy makers and others) to understand gaps in capacities Iterative toolbox matching needs with tools (WP4), making sure needs are identified Match initiatives: bringing together organisations and matching the ‘right’ Designscapes partner to support ‘skilling up’ Assumptions: Public and private sector organisations use DEI differently and have different capabilities. The project focuses on the urban system not specific (private) businesses Are we able to provide enough funding or support or resources to sustain a community or availability of financial incentives ? Are we able to build a community? Are we able to fund diversity (e.g. geographic) and involve disadvantaged people? Are we able to exploit the innovation potential of urban environments? Are we able to develop consensus among policy makers? Assumptions: Designscapes is able to fund geographically diverse projects. We are funding the ‘right’ projects to provide data and epxerience helping to achieve the outcomes. Projects and stakeholders access the training, support and tools provided Risk: reality of funded pilots might not help us achieve outcomes; capacities in peripheral regions limited. Assumption: we are able to ‘attract’ or develop interest among policy audiences. Risk: challenge of communicating Designscapes vision due to the multitude of terminology used. Webinars and coffee mornings
  • 9. Global problems as manifested at urban level improve. Design enabled Innovation creates economic and social value The urban environment presents a specific manifestation of wicked problems and is at a scale where you can develop solutions. Design capabilities are not diffused in the urban context (ie many different actors in the urban environment know about DEI and can use it) and cannot capture the potential of urban contexts to generate innovation. Pilot teams advance these solutions alleviate wicked problems in the urban space Applicants create their own solutions specific to their urban environments. Designscapes funds 100 pilots: 50 feasibility studies, 40 prototypes, 10 scalability proofed projects Designscapes develops 3 rounds of an open call call, addressing everyone in Europe and stressing user co- creation and social inclusion DEI creates sustainable innovation capability at urban level. Effectiveness and efficiency causal strand Assumptions: • Designscapes is able to fund geographically diverse projects. • We are funding the ‘right’ projects to provide data and experience helping to achieve the outcomes. • Projects have the potential to be further developed Risks: • Reality of funded pilots might not help us achieve Outcomes; • Capacities in peripheral regions limited M1: Designscapes funding instrument M2: Performance and efficiency
  • 10. Global problems as manifested at urban level improve. Design enabled Innovation creates economic and social value The urban environment presents a specific manifestation of wicked problems and is at a scale where you can develop solutions. Design capabilities are not diffused in the urban context (ie many different actors in the urban environment know about DEI and can use it) and cannot capture the potential of urban contexts to generate innovation. Pilot teams advance these solutions alleviate wicked problems in the urban space Applicants create their own solutions specific to their urban environments. Designscapes funds 100 pilots: 50 feasibility studies, 40 prototypes, 10 scalability proofed projects Designscapes develops 3 rounds of an open call call, addressing everyone in Europe and stressing user co- creation and social inclusion DEI creates sustainable innovation capability at urban level.
  • 11. 3 rounds of an open call • Applicants to focus on: • Urban dimension conducive to use of design • Using design to find more effective responses to big challenges • DEI projects that can be transferred to other contexts (including to less innovative contexts) • Feasibility studies, prototypes, scalability pilots • Relatively modest funding: • between EUR 5,000 (feasibility studies) and • EUR 25,000 (prototypes and scalability pilots) “We liked the approach in terms of design and innovation and urban, and we think it was also made for creative people (not super scientific and bureaucratic as some other calls), we liked the process approach where the outcome was not the main focus – that fitted well in a prototytpe call; the division of the call was intriguing – to have the option to proceed further, feels understood for a creative process” [project interview, round 2]. “When it comes to Swedish funding, any calls so clearly focused on using design for urban sustainability is unique. That is what we’re trying to do, design something that fits into what society today, (…).” [project interview, round 3]
  • 12. 99 pilots funded 0 10 20 30 40 South West North East Location of Designscapes pilots • 50 % of successful pilots would not have gone ahead without Designscapes funding at feasibility stage • 3% of successful applicants would not have gone ahead without Designscapes funding at prototype stage 11 projects received funding in two rounds 3 projects received funding in all three rounds Majority of survey respondents not applying to round 3 had continued to develop their project in some form (as designed, small changes, large changes)
  • 13. Solutions to urban problems 46% 65% 40% 37% 62% 59% 0% 20% 40% 60% 80% Climate Change Social Exclusion Economic Crisis Crisis of Democracy Low Quality of Life Crisis of Values Average priority areas across all stages by success Successful •Lived experience •Users as content experts •Professional experience / existing product •Shared knowledge between users and project leads •Users as providers of information •Research: •Users as providers of information 36 % of successful applicants at feasibility stage were looking for funding for a new idea Self-reported perceptions that funded design enabled innovations unique in their country or in their focus (e.g. social dimension)
  • 14. Mechanism M1: Designscapes funding instrument. Designscapes provides a flexible funding instrument to which applicants respond by applying design thinking and tools to generating and developing new and innovative ideas to tackle problems linked to their urban environment.
  • 15. Using design methods 40% 50% 60% 80% 90% 90% 66% 78% 56% 90% 63% 68% 78% 68% 46% 84% 74% 74% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Design methods to generate ideas Prototyping methods Usability Evaluation methods Participatory design Use of design thinking methodologies Participatory process to validate ideas Design methods proposed by successful projects by stage Stage 1 Stage 2 Stage 3 2578 citizens 54 public administrations 103 not for profits 47 for profit enterprises
  • 16. • 81% ‘quite likely’ they will take prototype to market (round 2) • 30% greater readiness to introduce innovation to new markets (round 3) Benefits of co-creation using design thinking Bringing different people and perspectives together enriches final output and outcomes Working through a problem with stakeholders so the solution works in a given context Understanding user needs and market interest in the innovation Refining the innovation and creating ownership by users “It wasn’t, ‘Take this project and see what you think of it and how you would improve it’; it was, ‘Here you have the square – what would you do to it, what would you remove, what would you keep, and what would you add?’ In this sense, it was more enriching.” [citizen interview, scalability pilot]
  • 17. Value of Designscapes grant “It has lowered our thresholds to incorporating design methodology in our work significantly, so we will now use it much more frequent and in a more self-confident and relaxed manner” [Case study interview, round 2] We used design tools that allowed us to better understand the calibration of the project according to the different types of people/companies that might be interested in using our technology and led us to diversify the technology we developed according to the needs of those who might use it. [survey, round 3] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Stimulating new ways of working Ability to generate evidence on impact of the innovation Using design thinking and tools To what extent has particiaption in Designscapes enabled you to: round 3 round 2
  • 18. Sustainable innovation capacity – behavioural additionality effects Professional network gains Round 1: 80%* Round 2: 55% Round 3:77% * Statistically significant difference between funded and non-funded pilots 0 5 10 15 20 Other Desigsncapes project… Potential funders Potential customers Experts in design thinking… Higher education institutions Public administrations Maintaining links after end of Designscapes funding “It has lowered our thresholds to incorporating design methodology in our work significantly, so we will now use it much more frequent and in a more self- confident and relaxed manner” [Case study interview, round 2] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Stimulating new ways of working Ability to generate evidence on impact of the innovation Using design thinking and tools To what extent has particiaption in Designscapes enabled you to: round 3 round 2
  • 19. Economic value – business impact Additional funding generated by successful pilots • 21 % of round 2 pilots: • EUR 74,428 (average) • higher than unsuccessful applicants • 23% of round 2 pilots • EUR 16,170 (average) • lower than unsuccessful round 3 pilots (though fewer of these were able to secure this additional funding in the first place)
  • 20. Social impact • Projects more confident in achieving social inclusion and quality of life impacts. • Less confident in economic and climate change impacts. 0 1 2 3 4 5 6 7 8 9 10 Climate impacts Social inclusion impacts Economic impacts Quality of life impacts Crisis of democracy impacts Crisis of values impacts Not at all To a small extent To a moderate extent To a great extent Completely
  • 21. Mechanism M2: Performance and efficiency. By locating their pilots in the urban context and using design thinking and tools, end-users creativity is mobilised and combined with pilot teams’ technical and professional expertise. This leads to new solutions or the adaptation of existing ones which meet user needs better and hence get adopted. This generates business and social value.
  • 22. Towards a common impact methodology
  • 23. The Co-Creation Call Theory of Change user needs responsiveness design thinking increased performance & efficiency competitiveness Presenting problem: Europe needs to become more competitive – by identifying and capitalising on untapped sources of growth and employment User involvement central to innovation process Design thinking and co-creation encourages user involvement Leads to service & business model innovation, then increased performance & efficiency Leads to increased competitiveness of public, private and third sectors
  • 24. Application areas and evaluation questions addressed by CIM Application area Evaluation question Examples from the Designscapes evaluation 1. User benefits and business impacts What user benefits and business impacts are associated with design- related policies and programmes? Calculating the economic and social benefits of the funding provided to projects by Designscapes using Cost Consequence Analysis (CCA) 2.Relationship between design and innovation What contribution does design thinking make to innovation? Using Regression Analysis to predict the likely effects of the application of design thinking methods and tools to increase innovation 3.Value-creating networks and efficiency and competition How does DEI support value-creating networks and in what ways do they increase efficiency and competitiveness? The contribution of the Designscapes financial instrument to creating value-driven networks and their impact on the efficiency and competitiveness of funded projects
  • 26. Why use Theory of Change in DEI Evaluation? Not only does Theory of Change provide a holistic view of the vision of the intervention and the interconnections that make up this vision, it situates that vision within the wider social context. This helps to better articulate the contribution of the intervention to the broader societal impacts of DEI.
  • 27. Applicable at different scales Scale Presenting Problem Activities Outputs Outcomes Impacts Organisational (single actor) Small atelier making customised designer-wear is experiencing mounting production costs and decreasing profit margins Owners join EU-wide textile and clothing innovation network Installation of laser cutting and 3D printing equipment Production time per output unit decreased by 15% Profit increased by 12% after 1 year Project (partnership) Across Europe, cities and towns are losing their cultural heritage artefacts. 10 partners in 6 EU countries involved in using design thinking to explore factors affecting cultural heritage loss Cultural Academy set up And delivers 10 international workshops and 10 online webinars, attended by 650 people in total Over 75% of educational programme participants report increased awareness of cultural heritage Over 60% of city administrations report increased attendance at heritage events in their cities Community/regional Urban spaces in the region have untapped potential to develop innovative and sustainable farming methods Research on feasibility of urban ‘vertical farms’, including use of AI and Industry 4.0 tools Vertical Farm Prototype in 8 cities across the region 5,000 customers purchase produce from vertical farms in 6 month trial period 5% of derelict urban properties rehabilitated to farming use in region National Rates of unemployment of low skilled young people are increasing across the country in all cities. Develop a national jobs database co-ordinated nationwide 6,000 employment service staff participate in Training programme for employment service staff 55% of users placed in jobs after 1 year of programme operation Reduction in youth unemployed rate of 15% nationally Trans-national The EU needs to become more competitive by capitalising untapped sources of growth through co-creation EC launches 5 year programme to support design-enabled innovation through funding pilot projects Design-enabled Innovation funding instrument (DEIFI) launched through ESF Participating organisations introduce innovations into their practice Behavioural additionality analysis shows net increase in efficiency of 12% across EU
  • 28. And for different actors Actor Presenting Problem Activities Outputs Outcomes Impacts SME A small specialist furniture maker is experiencing falling demand due to high costs and changing consumer attitudes Apply design thinking to ‘outside the box’ business re-structuring strategies Prototype Circular Economy furniture service for landlords and tenants Reduction in volume of furniture thrown away Contribution to reduction in use of landfill sites Public Administratio n Public administrations have a duty to support lonely but they lack resources Design platform and App to help neighbours connect with each other Anti-isolation network set up involving 80 public administrations Over 2,000 people use the platform and App in 6 month period Self-reported loneliness and isolation levels have reduced by 23%
  • 29. ToC • Design Theory of Change • Fit to scale and audience Plan • Implementation Plan • Purposes, resources, actions Measure • Design indicators • Select and apply MoV Interpret • Analyse and integrate results • Triangulation, Mechanisms, Contribution analysis 6 Presenting problem Inputsand activities Outputs Outcomes Impacts STEP 7: What are the assumptionsthat link each step? The things you do The changes you wish to see Step 1 Step 2 Steps 4-5 Step 3 STEP 6: IFwe invest these resourcesand deliver these activities, THEN these outputs will be produced, which will trigger these outcomesand THEN lead to our impact. SMART SPICED Methodology Methods Tools 2 BIPOLAR DISORDER REDUCED MOOD SWINGS GSK3B BDNF SEROTONIN LITHIUMONLY WORKS FOR ONE THIRD OF PATIENTS WE DON’T REALLY KNOWWHY OR HOW
  • 30. Application area 1: User benefits and business impacts SROI = (social impact value – initial investment amount) ÷ initial investment amount x 100% Stage What this involves 1. Mapping Produce an Impacts Map showing the expected impacts of the changes realised by the intervention 2. Measurable financial consequences Select and quantify the impacts and outcomes that have measurable financial consequences attached 3. Non-financial consequences Select and evaluate the impacts and outcomes that have measurable non- financial consequences attached 4. Non-recurrent costs Identify and quantify non-recurrent costs 5. Analysis Compare cost consequences of alternatives and review results
  • 31. Application area 2: Relationship between design and innovation  Providing an explanation of the causal relationships between variables and how they lead to observed outcomes  Providing a measure of the relative strength of the contribution each variable of interest makes to the observed outcomes  Making predictions about the likely effects on outcomes of interest if the values of the contributing variables are changed  Producing simulation models based on ‘what-if’ scenarios – for example if you increase the level of investment design-enabled innovation, what is the likely effect on public sector competitiveness?
  • 32. Application Area 3: Value-creating networks and their contribution to efficiency and competition • Social Network Analysis (SNA) maps and evaluates the effects of interconnections between actors in a network, including more complex networks that make up eco- systems • Main focus is on the relationships between network members – but also maps the characteristics of the members • Captures longitudinal evolution of network/ecosystem over time

Editor's Notes

  1. Mechanism M1: Designscapes funding instrument. Designscapes provides a flexible funding instrument to which applicants respond by applying design thinking and tools to generating and developing new and innovative ideas to tackle problems linked to their urban environment. Mechanism M2: Performance and efficiency. By locating their pilots in the urban context and using design thinking and tools, end-users creativity is mobilised and combined with pilot teams’ technical and professional expertise. This leads to new solutions or the adaptation of existing ones which meet user needs better and hence get adopted. This generates business and social value.
  2. 10 that were successful in both the first and second Call. Only three projects were funded for all three Calls, illustrating Designscapes’ innovative emphasis on allowing new projects (which may have previously been unsuccessful) to enter the funding rounds while also enabling those with the highest potential to reapply for funding in each Call.
  3. 10 that were successful in both the first and second Call. Only three projects were funded for all three Calls, illustrating Designscapes’ innovative emphasis on allowing new projects (which may have previously been unsuccessful) to enter the funding rounds while also enabling those with the highest potential to reapply for funding in each Call.
  4. User involvement for round 2+3 only, self-reported data from prototype and scaling surveys
  5. Respondents answering ‘completely’ or ‘to a great extent’
  6. 14 projects funded in rounds 1 and 2 who responded to the survey continued to deliver their project in some form Slightly over half said that this involved people or organisations that they had built links with as part of their Designscapes funded project. -> ongoing benefit of Designscapes participation. Statistically signgiciant skills gains: Round 1: However, when comparing responses to this question with unsuccessful bidders – those who had received funding were significantly more likely to have improved their knowledge or practice in relation to all the different items asked about (p<.001) – range of items linked to design led working (using design methods and tools, considering the urban dimension, involging socially excluded people morem ..) Round 2: theory of change. The only exception is applicants’ confidence in developing a Theory of Change – while the confidence level significantly improved for successful projects (p<.05) it did not for unsuccessful projects (p>.05). The improvement in projects’ confidence level of developing of Theory of Change can be linked with the Theory of Change webinar that some of the funded projects attended.
  7. Co-creation prominent feature in call, so nearly impossible to get funding without including it