1 Introduction
This paper contributes to the ongoing debate on public governance by exploring the role of technology in facilitating the co-production of public services [
31,
76,
53]. Public service design and delivery is expanding beyond organizational and sectoral boundaries and new paradigms that encourage the collaboration between public legal entities and the civil society are receiving growing attention [
9,
82]. In this context, the role of ICT in supporting these innovative practices is considered crucial, even though it is still debated [
47]. An example of co-production is the collaborative development of childcare services where the
Public Administration (PA), third sector education associations, private insurance companies and parents themselves collaborate to the design and directly contribute to the delivery of the service [
37,
18]. Policymakers and governments can play a valuable role by creating and providing an ecosystem that supports co-production. This requires consolidating legal, regulatory and policy framework conditions able to foster coordination between actors, ensure the provision of information and evidence on needs and demands, create opportunities, and facilitate cross-sectoral coordination, as well as provide guidance and support [
72]. The need for effective measures to help public-private heterogeneous networks of stakeholders work together is also of utmost importance.
There are several examples of co-production initiatives that have taken advantage of the potential of digital technologies to create innovative services [
20,
70], often with a focus on solving local challenges around mobility, education, health, energy, and digitalisation [
23,
71,
57]. However, even though
Information and Communication Technology (ICT) is often at the core of the co-delivery of the created services (e.g., in the form of online digital portals or mobile apps for citizens), digital tools are used in a more fragmented way throughout the overall complex co-production process that unfolds into stakeholders’ engagement, service co-design, decision-making, service development, service execution, iterative assessment, and sustainability planning. Moreover, financial, technical, and cultural barriers may exist that prevent the pervasive use of ICT in the different phases [
16]. Despite these entrance barriers, ICT holds the promise of addressing key challenges that hinder co-production.
To contribute advancing the scientific debate on ICT-based co-production, this paper investigates the following overarching research question:
•
RQ: How can we systematically and effectively design a digital platform that can support the co-production among the government and its multiple stakeholders?
The Design Science research methodology [
63] suggests starting the research journey by understanding the problem to solve and the objectives for the solution to design, before proceeding with the design and development, demonstration, evaluation, and communication of the lessons learnt. In our case, this first meant understanding the barriers and challenges to co-production and to the usage of ICT in its support, as well as defining the objectives and main principles at the bases of the proposed platform.
Studies showed that among the main challenges in implementing co-production processes, key issues include the lack of clear theoretical frameworks guiding public managers, insufficient managerial skills, the need of tools for effective implementation and capacity-building, and a lack of awareness among public managers about the concept of co-production. This calls for enhanced management education in this area [
73]. Other studies pointed out that PAs often struggle to define “where to start from” when initiating co-production efforts [
49]. They may lack structured ways to manage workplans and documentation, they may encounter difficulties in managing tasks and handovers, or in keeping up with processes that last many months and even years. Technology is not used systematically, and a mixture of different devices are exploited. Inadequate planning for the regular use of ICTs is recognized as a main barrier [
16]. Public Administrations do not rely on a consolidated corpus of practices or past experiences; they must instead reconfigure the process each time [
48]. In summary, research indicates that PAs require guidance in effectively managing the numerosity and heterogeneity of actors involved and the evolving structure of the collaborative network, with the related bureaucratic and cross-organizational barriers [
33,
48]. Support is also needed to lower the costs of citizens engagement and to improve its effectiveness [
34], by providing collaboration settings able to lower the perceived power asymmetry between public officers and citizens [
46]; by enabling the monitoring and iterative evaluation of the process [
48]; and ensuring its accountability and sustainability [
24].
Addressing these challenges with the usage of ICT requires a change of perspective, since the problem is not focused on a specific stage or aspect of co-production but spreads through the process and requires an overarching approach. However, the concept of providing digital support throughout the entire co-production journey, considering the flexibility required to deal with diverse governance contexts and diverse application domains, is still underexplored [
6]. More studies are also needed to evaluate the impact and effects of ICT solutions on co-production practices [
47]. Research gaps in the current literature are thus related to (i) how technology designers and developers should approach the implementation of digital platforms supporting co-producers throughout their venture, including the possible replication of concluded successful processes, as well as to (ii) the evaluation of systems deployed in real use cases.
The work presented in this paper contributes to filling these specific research gaps by making sure that the investigation to the main research question RQ above included the following research subquestions:
•
RQ1: Which are the requirements that should drive the design of a digital platform facilitating the entire co-production process, starting from the initial idea generation to the actual collaborative delivery of public services and their sustainability?
•
RQ2: Which are the aspects of digital support that are most valued by PAs, when evaluated in diverse and concrete co-production scenarios? And what can we learn to improve the design of governance platforms for ICT-based co-production?
Stemming from the research questions and adopting the Design Science research methodology [
63], our research aims to tackle the social and technical challenges of ICT-based co-production. This serves as the foundation for designing, developing, and evaluating a computer-supported cooperative work environment [
36] to facilitate co-production processes, i.e., those collaborative processes between PAs, private stakeholders and citizens that aim at the design of public services, their implementation and shared delivery to the community [
76]. The ultimate objective is the realization of a digital collaboration platform with a user friendly and accessible front-end that offers: (i) guidance on how to organize a co-production initiative and a network of stakeholders; (ii) functionalities to support collaborative work; (iii) a catalogue of enablers (in the form of reusable knowledge and digital resources) to aid the execution of a range of actions required to design and produce a public service; (iv) guidelines and methods to make sure the produced service complies with national and international regulations and directives (e.g., for electronic identification and General Data Protection), and properly addresses governance issues, business and societal aspects (e.g., through partnership agreements, sustainable business models). To reach this ambitious objective, our research started from a background analysis of the problem, which then informed the identification of the objectives and of the multi-dimensional requirements that come into play and that depend on the governance models impacting on co-production processes, on the specificities related to the type of public service to realize, and on other recognized standards on usability and technological constraints (RQ1). Requirements were then converted into design solutions for a digital collaboration platform and into constraints for prioritizing its functionalities and developing the first software prototype. An evaluation study allowed testing the perceived usefulness and impact of the system with three different PAs and related networks of stakeholders, which resulted in further requirements for platform extension. The lessons that have emerged point at the importance of step-by-step guidance and of potential reuse (with adaptation) of ready to use resources and processes, but also at open challenges for future work in this field (RQ2).
The paper is organized as follows. Section
2 discusses the related literature. Section
3 introduces the envisaged concept of a digital platform to support processes of public service co-production and describes the overall methodology used to set up our research plan to answer the general research question RQ. These introductory sections provide the background for the multidimensional requirements that are discussed in Section
4 and that provide an answer to research subquestion RQ1. Section
5 describes the first prototype that was designed and implemented out of the distilled requirements, while Section
6 presents the results of the user study aimed at investigating RQ2 and that was conducted with representatives of three different PAs (in Latvia, Italy, and Spain), involving about 224 participants engaged in co-production. The discussion in Section
7 elaborates on the research findings to distill general guidelines to help the work of designers working on platforms for ICT-based co-production. The conclusion summarizes major findings and critical issues, with directions for future work.
3 Methodology
The investigation presented in this paper was conducted as part of the INTERLINK research project that aimed at answering research questions RQ, RQ1 and RQ2 by designing and testing an ICT-based platform that facilitates public service co-production and provides knowledge and software resources to guide and monitor service creation and delivery [
51]. Digital functionalities required to support co-production go beyond the functionalities offered by general-purpose platforms for online teamwork or project management. It is expected that more targeted information and guidance is necessary on how to create inclusive networks of stakeholders; how to motivate them to participate in the long run; and how to organize and facilitate activities for collaborative problem analysis and service design. Practical examples, templates and reusable materials can make a difference in creating PA's capacity for co-production. Reusable applications and software components compliant with national and international regulations for public services could also reduce development costs and effort. At the same time, clear examples of co-business models for the long-term sustainability of services may be determinant to incentivize co-delivery. In addition, regulations impose specific restrictions and recommendations on the digital services adopted by PAs [
25,
26]. Therefore, the INTERLINK research work lies at the intersection between the fields of
Computer Supported Cooperative Work (CSCW) [
36], eGovernment [
32,
19], and software development for PAs [
26,
3].
The adopted research methodology is grounded on the
Design Science Research Process (DSRP) [
63] which identifies six main steps for information systems research: problem identification and motivation, objectives for a solution, design and development, demonstration, evaluation, and communication. The communication step, in particular, maps what in our case study is represented by the distilling of lessons learned and guidelines for ICT-based co-production and their sharing with the research community. DSRP was integrated with the
Human-Centred Design (HCD) methodology [
44], which is based on a multi-step journey beginning with understanding and specifying the context of use, followed by defining user requirements, generating design solutions, and finally evaluating results. HCD advocates the adoption of multidisciplinary skills and perspectives, as well as the involvement of users and stakeholders throughout design and development. According to the iterative approach at the base of HCD, our research was structured in three iterative cycles of (re)design, development, and evaluation. The initial phases of context analysis and requirement elicitation took advantage of governance research to better understand known barriers and opportunities for ICT-based co-production. Requirements elicitation also integrated the Scenario-Based Design methodology [
14] to directly involve stakeholders from the PA sector for envisioning technology future use possibilities.
The resulting methodological approach unfolded in the phases illustrated in Figure
1 and detailed here below.
(1)
Problem understanding, objectives definition, and socio-technical requirements. In the initial phase, desk research on co-production and governance literature was performed to identify the known needs and barriers related to the potential usage of ICT to support the execution of co-production processes. This study was complemented with the collection of technical standards for digital platforms for PAs and eGovernment platforms from EU guidelines and publications. To guarantee the imagined technological intervention grounded on actual ways of working, different activities involving local and central PAs were organized using several methodologies, i.e., the
focus group methodology [
85] to analyze current co-production practices; the
personas methodology to analyze stakeholders' needs and roles [
17]; and the
scenarios methodology to refine shared views of how technology can be incorporated into existing practices [
14]. User-centered activities during this phase involved 15 representatives from three different PAs. The collected information was elaborated to translate theory, standards, and user desiderata into actual system requirements, formally described according to an approach similar to the Volere methodology [
67].
(2)
Design. A list of functionalities was elaborated and mockups of the user interface were sketched and iteratively validated by user-centered design experts to check the interaction flow before implementation.
(3)
Development of system alpha version. An alpha version of the digital platform was implemented featuring a set of major functionalities to offer guidance to PAs for setting up co-production and for supporting collaborative activities of heterogeneous networks of stakeholders, including citizens.
(4)
User evaluation. Afterwards, usability tests considered the implemented functionalities one by one, to investigate how to improve the user experience and check whether the design succeeded in implementing the initial requirements. Usability testing was carried out by means of a
heuristic evaluation session (with three usability experts) [
59], and of end-user testing. Six individual sessions were organized with PA representatives to collect focused and structured feedback while users executed a controlled task with the platform, by means of observations and concurrent
think-aloud methodology [
84]. After the individual user tests, further reflection was stimulated in a collective brainstorming session. Additional usability shortcomings emerged from sessions of free system exploration that involved 22 end-users in total.
(5)
Development of system first prototype. The list of identified usability issues informed a refinement of the system interface design and eventually brought to the finalization of the first consolidated prototype of a digital collaborative environment supporting co-production.
(6)
Pilot system usage. The digital platform was subsequently used by three PAs to carry out three sample use cases of public service co-production involving 224 participants overall, during a time span of 4 months. Usage data was automatically collected by the system, while researchers observed some end-users' activities, and end-users themselves reported their experience in ad-hoc questionnaires (88 respondents).
(7)
Post-pilot reflection. At the end of the pilot experiment, a reflection phase was organized in collaboration with the three involved PAs (12 representatives) to evaluate how the digital platform was used in practice and how it impacted on co-production activities. The evaluation was facilitated by a structured questionnaire and an online workshop where participants were asked to rank current and potentially new functionalities by perceived usefulness.
(8)
Design refinement. The evaluation findings were complemented with a brainstorming session during which end-users were invited to envisage revisions and extensions of the digital collaborative environment.
(9)
Development of the final prototype and lessons learned. The redesign indications enabled the development of a second, improved system, and were generalized into lessons learned and guidelines for ICT-based co-production.
In this paper we particularly focus on the steps highlighted in gray in Figure
1, i.e., on discussing the analysis of the socio-technical requirements (step 1); the design choices related to system functionalities with a description of what was implemented in the system first prototype (steps 2 and 5); and the results of the system evaluation and post-pilot reflection phase, with related re-design requirements, and the lessons learned that emerged from this articulated process for ICT-based co-production (steps 7, 8, 9).
4 Understanding the Context and Distilling Socio-technical Requirements
To start exploring our general
research question (RQ) on how to effectively design a digital platform that can support co-production, we went through the first two steps of the Design Science methodology, i.e., the problem identification and the definition of the objectives for the information system. The aim of this stage was the specification of a detailed set of socio-technical requirements that describe in which way the system should solve the many facets of the problem. Indeed, requirements elicitation is the process of learning and understanding the problem to be solved and the needs of users and related stakeholders, with the ultimate aim of communicating these needs to the system developers. It is a multifaceted process that requires understanding the application domain, current practices and imagined scenarios, as well as the lessons learned from similar systems [
88]. Identifying socio-technical requirements entails considering human, social, and organizational factors as well as technical factors to guarantee that the technical and organizational aspects of a system are considered together [
5].
This means that for the elicitation of the socio-technical requirements informing the design and implementation of a digital platform for co-production, it was necessary to integrate different perspectives. First of all, to improve our understanding of the application domain we turned to contemporary research studies addressing new governance models open to civil society participation. This analysis allowed us to distill key insights from existing findings on opportunities and challenges related to ICT-based co-production to be translated into practical guidelines for high-level system design (top-down governance requirements). Secondly, talking with potential users about concrete and diverse use cases helped understand how the specificity of the public services to design impacts on the necessary flexibility and customization of the digital platform (bottom-up use case requirements). Lastly, a collaborative environment that supports the co-production of public services (and thus provides digital functionalities to PAs, private stakeholders and citizens for them to work together, contribute ideas, take decisions, negotiate service co-delivery,…) needed to comply to all the standards envisaged for digital platforms used by PAs as well as for e-Government platforms that provide a contact point between PAs and citizens (technical requirements).
For simplicity, in the following sections the targeted platform will be named “Collaborative Environment” (often abbreviated as CE).
4.1 Co-production Problems and Top-down Governance Requirements
As principles related to collaborative governance models are essential to understand how technology should be designed to support co-production practices and cross-organizational collaboration, a systematic literature review was conducted on the influential factors and methods of citizen engagement within co-production processes, which attempts to summarize and combine existing research on the topic of digital and non-digital co-production and identify both findings and gaps [
7]. Further study work was focused on the challenges and opportunities for ICT in co-production [
16,
38,
47,
48,
53,
61,
76,
87]. The findings that emerged from the literature were analyzed and translated into high-level requirements for the Collaborative Environment, as follows:
R1. Raise awareness and build capacity in co-production. PAs who are new to co-production might have difficulties in understanding if a collaborative approach is suitable for them. This is related to the novelty that participatory approaches represent for PAs, to their scarce experience, and to the lack of good practices and success stories that help build capacity [
49]. The CE should provide orientation information to help PAs better frame their context and understand how co-production can help.
R2. Overcome cultural barriers by providing clear added value for co-production. Negative perceptions and fears on the part of government staff about ICTs may significantly undermine their active participation in ICT-enabled co-production [
16]. Flexible and adaptable solutions are required that do not introduce additional bureaucracy to processes. New digital tools should provide information and guidance closely related to existing work practices and to the specific co-production task users need to undertake, so that they can clearly perceive the added value with respect to general-purpose tools they already use [
49].
R3. Guide users in managing the process. There may be different co-production paths according to different government models, stakeholders involved, and types of services [
9,
11,
19,
1,
55]. To avoid PAs getting lost in the process, the CE should provide effective step-by-step guidance by suggesting actions to perform and appropriate resources to use to help complete such tasks, even though flexibility should be guaranteed.
R4. Support the management of the network. The lack of proper management and leadership of the collaboration network, and the fact that nobody “owns” the outcome results might lead to low commitment to the project [
64]. This is particularly amplified by the specific inter-organizational and intra-organization specificities of public bodies and the complexities of institutional relationships [
49]. A collaborative venture needs a reputable stakeholder to play the role of initiator and facilitator of the collaboration, i.e., the "convener" of the collaborative network [
80,
45]. The CE should guide coordinators (or conveners) in understanding which skills are needed in the different phases of the process, defining which the related roles and responsibilities are and making them transparent to other members, supporting the network of stakeholders in sharing knowledge, open discussion and decision making. A clear coordinator role should be defined, with access to advanced functionalities that are hidden from other participants.
R5. Offer a catalogue of reusable resources to lower the financial impact of ICT use. Shortage of finance is a common barrier to a government's promotion of ICT-enabled co-production and financial impediments to the deployment of ICTs in co-production are reported as lower when governments or public agencies opt to use low-cost ICTs [
16]. This approach implies the risk of fragmenting the digital support to co-production into many different tools, thus increasing complexity and conflicting with requirement R2 above. As an alternative approach, the CE should experiment with a centralization of resources, so that PAs dealing with co-production can find in a unique reference place reusable resources to ease their task. Digital functionalities should be made available within the platform to support collaboration dynamics. At the same time, a selected number of reusable software components should be made available to lower the implementation costs of the targeted public services. Guidance must be provided on sustainability issues, and on how to select the most appropriate third-party ICTs considering different aspects, like maintenance costs and regulatory framework.
R6. Contribute to technical capacity with knowledge resources. The lack of a skilled workforce constitutes a barrier to co-production. Furthermore, the lack of planning for the day-to-day ICT use may lead to the failure of ICT-enabled initiatives [
16]. The CE should provide information on how ICT can be exploited throughout the whole co-production process (e.g., practical recommendations, best practices, risks related to potential technical failures, etc.). Furthermore, the CE should incorporate targeted knowledge resources to meet the specific requirements of the PAs that are opening their processes of public service design to relevant stakeholders [
56,
83]. The objective is to operationalize knowledge by combining co-production know-how with a digital platform that guides and supports its use [
79].
R7. Enforce transparency and ensure privacy. A lack of trust in the government tends to reduce the participation of citizens in ICT-enabled co-production. Additionally, citizens could fear that ICTs may invade their privacy [
16]. The CE should ensure the transparency of collaborative processes and deploy privacy-preserving solutions that are clearly communicated to users.
R8. Provide legal framework. Another barrier experienced by governments is the complexity of legal issues that can prevent them to take-up ICTs [
16]. For instance, the deployment of ICT solutions in co-production faces significant delays due to complex challenges related to legal regulations concerning data protection and privacy. The digital platform should support privacy by design [
15] and offer guidance on how reusable software may be deployed and managed in compliance with norms and regulations.
R9. Usability of solutions and training. Lack of technical skills, along with a negative attitude toward ICTs tends to reduce the participation of citizens in ICT-enabled co-production [
16]. More in general, the adoption of a new technology is affected by many factors. Usability is an important factor, but other factors play a crucial role: accuracy, price, physical appearance, security, function, interoperability, and robustness are all independent factors affecting user acceptance [
42]. The CE should promote bottom-up processes and ICT training as measures to encourage stakeholders’ participation. Besides, the CE should ensure high usability and acceptability through a design process that iteratively evaluates the developed solution with end-users and incorporates their feedback.
4.2 User Research and Bottom-up Use Case Requirements
To ensure the design of the CE was user-centered, a scenario-based design approach [
14,
68], enriched with the use of personas [
17], was adopted to foster an active participation of stakeholders and end-users in the definition of requirements. Scenario-based design consists in presenting and discussing stories that represent a specific problem or technology in use with different purposes [
10], namely to identify needs and problems; to present and situate potential solutions; and to illustrate and discuss alternative solutions.
Three use cases of PAs in Europe in need of co-producing a public service in collaboration with a heterogeneous network of public-private stakeholders were involved in the requirements elicitation process. The three use cases were selected to represent a variety of intents and dynamics: e.g., central vs. local PAs; low vs. high citizens involvement; and governance models involving different shares of power and responsibility. This variety ensured that distilled requirements incorporated the flexibility inherent in the co-production process, as discussed in Sections
2.1 and
2.2 above.
(1)
The
Italian Ministry of Economy and Finance (
MEF for short) was interested in investigating whether co-production methodologies can help produce digital services for the public sector with high acceptance and adoption rates. To this aim, they focused on testing the co-design of a new digital tool to support strategic planning tasks of public bodies. They envisaged a co-production process involving the participation of PAs, civil servants, government stakeholders, and service companies [
49]. The governance model underlying this use case is
Government as a Platform (G2C) [
50], a specific type of co-delivery, in which the government offers a digitally enabled governance framework for the delivery of services by other public bodies, citizens and private actors.
(2)
The
Latvian Ministry of Environmental Protection and Regional Development (
VARAM for short) was interested in investigating whether a co-production approach increases the usability of citizen sourced information for policy shaping. To this aim, they experimented with the collaborative refinement of municipal digital services’ descriptions with the involvement of municipalities, citizens, digital agents, and state customer service centers. The governance model underlying this use case is
Citizen sourcing (C2G) [
50], which occurs when the government designs and delivers a service but asks citizens for the voluntary commitment of resources to improve the service itself.
(3)
The
Municipality of Zaragoza, Spain (
ZGZ for short) was interested in widening Open Innovation within the city by co-creating and co-delivering services together with citizens. To this aim, they experimented with the co-production of initiatives to take place in Etopia, a public space in the city hosting creative and innovation activities. The governance model underlying this use case is
Public-Civic partnership (P+C) [
50], which occurs when the government, citizens and private actors share equal power and responsibility. Each side brings their unique advantages to the table to collaboratively solve problems and create public value.
The three analyzed use cases highlight the potentially large heterogeneity of co-production processes that may differ by objectives of PAs in relation to co-production; actors involved; level of participation of citizens or associations of citizens; type of service to be co-produced; and resources needed to carry out the co-production process.
During an initial focus group, 15 PA representatives (10 for MEF, 3 for VARAM, and 2 for ZGZ) were invited to reflect on their co-production objective and on the social and organizational context in which co-production would take place. A two-page template was distributed to collect information on the public service to be co-produced; on the desired future scenario in which the service would be delivered; key actors and roles; related initiatives; available resources, desired additional resources and desired digital support to sustain the co-production of the public service.
After this preparatory activity, use case coordinators were guided through the generation of personas (6 for MEF, 6 for VARAM, 4 for ZGZ, taking into account different actors engaged in co-production) and the description of their relationships. Personas, i.e., fictitious profiles of service end-users and stakeholders [
69] that act as a benchmark for design, encouraged teams to reflect on the actors involved in the service to be co-produced, i.e., on the potential co-producers. The generated personas were used to describe three scenarios of co-production, i.e., the process by which a network of stakeholders collaborates to create and deliver a new public service. PA representatives were invited to imagine co-production needs that could be solved by means of digital technology and reusable knowledge resources. Table
2 summarizes the requirements that PAs mentioned as useful in their use case. As it can be observed, some of the requirements refer to functionalities that could be integrated directly inside the Collaborative Environment, such as tools for workplan scheduling or tools for managing incentives and rewards to sustain participation (U1, U9). Other requirements call for software modules to be used to support specific tasks, such as an editor to collaboratively revise texts appearing in web pages (U8, U11–U14). A third class of requirements refers to reusable knowledge (guidelines and template documents) to bootstrap activities (U2–U7, U10).
Reusable knowledge and reusable software may vary significantly across use cases and a rich and extendable catalogue of such resources, which we call INTERLINKERs, should be made available.
4.3 Standards and Technical Requirements
Janowski [
39] argues that the Digital Government concept has evolved across four main stages: Digitization (Technology in Government), Transformation (Electronic Government), Engagement (Electronic Governance) and Contextualization (Policy-Driven Electronic Governance). ICT-based co-production falls within the category of technology that involves the engagement of a plurality of public and private actors in the generation of solutions and the co-creation of public value, thus enabling government to make a transformation to a more open and participatory model that they see as more desirable [
54]. Therefore, ICT-based co-production can be seen as spanning across Electronic Government and Electronic Governance, implying a transformation of the internal working and structures of government as well as its relationships with citizens, businesses, and other stakeholders [
39]. Previous studies and projects on Digital Government let emerge particular requirements that must be taken into account, not only during the implementation phase but also at the early design or architecture modeling phases. Technical requirements refer to all those features that the platform should implement and the constraints it should satisfy to be interoperable and compliant with national and international regulations. Table
3 summarizes a set of common characteristics of Digital Government architectures that have emerged from a literature review by Baheer et al. [
2]. Most of these requirements are also explicitly endorsed by national and international regulations and recommendations, such as for example the EU eGovernment Action Plan that provides guidance for all countries in the European Union [
25]. Specific requirements address the issue of ensuring that the platform can be easily adapted and used by different PAs and with a cross-sector and cross-border dimension.
These technological requirements guided the baseline architectural choices of the platform and the development process. For example, to put in practice technological neutrality and platform independence, a web-solution was preferred (T15, T16), with the possibility for the platform to work both in single-tenant configuration (i.e., with one instance of the platform for each PA) and in multiple-tenant configuration (i.e., with the same server safely managing different PAs), to help reduce operation costs to a minimum (T14). An authentication and authorization service implementing the Single Sign-On protocol
1 was put in place (T8), supporting European identification standards, thus also complying with cross-border characteristics (T11). To comply with the requirement of traceability and accountability of actions, methods for logging user interactions and analyzing behavior were conceived (T9). Multilinguality of the user interfaces and the possibility of customizing them according to PAs' needs was included in accordance with the adaptability and flexibility requirements (T3, T4). As explained in Section
5, an extendable catalogue of reusable resources facilitating co-production was envisaged to cater for the requirements of reusability, resilience, and scalability (T5, T6, T12). To ensure interoperability and integrability, the formalization of reusable software resources was inspired by the Digital Europe Building Blocks methodology [
28] (T1, T7). FAIR principles and the
General Data Protection Regulation (GDPR) constraints for data and metadata management [
86,
30,
75] were percolated through the system architecture to comply with the security and legality requirements (T2, T13). The user-centered design approach adopted as a baseline methodology for the overall project, ensured citizen-centric and usability requirements right from the start (T10, T19). Tutorials and user manuals addressed the comprehensibility and minimal learning curve for all potential users of the platform (T17, T18).
The following section describes the first version of the Collaborative Environment that was designed and implemented according to the collected multiple types of requirements, namely, top-down governance requirements, bottom-up use case requirements and technical requirements.
6 User Evaluation Study: Pilot Adoption of the Platform in Actual Use Cases and Identified Extensions
To execute the phases that in the Design Science methodology are called as demonstration and evaluation, the three use cases described in Section
4.2 were used as a testbed to evaluate the CE both from a technical perspective and, most importantly, from a user standpoint. As explained in Section
3, preliminary usability tests involving three human-computer interaction experts and 22 end-users evaluated the CE interface against Nielsen's usability heuristics [
58]. The collected feedback allowed identifying several usability problems that were immediately passed on to developers to improve the user interaction experience and consolidate the first prototype of the system - readers interested in how the usability study was organized can refer to [
52]. This section reports instead the results of the second evaluation study aimed at gaining a deeper understanding of how the digital platform was used in co-production use cases and what can be learned on ICT-based co-production, to reply to research question RQ2 stated in the Introduction section.
The three PAs coordinating the use cases used the CE described above to facilitate their co-production activities involving 224 stakeholders in total (68 for MEF, 100 for VARAM, 56 for ZGZ), during a 4-month period. Several monitoring and evaluation tools were used to capture users' experience throughout the experiment: interaction logging and automatic measures of the usage of system functionalities; self-reported activities description [
52]; observations and interviews [
49]; a questionnaire investigating the experience of participating in co-production; a questionnaire investigating the quality of co-created public services; and final post-pilot reflection questions and focus groups on the usage of the system. In the following paragraphs, we briefly summarize the major aspects that are interesting from a
Human-Computer Interaction (HCI) point of view, addressing in particular the final reflections on perceived usefulness, appropriation, adequacy to governance models, and possible requirements for re-design or extension of the system. Other investigated dimensions addressing the evaluation of digital co-production quality are described in [
74,
66].
The reflection questionnaire distributed after the end of the use case activities, included 31 open questions, grouped into nine investigation areas, namely: (1) how the co-production working group was structured in the CE; (2) the level of participation observed in stakeholders; (3) how the co-production process was represented in the CE following the system's guidance; (4) impressions on the alternative co-production schemas offered by the platform; (5) the possible role of examples of what other PAs have done; (6) the perceived need for different roles and access rights to the shared documents; (7) the most used and most useful functionalities offered by the CE; (8) the most used and most useful INTERLINKERs; and (9) the perceived added value offered by the CE. The coordinating teams of each use case (6 PA representatives for MEF, 2 for VARAM, and 4 for ZGZ) were asked to gather together, reflect on the use case outcome, and elaborate an agreed answer to each of the reflection questions.
Two further reflection focus groups were organized with the same 12 PA representatives to elaborate on the questionnaire results and on suggestions for future extensions, and to draw comparisons across the three use cases. The discussion was structured around themes that had emerged from the questionnaire and that focused on how co-production coordinators actually interacted with the technology to organize their co-production processes and their perceptions about workflows and other stakeholders' engagement with the platform.
Bespoke functionalities and guidance for co-production. On average, use case coordinators observed a good level of motivation to participate in co-production in stakeholders. This was referred especially to those actors that already had awareness of and direct experience with the problem tackled by the co-production process. In the words of the VARAM team,
“higher digital skills and awareness of public services delivery significantly raised the motivation”. The real challenge lies in requesting co-producers to introduce a new work methodology or to use a different digital platform than the one already established in their daily work practice. It was mentioned that
“for the Collaborative Environment to be adopted as a working tool, it is necessary to maximize the value provided to the team involved in co-governance, co-design, and co-production. This includes enabling time savings and a better quality of the final service” (ZGZ team comment). The use case coordinators noted that intuitiveness of use and simplicity of the system are not the only important factors, stressing the importance of other distinguishing functionalities that are tailored for co-production (confirming governance requirement R2). These include (i) step-by-step guidance to help users understand where to begin and determine the appropriate path to follow depending on their specific co-production model and problem to solve; (ii) efficient strategies to reach out participants and incentivize their participation; and (iii) template materials to speed up the preparation of communications and joint activities. Focus groups also let emerge the importance of additional methods to monitor and stay updated about process progress and of a clear overview of the work carried out so far to facilitate handovers, for example when decision makers delegate practical tasks to other public servants. Figure
5 shows an example Jamboard screenshot with the collective reflections and recommendations generated by users in this regard. These users' recommendations that emerged from the evaluation were immediately taken into consideration by system developers to extend the set of platform functionalities with mailing list support and email and in-app notifications to keep participants updated about latest changes on the process’ resources and task completion, as shown in Figure
6.
The second version of the CE also featured an enriched information part that provides clearer textual and visual representation of the steps that the users (both coordinators and participants in the co-production process) can follow to create/navigate the phases and the resources for their co-production processes.
Flexible management of the co-production work group. As emerged from the logs automatically collected by the system, different PAs, with different internal organizational/hierarchical structures, interpreted the concepts of “organization” and “team” differently according to their usual work practices and grouped people in their network according to different rules. This is an interesting issue to investigate from the Human Computer Interaction (HCI) point of view, as there might be forms of
appropriation, i.e., observed uses of the functionalities, different from what originally conceived by designers, to adapt to existing work practices [
21]. The version of the CE that was tested by end-users provided a flexible way of creating virtual organizations and teams to structure the work group of a co-production process. According to the initial socio-technical requirements, this functionality was introduced to control the visibility of activities and of shared documents. Users were free to decide how to group participants in lists and they did actually exploit this freedom to accommodate different interpretations of the available functionality, different co-production goals, and different governance approaches of the process. The comparison between the three use cases revealed that the functionalities for managing the work group were appropriated by co-production coordinators in two main ways: (i) to create just one main organization to represent the overall initiative and different teams to sub-group people from different stakeholder institutions or associations, as in the MEF use case; or (ii) to create multiple organizations, one for each involved stakeholder institution/association, and to create new teams following more practical and extemporaneous needs (e.g., to group participants invited to specific meetings or activities or to separate categories of participants), as in the VARAM and ZGZ use cases. In both cases, co-production coordinators deemed it important to explicitly name institutions (either in teams or in organizations) for formal partnership recognition and to enhance the visibility of the network. In the MEF team's words, “
this allowed us to better handle and organize the involvement of the various stakeholders, assign tailored permissions, and be sure that each participating Administration was represented in the MEF co-production process and in the collaborative environment”.
Following the evaluators' feedback, the second version of the CE also featured the possibility of activating a “Call for Collaborations”, by announcing the profiles and skills needed to be part of an open process collaboration force.
Roles and Management of access rights. To the users that participated in the evaluation tests it was clear that the team coordinating the co-production initiative plays the role of administrator of the collaborative process and should have access to the most advanced functionalities made available by the digital platform. Other stakeholders participating in co-production would mostly be contributors or observers, based on their level of engagement in the activity, and should be presented with simplified views that hide unnecessary functionalities of the CE (ZGZ team: “The incorporation of team and user roles and privileges would be more than welcome.”; MEF team: “As we were the ‘administrators’ of the co-production process, this feature allowed us to set up the process and create some guidance resources without the risk of them being modified or deleted by mistake by the stakeholders”). It was mentioned that a “tech-savvy administrator is required” (VARAM team).
According to previous research, different team members may participate in different phases of co-production [
48,
49]. This was the motivation for including a fine-grained method in the implementation of the CE for (optionally) assigning permissions at different levels of the co-production schema, to restrict the number of participants that can read or manipulate shared documents (resources) within different phases, objectives, and tasks. Evaluators commented that default permissions assigned to the overall co-production schema would suffice in most cases, possibly using the main categories of participants (administrator, contributors, observers) to differentiate default permission values (as mentioned by ZGZ team: “
It would be useful if these different categories of users could be associated with different levels of access to the platform so that what they can do within the environment depends on the level of authorization they have.”).
Guidance through co-production schemas. On the one hand, co-production coordinators highly valued the availability in the digital platform of pre-prepared co-production schemas that recommend which phases and activities would be appropriate to co-design and co-deliver public services in certain governance contexts. On the other hand, choosing which schema fits their specific context was considered challenging (VARAM team: “
it was unclear which one to choose”). This choice actually implies being able to recognize the specificities and differences between different co-production processes and schemas. This is in line with the literature findings summarized in previous section
2.2, which recognize factors like the type of stakeholders’ network, the type of service, and who is taking the initiative as impacting on the strategies, activities, and responsibilities characterizing co-production processes [
9,
11,
1,
55,
83]. As an important lesson learned for the refinement of the system user interface, it emerged that the step of co-production schema selection requires an appropriate dialogue to guide coordinators' choice based on recognizable properties (e.g., type of organizations involved, level of openness of the process, etc.), in line with MEF team's suggestion:
“maybe you could consider adding keywords to further simplify the choice”. In addition, the inherent flexibility of the co-production process implies that a standard co-production model hardly fits a use case without any adaptation; changes to the workplan may also occur in time, or some decisions can be taken just later on (e.g., how to organize the maintenance of the service). For this reason, PA representatives highly valued the possibility, made available by the CE, of customizing a chosen co-production schema by removing or adding phases/objectives/tasks to their plan of work. In the PA representatives' words:
“The proposed schemas, although theoretically perfect, are too ambitious to be implemented in real cases” (ZGZ team).
“Based on the feedback from participants, it must be possible to adjust schemas manually.” (VARAM team).
“The possibility to change the title and description of objectives and tasks, as well as to delete phases/objectives/tasks not useful for our activities allowed us to create a schema suited for our co-production objectives” (MEF team).
It was also mentioned that PAs who are new to co-production may benefit from learning how similar public bodies have successfully applied co-production [
49]. Needs like the following emerged:
“know what was the impact, the benefit”, “know what worked and also what did not work”, “draw inspiration”, “make an analysis to know if the process is replicable in a similar context”. A catalogue of “co-production stories” could be used to draw inspiration and to understand if a past process is transferable to a similar context by carefully considering whether there is technological, organizational, and institutional fit. Furthermore, coordinators could even clone existing co-production schemas and related resources to avoid starting from scratch when creating new or revised public services. This practice would be in line with what Dolowitz and Marsh [
22] call
voluntary policy transfer, “the emulation and lesson drawing [which] refer to a process in which knowledge about policies, administrative arrangements, institutions and so on in one time and/or place is used in the development of policies, administrative arrangements and institutions in another time and/or place”. A digital catalogue of best practices in co-production would also help overcome the difficulty of formalization, scaling up and dissemination of the experiences related to the often informal and experimental nature of most co-production initiatives [
76]. To implement this form of Open Innovation based on the promotion of replicability, the second version of the CE was extended with the possibility of cloning past processes of a team or cloning from publicly available success cases (“Stories”), with the possibility to subsequently customize the cloned process and the related resources to best fit your own co-production context.
Awareness of co-production process status. PA representatives acknowledged that face-to-face gatherings offer the optimal opportunity for coordinators to monitor stakeholders' level of involvement and to stimulate active contribution. More difficult is to get a sense of the co-production activities performed through the digital platform, such as the post-meeting elaboration of shared documents. Automatic tracking of collaborative activities performed by users was considered useful to generate summaries reporting which stakeholders contribute to what specific collaboration activities. However, more specific suggestions were made on how to support teamwork for co-production. It was mentioned, for example, that it would be useful to directly display in the interface which users have participated in each phase or activity (Figure
5), in order to keep track of participation, to possibly adjust the composition of work groups, and to provide a way to enforce accountability [
78]. Visual cues that signal changes and updates to the digital representation of the co-production process since the last login were also considered essential, to let participants keep up with the latest progress. Additionally, there was also a demand for personalization: co-producers should be able to immediately see which actions require their contribution (in PA representatives' words:
“which tasks need my work?”, “deadlines and alerts of tasks”).
Another important aspect pointed out by PA representatives was the need to get an intuitive view of the status of the entire process, such as a progress board summarizing the phases and tasks that have already been worked on, those that are in progress and those that are forthcoming, in order to have a sense of how the group work is progressing towards to the final goal (Figure
5). In the words of the VARAM team: “
It is hard to navigate within the process, to determine where is what. The process dashboard could be more simplified, including a progress board - maybe something like Kanban?”.
4As a response to these new requirements emerged from the evaluation, in the second version of the CE the possibility of assigning resources to people and see in-app personal notifications with the assigned work was added, as well as a bespoke “Assignments” tab within the Resources view, which summarizes a personal to-do list. Besides, a new “Contributions” tab was added for each task view to allow for the visualization of who has contributed with what level of engagement for the whole task or per individual resource in the task (Figure
7).
8 Conclusion and Future Work
Existing general-purpose digital collaboration platforms offer a range of tools designed to support diverse (subsets of) functionalities, including the setting of work plans, task management, the collaborative editing of documents, textual or visual discussion boards, ideas crowdsourcing, joint deliberation, or data visualization to support informed analysis and decisions, among others. However, flexibility and freedom of use come at the expense of guidance, since platforms assume that users are familiar and knowledgeable about the specific collaboration task they must perform. This is a severe limitation for PAs and related networks of stakeholders that are novice to co-production, i.e., simply wishing to involve citizens, NGOs, and private companies in virtuous collaborative processes may not guarantee project success. Background literature reports that proper bespoke support is necessary for building capacity in co-production methods; finding the right workflow of steps to follow; managing the different skills, backgrounds, and competences in heterogeneous networks of stakeholders; finding materials and examples that help bootstrap activities; lowering the financial impact of ICT use; and for periodically checking whether the legal, transparency and sustainability requirements are met. Guidance and exemplification are thus distinguishing features that are essential to provide an added value for co-production.
The research study reported in the paper focused on the problem of investigating how designers and developers should approach the implementation of technology supporting co-producers, as well as on the evaluation of systems deployed in real use cases. We started from an analysis of the problem and of the socio-technical multidimensional requirements to inform the design and development of a digital collaboration platform, where guidance and exemplification fostering reuse are pivotal. Technical requirements for digital government software architectures were integrated with knowledge on governance models and co-production processes and were validated through the analysis of three specific use cases coordinated by one central PA in Latvia, one central PA in Italy, and one local PA in Spain. The developed Collaborative Environment is based on four main pillars: (i) the availability of (an extendible set of) digital schemas representing models of co-production processes and illustrating the phases/objectives/tasks that stakeholders are suggested to go through for the collaborative design and delivery of public services; (ii) the recommendation of knowledge and software co-production enablers (named INTERLINKERs), which are reusable resources that may help bootstrap stakeholders' activities and lower the costs of public service implementation; (iii) the availability, for each co-production task, of a work space with the possibility to assign teams, create documents by instantiating INTERLINKERs or by using general purpose collaboration tools (like Google Drive and Loomio) and links to external resources; and iv) the availability of a catalogue of INTERLINKERs, success stories, and open processes promoting reuse, replicability and adoption. The core innovation that is brought about with respect to existing general-purpose platforms for computer supported cooperative work is the operationalization of co-production processes.
An evaluation study conducted by three different PAs with the involvement of 224 stakeholders tested the usage of the platform during the execution of three co-production processes. The findings underlined the overall appreciation of the offered functionalities, with observed forms of appropriation of the tool. Participants clearly stated that the main added value that the Collaborative Environment can offer is represented by the step-by-step guidance and potential reuse (with adaptation) of ready-to-use resources. They also stressed the importance of initial training and sense making to help users familiarize themselves with the platform and reduce the cultural and technical barriers of adopting a new tool.
The evaluation study performed also sheds light on a few challenges that clearly point to future work. Firstly, there is the need to harmonize activities that are performed within the digital platform with the (possibly large) number of activities that are held physically (e.g., meetings, public gatherings), so as to add the possibility of accounting the latter within the system too. Future extensions of the platform will extend the internal data models for a clear representation of the hybrid nature of co-production processes, with functions for administrators and users to keep track of off-line activities. Secondly, co-production processes may span over months, sometimes even years. The engagement of heterogeneous networks of stakeholders for long periods of time might require incentives, both material and immaterial, in order to foster participation and active contribution. In a second round of user evaluation, we are currently testing the integration of a gamification engine that is able to track user actions inside the platform to register contributions and value them through an incentives-and-rewards system that has already proved successful in engaging citizens in sustainability actions [
40,
74]. A third aspect that deserves attention is the potential initial need of PA representatives to gain a deeper understanding of whether co-production is the appropriate approach to follow, possibly benefiting and learning from the experience of other similar PAs. On this regard, in order to support the voluntary policy transfer [
22], the second version of the Collaborative Environment features a catalogue of "co-production stories" for PAs to draw inspiration from, with clone facilities that help start from specifically customized co-production schemas already tested successfully by other PAs [
74].