Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article
Open access

Designing a Digital Environment to Support the Co-production of Public Services: Balancing Multiple Requirements and Governance Concepts

Published: 13 September 2024 Publication History

Abstract

This paper investigates the challenges of designing a computer supported collaborative environment aimed at facilitating co-production processes, i.e., those collaborative processes between Public Administrations, private stakeholders and citizens that aim at the design of public services, their implementation, and their shared delivery to the community. We argue that, for such a digital platform, different types of socio-technical requirements should be considered, i.e., those related to governance models and associated collaboration dynamics; requirements that may emerge from the specific type of public service to design; as well as user and technical requirements common to all e-government platforms. This research informed the development and testing of a digital collaboration platform that offers guidance on how to organize a co-production initiative and a network of stakeholders, functionalities to support collaborative work, and enablers (in the form of reusable knowledge and digital resources) to perform the sequence of steps to produce a public service. The lessons learned from the iterative platform development process and a preliminary evaluation study conducted with three Public Administrations in different European countries pointed at specific functionalities that are perceived as most crucial and at different appropriation practices that depend on the organizational structure of the involved Public Administrations and related multi-stakeholder networks. The innovation that is brought about with respect to general-purpose platforms for computer supported cooperative work is represented by the operationalization of co-production processes, with step-by-step guidance and potential reuse (with adaptation) of ready to use resources and processes. Based on the results of our research, general guidelines are also proposed for the design of future digital platforms supporting co-production.

1 Introduction

This paper contributes to the ongoing debate on public governance by exploring the role of technology in facilitating the co-production of public services [31, 76, 53]. Public service design and delivery is expanding beyond organizational and sectoral boundaries and new paradigms that encourage the collaboration between public legal entities and the civil society are receiving growing attention [9, 82]. In this context, the role of ICT in supporting these innovative practices is considered crucial, even though it is still debated [47]. An example of co-production is the collaborative development of childcare services where the Public Administration (PA), third sector education associations, private insurance companies and parents themselves collaborate to the design and directly contribute to the delivery of the service [37, 18]. Policymakers and governments can play a valuable role by creating and providing an ecosystem that supports co-production. This requires consolidating legal, regulatory and policy framework conditions able to foster coordination between actors, ensure the provision of information and evidence on needs and demands, create opportunities, and facilitate cross-sectoral coordination, as well as provide guidance and support [72]. The need for effective measures to help public-private heterogeneous networks of stakeholders work together is also of utmost importance.
There are several examples of co-production initiatives that have taken advantage of the potential of digital technologies to create innovative services [20, 70], often with a focus on solving local challenges around mobility, education, health, energy, and digitalisation [23, 71, 57]. However, even though Information and Communication Technology (ICT) is often at the core of the co-delivery of the created services (e.g., in the form of online digital portals or mobile apps for citizens), digital tools are used in a more fragmented way throughout the overall complex co-production process that unfolds into stakeholders’ engagement, service co-design, decision-making, service development, service execution, iterative assessment, and sustainability planning. Moreover, financial, technical, and cultural barriers may exist that prevent the pervasive use of ICT in the different phases [16]. Despite these entrance barriers, ICT holds the promise of addressing key challenges that hinder co-production.
To contribute advancing the scientific debate on ICT-based co-production, this paper investigates the following overarching research question:
RQ: How can we systematically and effectively design a digital platform that can support the co-production among the government and its multiple stakeholders?
The Design Science research methodology [63] suggests starting the research journey by understanding the problem to solve and the objectives for the solution to design, before proceeding with the design and development, demonstration, evaluation, and communication of the lessons learnt. In our case, this first meant understanding the barriers and challenges to co-production and to the usage of ICT in its support, as well as defining the objectives and main principles at the bases of the proposed platform.
Studies showed that among the main challenges in implementing co-production processes, key issues include the lack of clear theoretical frameworks guiding public managers, insufficient managerial skills, the need of tools for effective implementation and capacity-building, and a lack of awareness among public managers about the concept of co-production. This calls for enhanced management education in this area [73]. Other studies pointed out that PAs often struggle to define “where to start from” when initiating co-production efforts [49]. They may lack structured ways to manage workplans and documentation, they may encounter difficulties in managing tasks and handovers, or in keeping up with processes that last many months and even years. Technology is not used systematically, and a mixture of different devices are exploited. Inadequate planning for the regular use of ICTs is recognized as a main barrier [16]. Public Administrations do not rely on a consolidated corpus of practices or past experiences; they must instead reconfigure the process each time [48]. In summary, research indicates that PAs require guidance in effectively managing the numerosity and heterogeneity of actors involved and the evolving structure of the collaborative network, with the related bureaucratic and cross-organizational barriers [33, 48]. Support is also needed to lower the costs of citizens engagement and to improve its effectiveness [34], by providing collaboration settings able to lower the perceived power asymmetry between public officers and citizens [46]; by enabling the monitoring and iterative evaluation of the process [48]; and ensuring its accountability and sustainability [24].
Addressing these challenges with the usage of ICT requires a change of perspective, since the problem is not focused on a specific stage or aspect of co-production but spreads through the process and requires an overarching approach. However, the concept of providing digital support throughout the entire co-production journey, considering the flexibility required to deal with diverse governance contexts and diverse application domains, is still underexplored [6]. More studies are also needed to evaluate the impact and effects of ICT solutions on co-production practices [47]. Research gaps in the current literature are thus related to (i) how technology designers and developers should approach the implementation of digital platforms supporting co-producers throughout their venture, including the possible replication of concluded successful processes, as well as to (ii) the evaluation of systems deployed in real use cases.
The work presented in this paper contributes to filling these specific research gaps by making sure that the investigation to the main research question RQ above included the following research subquestions:
RQ1: Which are the requirements that should drive the design of a digital platform facilitating the entire co-production process, starting from the initial idea generation to the actual collaborative delivery of public services and their sustainability?
RQ2: Which are the aspects of digital support that are most valued by PAs, when evaluated in diverse and concrete co-production scenarios? And what can we learn to improve the design of governance platforms for ICT-based co-production?
Stemming from the research questions and adopting the Design Science research methodology [63], our research aims to tackle the social and technical challenges of ICT-based co-production. This serves as the foundation for designing, developing, and evaluating a computer-supported cooperative work environment [36] to facilitate co-production processes, i.e., those collaborative processes between PAs, private stakeholders and citizens that aim at the design of public services, their implementation and shared delivery to the community [76]. The ultimate objective is the realization of a digital collaboration platform with a user friendly and accessible front-end that offers: (i) guidance on how to organize a co-production initiative and a network of stakeholders; (ii) functionalities to support collaborative work; (iii) a catalogue of enablers (in the form of reusable knowledge and digital resources) to aid the execution of a range of actions required to design and produce a public service; (iv) guidelines and methods to make sure the produced service complies with national and international regulations and directives (e.g., for electronic identification and General Data Protection), and properly addresses governance issues, business and societal aspects (e.g., through partnership agreements, sustainable business models). To reach this ambitious objective, our research started from a background analysis of the problem, which then informed the identification of the objectives and of the multi-dimensional requirements that come into play and that depend on the governance models impacting on co-production processes, on the specificities related to the type of public service to realize, and on other recognized standards on usability and technological constraints (RQ1). Requirements were then converted into design solutions for a digital collaboration platform and into constraints for prioritizing its functionalities and developing the first software prototype. An evaluation study allowed testing the perceived usefulness and impact of the system with three different PAs and related networks of stakeholders, which resulted in further requirements for platform extension. The lessons that have emerged point at the importance of step-by-step guidance and of potential reuse (with adaptation) of ready to use resources and processes, but also at open challenges for future work in this field (RQ2).
The paper is organized as follows. Section 2 discusses the related literature. Section 3 introduces the envisaged concept of a digital platform to support processes of public service co-production and describes the overall methodology used to set up our research plan to answer the general research question RQ. These introductory sections provide the background for the multidimensional requirements that are discussed in Section 4 and that provide an answer to research subquestion RQ1. Section 5 describes the first prototype that was designed and implemented out of the distilled requirements, while Section 6 presents the results of the user study aimed at investigating RQ2 and that was conducted with representatives of three different PAs (in Latvia, Italy, and Spain), involving about 224 participants engaged in co-production. The discussion in Section 7 elaborates on the research findings to distill general guidelines to help the work of designers working on platforms for ICT-based co-production. The conclusion summarizes major findings and critical issues, with directions for future work.

2 Related Work

2.1 Co-production and New Forms of Governance

Co-production of public services means that services are not only delivered by professional and managerial staff in public agencies but also collaboratively produced and then delivered by citizens and communities [8]. For this reason, co-production is a more inclusive way of developing and delivering public goods and services, a new form of governance in which citizens and private actors - including non-governmental and civil society organizations - are involved in the overall creation and execution process, as well as in the long-term sustainability of services [76]. Co-production emphasizes people as active agents, not passive beneficiaries; and, in large part because of this alternative process, it tends to lead towards better outcomes in the long-term. This growing interest towards new and more participative governance models has emerged because of the failure of the traditional top-down system for providing public goods and services [27] and because of the recent socio-economic crisis and the related challenges public services are facing, such as increased demand, rising expectations and, in many cases, reduced budgets [12]. Linders [50] points out that the distribution of power and responsibility among public and private actors may vary significantly, giving rise to different governance models underlying co-production. “Citizen Sourcing” (or Citizen to Government, C2G) happens when citizens help the government improve public performance and more effectively satisfy the public's wants and needs. “Public-Civic Partnership” (or Government with Citizen, G+C) occurs when the government and citizens share equal power and responsibility. “Government as a Platform” (or Government to Citizen, G2C) occurs when the government helps citizens improve their day-to-day productivity, decision-making, and well-being by offering a facilitating platform. “Do It Yourself Government” (or Citizen to Citizen, C2C) occurs when citizens help themselves and one another; in this case, the government plays no active role in day-to-day activities but may provide a facilitating framework. Different governance models imply different collaborative processes, which therefore can be facilitated and impacted by the introduction of ICT in a varied way [50].
Despite the growing recognition of the benefits of co-production, studies suggest that co-production complicates governance and accountability, and needs high levels of trust among the various parties to successfully achieve results [78]. To achieve co-production processes and effectively impact policies and mainstream public services, several requirements have been identified. Firstly, the right organizational structures and culture should be developed, with low centralization and high connectedness among actors, and with an ethic of mutuality and equality between the different actors involved, including civil society [73]. Three main interconnected sets of resources and capabilities are required to carry out successful co-production initiatives: (i) analytical resources to ensure policy action; (ii) operational resources to mobilize and deploy means, to carry out coordinated actions, and engage policy networks, communities, and individuals; and (iii) political support and resources to ensure political legitimacy and two-way communication with non-state actors [38]. Finally, co-production calls for citizen participation and for a more active citizenship in service delivery, in addition to a more active role for third sector organizations [60].
The work presented in this paper particularly focuses on operational resources to facilitate coordinated action and to monitor and manage the involvement of heterogeneous networks of stakeholders. We also take a technological stance to investigate the supportive role of ICT.

2.2 Models of Co-production

Co-production encompasses a diverse range of practices and activities. Depending on the level of citizens' involvement, on the type of service, and on who is taking the initiative, different strategies, activities, and responsibilities may be needed. The identification of the actual sequence of phases to be followed by networks of public-private stakeholders to co-produce services is a subject of debate and research. Besides the main co-design [83] and co-deployment phases [11, 1], studies recognize the crucial importance of the preliminary phases of commissioning [55], and of political sponsorship [48], stakeholders’ engagement [83], assessment [55], and sustainable service maintenance in the long-term [7]. Although it is not possible to freeze co-production into a one-fits-all model of programmatic steps to perform, it is possible to identify phases and activities that are crucial for every co-production initiative (like formulating a shared problem definition or identifying and engaging stakeholders), phases that should iteratively occur periodically (like evaluation points and go-no-go moments), and optional phases that may be included in certain co-production contexts (like the planning of maintenance for co-produced digital services). Brandsen et al. [7] propose an analysis of co-production phases and the related governance issues that stakeholders should consider and solve. Sample phases and subphases are shown in Table 1. For example, governance issues for the subphase “Identification and engagement of stakeholders” are: Which stakeholders should be involved in the project? Who would use/provide/be affected by the new service? When to involve stakeholders?
Table 1.
Co-EngagementCo-DesignCo-implementationCo-sustainability
- Identification and engagement of stakeholders
- Formulation of a shared problem definition
- Co-evaluation
- Problem and solution exploration
- Service design and specification
- Co-evaluation
- Service implementation
- Service delivery
- Co-evaluation
- Service handover and maintenance
Table 1. Sample Co-production Phases, as Discussed in [7]
In our ICT-based research, the need to accommodate the variety of co-production models and processes stemming from different governance contexts and application domains has been considered as an important requirement for flexibility and guidance: digital tools need to help co-producers build capacity and find the tailored path that solves their specific problem.

2.3 ICT-enabled Co-production

Governments increasingly rely on ICTs to encourage the co-production of public services by their administrations and citizens [53]. “ICT-enabled co-production” is the term used to refer to the use of ICTs to support engagement in the co-production of public services [16]. The role of ICT matters in both instrumental and institutional terms [53]: on the one hand technology facilitates practices of co-production; on the other hand, it transforms these practices into more social and ‘playful’ interactions. Digital technology has the potential to facilitate new connections within the community, enabling the establishment of relationships that were previously hindered by geographical dispersion of users. Moreover, it can empower participants by fostering shared sovereignty and responsibilities in the co-design of services [61]. The impact and role of ICT also depends on the policy domain, the institutional situation, and the existence of citizen communities. Lember et al. [47] propose an analytical framework to account for the conditions in which digital technologies impact on various elements of co-production, starting from the assumption that four elements can be identified in co-production: establishing interaction, motivating the people involved, bringing resources, and shared decision-making. A literature review on ICT-enabled co-production [87] suggests that ICTs are often used by PAs with the objective to improve effectiveness, efficiency, engagement, and inclusiveness in co-production. In fact, ICTs may improve the understanding of public needs due to the possibility of more easily collecting and exploiting large amounts of input from citizens (effectiveness). They also help reduce PAs' costs for service design and delivery by scaling-up networks of public involvement and taking advantage of citizens' and other stakeholders' time, knowledge, and expertise (efficiency). Furthermore, they allow democratic participation and interaction (engagement and inclusiveness). Yuan also notices that, when combined with the Internet of Things technology, ICTs also offer new ways to collect and process large amounts of data useful for delivering public services, both with the active collaboration of citizens or in an automatic way [87]. However, authors also suggest that it is hard to predict whether ICT will positively impact co-production and that the associated effects should be studied considering the different groups of actors involved as well as the type of digital technology considered [47]. As a matter of fact, the quality of citizens' contributions, the digital divide, the governments' commitment, and the capabilities to integrate citizens' contributions, might influence the effect of ICT-based co-production [87].
Studies also identified several challenges and barriers that prevent PAs, companies, and citizens from fully exploiting the potential of ICT to co-produce and co-deliver services. Clifton et al. [16] identified barriers such as the shortage of financial resources required to set-up and maintain an ICT-enabled co-production initiative, and the lack of a skilled workforce able to coordinate and execute related activities. Additionally, the lack of planning for the day-to-day ICT use may also lead initiatives to fail. Other barriers are related to the complexity of legal regulations that can prevent governments from adopting ICTs and can cause delays in the deployment of solutions. Finally, the lack of technical literacy along with cultural barriers - such as a negative attitude toward ICTs - could lead to a scarce participation of both citizens and government staff: while government staff may perceive ICTs as potentially controlling and intrusive, introducing too much rigidity to the organization, citizens might fear that ICTs could invade their privacy. Other barriers identified by Sorrentino et al. [76] are related to the often informal and experimental nature of most co-production initiatives, and to “the difficult scaling up and dissemination of the experiences”.
Brandsen et al. [6] elaborated an interesting comparative analysis of eight recent digital platforms supporting co-production. What emerged from the study is that often the devised digital solutions are tailored to support specific phases of co-production, either co-design or co-delivery, for example to help users create groups to unite people around specific themes, projects, or cases and let public organizations actively search for local residents and experts who can contribute to policy-making processes (co-design) [81]; or to offer citizens and companies a smart infrastructure to carry out and fund bottom-up services and activities in their local neighborhood, also leveraging open public data and energy management (co-delivery) [77]. Digital platforms spanning across different co-production phases do exist but are usually customized to work in specific application domains, like participatory urban planning [41] or collaborative information provision and refinement about public services [62]. The concept of providing flexible digital support throughout the entire co-production process, for diverse governance contexts, and for diverse application domains, is still underexplored.
In the research presented in this paper, we explain how the challenges and barriers to ICT-enabled co-production emerged from previous research have been translated into requirements for digital functionalities and properties to guide the design of a collaborative environment operationalizing co-production processes (in reply to research question RQ1). To contribute to filling current gaps in understanding how ICT impacts co-production, an iterative design-develop-evaluate approach was adopted involving PAs not only as initial informants, but also as essential evaluators and feedback providers of the implemented solutions through two evaluation iterations (to provide input for research question RQ2).

3 Methodology

The investigation presented in this paper was conducted as part of the INTERLINK research project that aimed at answering research questions RQ, RQ1 and RQ2 by designing and testing an ICT-based platform that facilitates public service co-production and provides knowledge and software resources to guide and monitor service creation and delivery [51]. Digital functionalities required to support co-production go beyond the functionalities offered by general-purpose platforms for online teamwork or project management. It is expected that more targeted information and guidance is necessary on how to create inclusive networks of stakeholders; how to motivate them to participate in the long run; and how to organize and facilitate activities for collaborative problem analysis and service design. Practical examples, templates and reusable materials can make a difference in creating PA's capacity for co-production. Reusable applications and software components compliant with national and international regulations for public services could also reduce development costs and effort. At the same time, clear examples of co-business models for the long-term sustainability of services may be determinant to incentivize co-delivery. In addition, regulations impose specific restrictions and recommendations on the digital services adopted by PAs [25, 26]. Therefore, the INTERLINK research work lies at the intersection between the fields of Computer Supported Cooperative Work (CSCW) [36], eGovernment [32, 19], and software development for PAs [26, 3].
The adopted research methodology is grounded on the Design Science Research Process (DSRP) [63] which identifies six main steps for information systems research: problem identification and motivation, objectives for a solution, design and development, demonstration, evaluation, and communication. The communication step, in particular, maps what in our case study is represented by the distilling of lessons learned and guidelines for ICT-based co-production and their sharing with the research community. DSRP was integrated with the Human-Centred Design (HCD) methodology [44], which is based on a multi-step journey beginning with understanding and specifying the context of use, followed by defining user requirements, generating design solutions, and finally evaluating results. HCD advocates the adoption of multidisciplinary skills and perspectives, as well as the involvement of users and stakeholders throughout design and development. According to the iterative approach at the base of HCD, our research was structured in three iterative cycles of (re)design, development, and evaluation. The initial phases of context analysis and requirement elicitation took advantage of governance research to better understand known barriers and opportunities for ICT-based co-production. Requirements elicitation also integrated the Scenario-Based Design methodology [14] to directly involve stakeholders from the PA sector for envisioning technology future use possibilities.
The resulting methodological approach unfolded in the phases illustrated in Figure 1 and detailed here below.
Fig. 1.
Fig. 1. The methodological approach followed for the design, development and evaluation of the digital platform supporting co-production. Phases in gray indicate the aspects that are discussed in more detail in the paper.
(1)
Problem understanding, objectives definition, and socio-technical requirements. In the initial phase, desk research on co-production and governance literature was performed to identify the known needs and barriers related to the potential usage of ICT to support the execution of co-production processes. This study was complemented with the collection of technical standards for digital platforms for PAs and eGovernment platforms from EU guidelines and publications. To guarantee the imagined technological intervention grounded on actual ways of working, different activities involving local and central PAs were organized using several methodologies, i.e., the focus group methodology [85] to analyze current co-production practices; the personas methodology to analyze stakeholders' needs and roles [17]; and the scenarios methodology to refine shared views of how technology can be incorporated into existing practices [14]. User-centered activities during this phase involved 15 representatives from three different PAs. The collected information was elaborated to translate theory, standards, and user desiderata into actual system requirements, formally described according to an approach similar to the Volere methodology [67].
(2)
Design. A list of functionalities was elaborated and mockups of the user interface were sketched and iteratively validated by user-centered design experts to check the interaction flow before implementation.
(3)
Development of system alpha version. An alpha version of the digital platform was implemented featuring a set of major functionalities to offer guidance to PAs for setting up co-production and for supporting collaborative activities of heterogeneous networks of stakeholders, including citizens.
(4)
User evaluation. Afterwards, usability tests considered the implemented functionalities one by one, to investigate how to improve the user experience and check whether the design succeeded in implementing the initial requirements. Usability testing was carried out by means of a heuristic evaluation session (with three usability experts) [59], and of end-user testing. Six individual sessions were organized with PA representatives to collect focused and structured feedback while users executed a controlled task with the platform, by means of observations and concurrent think-aloud methodology [84]. After the individual user tests, further reflection was stimulated in a collective brainstorming session. Additional usability shortcomings emerged from sessions of free system exploration that involved 22 end-users in total.
(5)
Development of system first prototype. The list of identified usability issues informed a refinement of the system interface design and eventually brought to the finalization of the first consolidated prototype of a digital collaborative environment supporting co-production.
(6)
Pilot system usage. The digital platform was subsequently used by three PAs to carry out three sample use cases of public service co-production involving 224 participants overall, during a time span of 4 months. Usage data was automatically collected by the system, while researchers observed some end-users' activities, and end-users themselves reported their experience in ad-hoc questionnaires (88 respondents). 
(7)
Post-pilot reflection. At the end of the pilot experiment, a reflection phase was organized in collaboration with the three involved PAs (12 representatives) to evaluate how the digital platform was used in practice and how it impacted on co-production activities. The evaluation was facilitated by a structured questionnaire and an online workshop where participants were asked to rank current and potentially new functionalities by perceived usefulness.
(8)
Design refinement. The evaluation findings were complemented with a brainstorming session during which end-users were invited to envisage revisions and extensions of the digital collaborative environment.
(9)
Development of the final prototype and lessons learned. The redesign indications enabled the development of a second, improved system, and were generalized into lessons learned and guidelines for ICT-based co-production.
In this paper we particularly focus on the steps highlighted in gray in Figure 1, i.e., on discussing the analysis of the socio-technical requirements (step 1); the design choices related to system functionalities with a description of what was implemented in the system first prototype (steps 2 and 5); and the results of the system evaluation and post-pilot reflection phase, with related re-design requirements, and the lessons learned that emerged from this articulated process for ICT-based co-production (steps 7, 8, 9).

4 Understanding the Context and Distilling Socio-technical Requirements

To start exploring our general research question (RQ) on how to effectively design a digital platform that can support co-production, we went through the first two steps of the Design Science methodology, i.e., the problem identification and the definition of the objectives for the information system. The aim of this stage was the specification of a detailed set of socio-technical requirements that describe in which way the system should solve the many facets of the problem. Indeed, requirements elicitation is the process of learning and understanding the problem to be solved and the needs of users and related stakeholders, with the ultimate aim of communicating these needs to the system developers. It is a multifaceted process that requires understanding the application domain, current practices and imagined scenarios, as well as the lessons learned from similar systems [88]. Identifying socio-technical requirements entails considering human, social, and organizational factors as well as technical factors to guarantee that the technical and organizational aspects of a system are considered together [5].
This means that for the elicitation of the socio-technical requirements informing the design and implementation of a digital platform for co-production, it was necessary to integrate different perspectives. First of all, to improve our understanding of the application domain we turned to contemporary research studies addressing new governance models open to civil society participation. This analysis allowed us to distill key insights from existing findings on opportunities and challenges related to ICT-based co-production to be translated into practical guidelines for high-level system design (top-down governance requirements). Secondly, talking with potential users about concrete and diverse use cases helped understand how the specificity of the public services to design impacts on the necessary flexibility and customization of the digital platform (bottom-up use case requirements). Lastly, a collaborative environment that supports the co-production of public services (and thus provides digital functionalities to PAs, private stakeholders and citizens for them to work together, contribute ideas, take decisions, negotiate service co-delivery,…) needed to comply to all the standards envisaged for digital platforms used by PAs as well as for e-Government platforms that provide a contact point between PAs and citizens (technical requirements).
For simplicity, in the following sections the targeted platform will be named “Collaborative Environment” (often abbreviated as CE).

4.1 Co-production Problems and Top-down Governance Requirements

As principles related to collaborative governance models are essential to understand how technology should be designed to support co-production practices and cross-organizational collaboration, a systematic literature review was conducted on the influential factors and methods of citizen engagement within co-production processes, which attempts to summarize and combine existing research on the topic of digital and non-digital co-production and identify both findings and gaps [7]. Further study work was focused on the challenges and opportunities for ICT in co-production [16, 38, 47, 48, 53, 61, 76, 87]. The findings that emerged from the literature were analyzed and translated into high-level requirements for the Collaborative Environment, as follows:
R1. Raise awareness and build capacity in co-production. PAs who are new to co-production might have difficulties in understanding if a collaborative approach is suitable for them. This is related to the novelty that participatory approaches represent for PAs, to their scarce experience, and to the lack of good practices and success stories that help build capacity [49]. The CE should provide orientation information to help PAs better frame their context and understand how co-production can help.
R2. Overcome cultural barriers by providing clear added value for co-production. Negative perceptions and fears on the part of government staff about ICTs may significantly undermine their active participation in ICT-enabled co-production [16]. Flexible and adaptable solutions are required that do not introduce additional bureaucracy to processes. New digital tools should provide information and guidance closely related to existing work practices and to the specific co-production task users need to undertake, so that they can clearly perceive the added value with respect to general-purpose tools they already use [49].
R3. Guide users in managing the process. There may be different co-production paths according to different government models, stakeholders involved, and types of services [9, 11, 19, 1, 55]. To avoid PAs getting lost in the process, the CE should provide effective step-by-step guidance by suggesting actions to perform and appropriate resources to use to help complete such tasks, even though flexibility should be guaranteed.
R4. Support the management of the network. The lack of proper management and leadership of the collaboration network, and the fact that nobody “owns” the outcome results might lead to low commitment to the project [64]. This is particularly amplified by the specific inter-organizational and intra-organization specificities of public bodies and the complexities of institutional relationships [49]. A collaborative venture needs a reputable stakeholder to play the role of initiator and facilitator of the collaboration, i.e., the "convener" of the collaborative network [80, 45]. The CE should guide coordinators (or conveners) in understanding which skills are needed in the different phases of the process, defining which the related roles and responsibilities are and making them transparent to other members, supporting the network of stakeholders in sharing knowledge, open discussion and decision making. A clear coordinator role should be defined, with access to advanced functionalities that are hidden from other participants.
R5. Offer a catalogue of reusable resources to lower the financial impact of ICT use. Shortage of finance is a common barrier to a government's promotion of ICT-enabled co-production and financial impediments to the deployment of ICTs in co-production are reported as lower when governments or public agencies opt to use low-cost ICTs [16]. This approach implies the risk of fragmenting the digital support to co-production into many different tools, thus increasing complexity and conflicting with requirement R2 above. As an alternative approach, the CE should experiment with a centralization of resources, so that PAs dealing with co-production can find in a unique reference place reusable resources to ease their task. Digital functionalities should be made available within the platform to support collaboration dynamics. At the same time, a selected number of reusable software components should be made available to lower the implementation costs of the targeted public services. Guidance must be provided on sustainability issues, and on how to select the most appropriate third-party ICTs considering different aspects, like maintenance costs and regulatory framework.
R6. Contribute to technical capacity with knowledge resources. The lack of a skilled workforce constitutes a barrier to co-production. Furthermore, the lack of planning for the day-to-day ICT use may lead to the failure of ICT-enabled initiatives [16]. The CE should provide information on how ICT can be exploited throughout the whole co-production process (e.g., practical recommendations, best practices, risks related to potential technical failures, etc.). Furthermore, the CE should incorporate targeted knowledge resources to meet the specific requirements of the PAs that are opening their processes of public service design to relevant stakeholders [56, 83]. The objective is to operationalize knowledge by combining co-production know-how with a digital platform that guides and supports its use [79].
R7. Enforce transparency and ensure privacy. A lack of trust in the government tends to reduce the participation of citizens in ICT-enabled co-production. Additionally, citizens could fear that ICTs may invade their privacy [16]. The CE should ensure the transparency of collaborative processes and deploy privacy-preserving solutions that are clearly communicated to users.
R8. Provide legal framework. Another barrier experienced by governments is the complexity of legal issues that can prevent them to take-up ICTs [16]. For instance, the deployment of ICT solutions in co-production faces significant delays due to complex challenges related to legal regulations concerning data protection and privacy. The digital platform should support privacy by design [15] and offer guidance on how reusable software may be deployed and managed in compliance with norms and regulations.
R9. Usability of solutions and training. Lack of technical skills, along with a negative attitude toward ICTs tends to reduce the participation of citizens in ICT-enabled co-production [16]. More in general, the adoption of a new technology is affected by many factors. Usability is an important factor, but other factors play a crucial role: accuracy, price, physical appearance, security, function, interoperability, and robustness are all independent factors affecting user acceptance [42]. The CE should promote bottom-up processes and ICT training as measures to encourage stakeholders’ participation. Besides, the CE should ensure high usability and acceptability through a design process that iteratively evaluates the developed solution with end-users and incorporates their feedback.

4.2 User Research and Bottom-up Use Case Requirements

To ensure the design of the CE was user-centered, a scenario-based design approach [14, 68], enriched with the use of personas [17], was adopted to foster an active participation of stakeholders and end-users in the definition of requirements. Scenario-based design consists in presenting and discussing stories that represent a specific problem or technology in use with different purposes [10], namely to identify needs and problems; to present and situate potential solutions; and to illustrate and discuss alternative solutions.
Three use cases of PAs in Europe in need of co-producing a public service in collaboration with a heterogeneous network of public-private stakeholders were involved in the requirements elicitation process. The three use cases were selected to represent a variety of intents and dynamics: e.g., central vs. local PAs; low vs. high citizens involvement; and governance models involving different shares of power and responsibility. This variety ensured that distilled requirements incorporated the flexibility inherent in the co-production process, as discussed in Sections 2.1 and 2.2 above. 
(1)
The Italian Ministry of Economy and Finance (MEF for short) was interested in investigating whether co-production methodologies can help produce digital services for the public sector with high acceptance and adoption rates. To this aim, they focused on testing the co-design of a new digital tool to support strategic planning tasks of public bodies. They envisaged a co-production process involving the participation of PAs, civil servants, government stakeholders, and service companies [49]. The governance model underlying this use case is Government as a Platform (G2C) [50], a specific type of co-delivery, in which the government offers a digitally enabled governance framework for the delivery of services by other public bodies, citizens and private actors.
(2)
The Latvian Ministry of Environmental Protection and Regional Development (VARAM for short) was interested in investigating whether a co-production approach increases the usability of citizen sourced information for policy shaping. To this aim, they experimented with the collaborative refinement of municipal digital services’ descriptions with the involvement of municipalities, citizens, digital agents, and state customer service centers. The governance model underlying this use case is Citizen sourcing (C2G) [50], which occurs when the government designs and delivers a service but asks citizens for the voluntary commitment of resources to improve the service itself.
(3)
The Municipality of Zaragoza, Spain (ZGZ for short) was interested in widening Open Innovation within the city by co-creating and co-delivering services together with citizens. To this aim, they experimented with the co-production of initiatives to take place in Etopia, a public space in the city hosting creative and innovation activities. The governance model underlying this use case is Public-Civic partnership (P+C) [50], which occurs when the government, citizens and private actors share equal power and responsibility. Each side brings their unique advantages to the table to collaboratively solve problems and create public value.
The three analyzed use cases highlight the potentially large heterogeneity of co-production processes that may differ by objectives of PAs in relation to co-production; actors involved; level of participation of citizens or associations of citizens; type of service to be co-produced; and resources needed to carry out the co-production process.
During an initial focus group, 15 PA representatives (10 for MEF, 3 for VARAM, and 2 for ZGZ) were invited to reflect on their co-production objective and on the social and organizational context in which co-production would take place. A two-page template was distributed to collect information on the public service to be co-produced; on the desired future scenario in which the service would be delivered; key actors and roles; related initiatives; available resources, desired additional resources and desired digital support to sustain the co-production of the public service.
After this preparatory activity, use case coordinators were guided through the generation of personas (6 for MEF, 6 for VARAM, 4 for ZGZ, taking into account different actors engaged in co-production) and the description of their relationships. Personas, i.e., fictitious profiles of service end-users and stakeholders [69] that act as a benchmark for design, encouraged teams to reflect on the actors involved in the service to be co-produced, i.e., on the potential co-producers. The generated personas were used to describe three scenarios of co-production, i.e., the process by which a network of stakeholders collaborates to create and deliver a new public service. PA representatives were invited to imagine co-production needs that could be solved by means of digital technology and reusable knowledge resources. Table 2 summarizes the requirements that PAs mentioned as useful in their use case. As it can be observed, some of the requirements refer to functionalities that could be integrated directly inside the Collaborative Environment, such as tools for workplan scheduling or tools for managing incentives and rewards to sustain participation (U1, U9). Other requirements call for software modules to be used to support specific tasks, such as an editor to collaboratively revise texts appearing in web pages (U8, U11–U14). A third class of requirements refers to reusable knowledge (guidelines and template documents) to bootstrap activities (U2–U7, U10). 
Table 2.
MEF use caseVARAM use caseZGZ use case
Functionalities in CE
U1 Tools for work plan scheduling
Reusable knowledge
U2 Template for an awareness campaign to raise participants’ interest
U3 Guidelines and templates for stakeholders mapping and engagement
U4 Service design tools for workshops, interviews, and surveys; guidelines for co-design
U5 Questionnaires for quality-of-service monitoring
U6 Tutorials on the CE and related resources
U7 Privacy consent forms compliant with GDPR
Reusable software
U8 Ideas crowdsourcing and e-voting tools
Functionalities in CE
U9 Incentives and rewards to sustain participation in collaboration
Reusable knowledge
U10 Quality of service surveys
Reusable software
U11 Document editor which allows users to annotate e-service descriptions on the web with comments, questions, answers, terms which can be browsed, queried, or even suggested to users when accessing different parts of a web document. The annotations can be voted, commented, extended by other users in a Wiki-like manner
Reusable software
U12 Software module that allows the collaborative planning of events through functionalities for managing and booking resources
U13 Engagement tracker to manage feedback received from citizens plus analysis of attendance and registration rates
U14 Loyalty module to reward participation of citizens in collaborative activities and long-term engagement
Table 2. Requirements Mentioned by PA Representatives to Support their Use Case
Reusable knowledge and reusable software may vary significantly across use cases and a rich and extendable catalogue of such resources, which we call INTERLINKERs, should be made available.

4.3 Standards and Technical Requirements

Janowski [39] argues that the Digital Government concept has evolved across four main stages: Digitization (Technology in Government), Transformation (Electronic Government), Engagement (Electronic Governance) and Contextualization (Policy-Driven Electronic Governance). ICT-based co-production falls within the category of technology that involves the engagement of a plurality of public and private actors in the generation of solutions and the co-creation of public value, thus enabling government to make a transformation to a more open and participatory model that they see as more desirable [54]. Therefore, ICT-based co-production can be seen as spanning across Electronic Government and Electronic Governance, implying a transformation of the internal working and structures of government as well as its relationships with citizens, businesses, and other stakeholders [39]. Previous studies and projects on Digital Government let emerge particular requirements that must be taken into account, not only during the implementation phase but also at the early design or architecture modeling phases. Technical requirements refer to all those features that the platform should implement and the constraints it should satisfy to be interoperable and compliant with national and international regulations. Table 3 summarizes a set of common characteristics of Digital Government architectures that have emerged from a literature review by Baheer et al. [2].  Most of these requirements are also explicitly endorsed by national and international regulations and recommendations, such as for example the EU eGovernment Action Plan that provides guidance for all countries in the European Union [25]. Specific requirements address the issue of ensuring that the platform can be easily adapted and used by different PAs and with a cross-sector and cross-border dimension.
Table 3.
RequirementDescription
T1. Interoperability and integrationInteroperability and integration between data and applications and with various information systems
T2. Secure architectureHaving a secure architecture to ensure higher security of hardware and software to build trust with users
T3. AdaptabilityAdaptability to changing requirements that can have technical, socioeconomic, legal, and/or political nature
T4. FlexibilityFlexible integration of architecture's components to better align business processes and technologies
T5. ReusabilityReusability of components to be used in more than one system
T6. ResilienceResilience to changes in the service environment
T7. CompatibilityCompatibility of a Digital Government architecture with the already existing infrastructures, such as legacy systems and multiple public institutions integration in different environments
T8. Single Sign-OnProviding citizens the Single Sign-On (SSO) service, i.e., the possibility to access several electronic services offered by the public sector with the same identification credentials, without re-entering authentication factors when more services are open
T9. TraceabilityTraceability of system operations performed by specific system users, also as a support to accountability
T10. UsabilityUsability, i.e., providing functions that are required for better system performance
T11. Cross-borderCross-border characteristics, i.e., providing Digital Government services in an international context and managerial settings 
T12. ScalabilityScalable to host a large number of digital services
T13. LegalityProviding Digital Government services according to relevant legislation and judiciary
T14. Cost-effectiveness The architecture should be implemented in a way that the deployment and operation resources are kept to a minimum
T15. Technological neutralityThe architecture must ensure that no components included in its definition advocate specific suppliers
T16. Platform independenceThe architecture is not dependent on a particular technology platform implementation nor assumes a particular technology
T17. Minimal learning curveNeed for limited training to government employees to implement or use the architecture
T18. ComprehensibilityThe architecture should be well-defined, and understandable with strategic clarity by the Digital Government leaders
T19. Citizen-CentricThe Architecture should be designed in a way to support the strengthening of the relationship between citizens and the government
Table 3. Technical Requirements for Digital Government Platforms, as Summarized by Baheer et al. [2]
These technological requirements guided the baseline architectural choices of the platform and the development process. For example, to put in practice technological neutrality and platform independence, a web-solution was preferred (T15, T16), with the possibility for the platform to work both in single-tenant configuration (i.e., with one instance of the platform for each PA) and in multiple-tenant configuration (i.e., with the same server safely managing different PAs), to help reduce operation costs to a minimum (T14). An authentication and authorization service implementing the Single Sign-On protocol1 was put in place (T8), supporting European identification standards, thus also complying with cross-border characteristics (T11). To comply with the requirement of traceability and accountability of actions, methods for logging user interactions and analyzing behavior were conceived (T9). Multilinguality of the user interfaces and the possibility of customizing them according to PAs' needs was included in accordance with the adaptability and flexibility requirements (T3, T4). As explained in Section 5, an extendable catalogue of reusable resources facilitating co-production was envisaged to cater for the requirements of reusability, resilience, and scalability (T5, T6, T12). To ensure interoperability and integrability, the formalization of reusable software resources was inspired by the Digital Europe Building Blocks methodology [28] (T1, T7). FAIR principles and the General Data Protection Regulation (GDPR) constraints for data and metadata management [86, 30, 75] were percolated through the system architecture to comply with the security and legality requirements (T2, T13). The user-centered design approach adopted as a baseline methodology for the overall project, ensured citizen-centric and usability requirements right from the start (T10, T19). Tutorials and user manuals addressed the comprehensibility and minimal learning curve for all potential users of the platform (T17, T18).
The following section describes the first version of the Collaborative Environment that was designed and implemented according to the collected multiple types of requirements, namely, top-down governance requirements, bottom-up use case requirements and technical requirements.

5 Design Choices and the Implemented Digital Platform to Support Co-production Processes

The multidimensional requirements and the governance concepts described in the previous sections were translated into initial architectural, functional and implementation decisions to design and develop a multilingual web-based portal featuring information services supporting public service co-production processes (see Figure 2). The resulting platform offers the following core functionalities:
public information pages on co-production concepts and potential benefits (in compliance with governance requirement R1 above);
an authentication and authorization service implementing the Single Sign-On property, with general data protection regulation compliance and cross-border authentication (eIDAS) [29], which allows users to efficiently and safely login to the CE and related services (requirements R8, T8, T11);
a guided process to help co-production coordinators set up and configure a digital space to be shared with other participants (Overview view) (requirements R2 and R3);
functionalities for the management of the stakeholders network, with the possibility of grouping participants into organizations and teams, assigning them different access rights to documents (i.e., co-production resources) and tasks (requirement R4);
visualization of alternative models of the co-production process to help users identify which are the most appropriate steps to perform that fit their use case (operationalized governance models) (requirement R3);
creation of a co-production schema that guides the co-production team through the suggested phases and tasks to perform (Guide view). Schemas are representative and adaptable to different co-production scenarios (e.g., G2C, C2G, P+C) and can be customized to suit the phases, objectives and tasks of different types of co-created public services or artifacts (e.g., physical or digital services, but also hackathons, citizen science experiments, and so on) (requirements R2 and R3);
recommendation of reusable knowledge and software resources –the so-called INTERLINKERs–, which may support the simpler and more effective execution of the co-production steps, in compliance with norms and regulations (requirements R2, R5, R6, R8, U2-U7, U8, U10, U11-U14);
in-platform opening and registry of use of INTERLINKERs, with document sharing and collaborative editing (requirement R7);
support for setting a timeline and deadlines for the process (Workplan view) (requirements R4, U1, U9).
Fig. 2.
Fig. 2. Contribution of multidisciplinary requirements to the Collaborative Environment specification.
Further important functionalities were implemented during a second development cycle as a result of new requirements that emerged from the pilot experimentation of the platform within the three project use cases, which will be discussed in the following Section 6.
Currently the languages supported by the end-user interface are English, Spanish, Italian, and Latvian. In the following subsections, we will discuss in more detail significant system behaviors, with the help of interface screenshots.

5.1 Helping Co-production Coordinators Setting Off

Public or private entities wishing to coordinate a new initiative for co-producing a public service might have difficulties in understanding how to structure and manage the collaboration process. For this reason, a special “Overview” area is available to process coordinators with a checklist on how to proceed, as advocated by governance requirements (R1–R3).
Coordinators are instructed (i) to create a title and goal description for their process; (ii) to assign administrators to the newly configured process; (iii) to select the schema that best describes the collaborative process matching their co-production goals; and (iv) to invite teams of other participants to join the process. Step (iii) in the sequence is particularly important, as it offers coordinators examples of co-production process models, which are called “schemas” in the platform. These schemas are represented in the form of suggested sequences of phases for the co-creation of a public service (e.g., Engage, Design, Deploy and Sustain), which can expand into more detailed objectives (e.g., Engage expands into Identify Stakeholders, and Engage Stakeholders), and tasks to perform to reach each objective (e.g., “Understand the different types of stakeholders” or “Map stakeholders, analyze motivation, skills, expectations”). Figure 3 illustrates how co-production schemas are graphically represented in the user interface. The devised Collaborative Environment offers alternative co-production schemas to choose from, with the possibility of additionally customizing templates by removing steps (i.e., phases, objectives, or tasks) that are not relevant for a specific use case, or by adding new ones (in compliance with the flexibility advocated by requirement R3). Figure 3 and Figure 4 illustrate two examples of different co-production schemas, one with default steps for the co-design and co-delivery of a public service where citizens are involved in all phases (Figure 3), and one with suggested steps to foster the ideation of social innovation through hackathons involving citizens (Figure 4). The set of predefined co-production schemas offered by the platform was identified through governance research [7] and through the analysis of co-production use cases [48, 18].
Fig. 3.
Fig. 3. Sample screenshot of the “Guide” view for a default co-production process articulated in phases: Engage, Design, Deploy, and Sustain. The image also shows (on the right-hand side of the interface) a set of shared resources associated with one sample task.
Fig. 4.
Fig. 4. Sample screenshot of the “Guide” view for a co-production process for the organization of a Hackathon articulated in phases: Engage, Design, Run, and Exploit. The image also shows the drop-down menu listing generic software enablers (INTERLINKERs).

5.2 Guiding Participants through the Co-production Phases

Once the coordinator has chosen the co-production schema and has possibly further customized it, the “Guide” view helps the network of stakeholders orientate within the process by visualizing the suggested list of tasks and what is required to accomplish them (requirement R3). For each task, a work area is available where the co-production team can share resources, like documents and links (under the “Resources” tab on the right part of the interface in Figure 3). A button allows users to visualize a list of recommended knowledge and/or software enablers (called INTERLINKERs) that could help to complete the task (see the button “Instantiate task resource through recommended INTERLINKERs” in Figure 3). For example, for the task “Ideas collection for service definition” that happens during the Design phase, a bespoke knowledge resource is suggested containing guidelines on how to organize focus groups. The public resource can be reused by cloning and editing it for the purposes of the specific co-production project and can then be saved in the task workspace.
A special category of generic software INTERLIKERs is also available offering baseline collaboration functionalities, such as the creation and sharing of documents (via integration with Google Drive APIs2); the creation of surveys; the creation of discussion forums (via integration with Loomio3); and the sharing of external links (see the drop-down menu shown in the right part of the interface in Figure 4).

5.3 INTERLINKERs as Knowledge and Software Enablers for Public Service Co-production

The catalogue of INTERLINKERs, defined as reusable resources offered to build capacity in co-production according to governance requirements R5, R6 and R8, represents one of the core concepts that characterize INTERLINK's CE. INTERLINKERs can be used for different purposes:
To guide co-production: These enablers are in the form of special contents and interactive flows that guide and support teams in the collaborative execution of the co-production initiative. Knowledge enablers in this category are for example: canvases to perform stakeholder analysis; templates for stakeholders’ engagement plans; and guidelines and materials for workshops for service design. Software enablers are for example: tools for ideas crowdsourcing and collaborative decision making; tools for surveys; and tools for team management and document sharing and management.
To build operational capacity: These are knowledge resources which tackle the legal, social, and business aspects to make co-delivered public services viable and feasible in time. Knowledge enablers in this category are for example: guidelines on GDPR for data protection; sample information sheets and consent forms; guidelines on the acquisition and reuse of software for PAs; and templates for business plans.
To aid service development: These are reusable software modules and development frameworks, aligned with other existing EU-wide initiatives, to foster interoperable and sustainable public services. Software enablers supporting service development are for example: a registration and authentication component compliant with eIDAS protocols; and a reusable gamification engine to introduce incentives and rewards to sustain citizens' participation to the co-delivery of public services.
Overall, 60 INTERLINKERs were uploaded in the Catalogue integrated in the first version of the CE. However, the approach used to ground the design and the development of the INTERLINKERs catalogue was purposefully aimed at ensuring future flexibility and extensibility of the catalogue with new enablers of different types (technical requirements T5, T6, T12). In fact, the set of INTERLINKERs resulting from the second development iteration of the platform, after the evaluation described in Section 6, counted more than 70 enablers.

6 User Evaluation Study: Pilot Adoption of the Platform in Actual Use Cases and Identified Extensions

To execute the phases that in the Design Science methodology are called as demonstration and evaluation, the three use cases described in Section 4.2 were used as a testbed to evaluate the CE both from a technical perspective and, most importantly, from a user standpoint. As explained in Section 3, preliminary usability tests involving three human-computer interaction experts and 22 end-users evaluated the CE interface against Nielsen's usability heuristics [58]. The collected feedback allowed identifying several usability problems that were immediately passed on to developers to improve the user interaction experience and consolidate the first prototype of the system - readers interested in how the usability study was organized can refer to [52]. This section reports instead the results of the second evaluation study aimed at gaining a deeper understanding of how the digital platform was used in co-production use cases and what can be learned on ICT-based co-production, to reply to research question RQ2 stated in the Introduction section.
The three PAs coordinating the use cases used the CE described above to facilitate their co-production activities involving 224 stakeholders in total (68 for MEF, 100 for VARAM, 56 for ZGZ), during a 4-month period. Several monitoring and evaluation tools were used to capture users' experience throughout the experiment: interaction logging and automatic measures of the usage of system functionalities; self-reported activities description [52]; observations and interviews [49]; a questionnaire investigating the experience of participating in co-production; a questionnaire investigating the quality of co-created public services; and final post-pilot reflection questions and focus groups on the usage of the system. In the following paragraphs, we briefly summarize the major aspects that are interesting from a Human-Computer Interaction (HCI) point of view, addressing in particular the final reflections on perceived usefulness, appropriation, adequacy to governance models, and possible requirements for re-design or extension of the system. Other investigated dimensions addressing the evaluation of digital co-production quality are described in [74, 66].
The reflection questionnaire distributed after the end of the use case activities, included 31 open questions, grouped into nine investigation areas, namely: (1) how the co-production working group was structured in the CE; (2) the level of participation observed in stakeholders; (3) how the co-production process was represented in the CE following the system's guidance; (4) impressions on the alternative co-production schemas offered by the platform; (5) the possible role of examples of what other PAs have done; (6) the perceived need for different roles and access rights to the shared documents; (7) the most used and most useful functionalities offered by the CE; (8) the most used and most useful INTERLINKERs; and (9) the perceived added value offered by the CE. The coordinating teams of each use case (6 PA representatives for MEF, 2 for VARAM, and 4 for ZGZ) were asked to gather together, reflect on the use case outcome, and elaborate an agreed answer to each of the reflection questions.  
Two further reflection focus groups were organized with the same 12 PA representatives to elaborate on the questionnaire results and on suggestions for future extensions, and to draw comparisons across the three use cases. The discussion was structured around themes that had emerged from the questionnaire and that focused on how co-production coordinators actually interacted with the technology to organize their co-production processes and their perceptions about workflows and other stakeholders' engagement with the platform.
Bespoke functionalities and guidance for co-production. On average, use case coordinators observed a good level of motivation to participate in co-production in stakeholders. This was referred especially to those actors that already had awareness of and direct experience with the problem tackled by the co-production process. In the words of the VARAM team, “higher digital skills and awareness of public services delivery significantly raised the motivation”. The real challenge lies in requesting co-producers to introduce a new work methodology or to use a different digital platform than the one already established in their daily work practice. It was mentioned that “for the Collaborative Environment to be adopted as a working tool, it is necessary to maximize the value provided to the team involved in co-governance, co-design, and co-production. This includes enabling time savings and a better quality of the final service” (ZGZ team comment). The use case coordinators noted that intuitiveness of use and simplicity of the system are not the only important factors, stressing the importance of other distinguishing functionalities that are tailored for co-production (confirming governance requirement R2). These include (i) step-by-step guidance to help users understand where to begin and determine the appropriate path to follow depending on their specific co-production model and problem to solve; (ii) efficient strategies to reach out participants and incentivize their participation; and (iii) template materials to speed up the preparation of communications and joint activities. Focus groups also let emerge the importance of additional methods to monitor and stay updated about process progress and of a clear overview of the work carried out so far to facilitate handovers, for example when decision makers delegate practical tasks to other public servants. Figure 5 shows an example Jamboard screenshot with the collective reflections and recommendations generated by users in this regard. These users' recommendations that emerged from the evaluation were immediately taken into consideration by system developers to extend the set of platform functionalities with mailing list support and email and in-app notifications to keep participants updated about latest changes on the process’ resources and task completion, as shown in Figure 6.
Fig. 5.
Fig. 5. An example of the comments that emerged during reflection focus groups. The screenshot shows post-its generated in reply to the question: “What overview information would you expect to find about your project each time you connect?”.
Fig. 6.
Fig. 6. Notification dashboard added in the second version of the CE showing latest changes on the process resources and task completion.
The second version of the CE also featured an enriched information part that provides clearer textual and visual representation of the steps that the users (both coordinators and participants in the co-production process) can follow to create/navigate the phases and the resources for their co-production processes.
Flexible management of the co-production work group. As emerged from the logs automatically collected by the system, different PAs, with different internal organizational/hierarchical structures, interpreted the concepts of “organization” and “team” differently according to their usual work practices and grouped people in their network according to different rules. This is an interesting issue to investigate from the Human Computer Interaction (HCI) point of view, as there might be forms of appropriation, i.e., observed uses of the functionalities, different from what originally conceived by designers, to adapt to existing work practices [21]. The version of the CE that was tested by end-users provided a flexible way of creating virtual organizations and teams to structure the work group of a co-production process. According to the initial socio-technical requirements, this functionality was introduced to control the visibility of activities and of shared documents. Users were free to decide how to group participants in lists and they did actually exploit this freedom to accommodate different interpretations of the available functionality, different co-production goals, and different governance approaches of the process. The comparison between the three use cases revealed that the functionalities for managing the work group were appropriated by co-production coordinators in two main ways: (i) to create just one main organization to represent the overall initiative and different teams to sub-group people from different stakeholder institutions or associations, as in the MEF use case; or (ii) to create multiple organizations, one for each involved stakeholder institution/association, and to create new teams following more practical and extemporaneous needs (e.g., to group participants invited to specific meetings or activities or to separate categories of participants), as in the VARAM and ZGZ use cases. In both cases, co-production coordinators deemed it important to explicitly name institutions (either in teams or in organizations) for formal partnership recognition and to enhance the visibility of the network. In the MEF team's words, “this allowed us to better handle and organize the involvement of the various stakeholders, assign tailored permissions, and be sure that each participating Administration was represented in the MEF co-production process and in the collaborative environment”.
Following the evaluators' feedback, the second version of the CE also featured the possibility of activating a “Call for Collaborations”, by announcing the profiles and skills needed to be part of an open process collaboration force.
Roles and Management of access rights. To the users that participated in the evaluation tests it was clear that the team coordinating the co-production initiative plays the role of administrator of the collaborative process and should have access to the most advanced functionalities made available by the digital platform. Other stakeholders participating in co-production would mostly be contributors or observers, based on their level of engagement in the activity, and should be presented with simplified views that hide unnecessary functionalities of the CE (ZGZ team: “The incorporation of team and user roles and privileges would be more than welcome.”; MEF team: “As we were the ‘administrators’ of the co-production process, this feature allowed us to set up the process and create some guidance resources without the risk of them being modified or deleted by mistake by the stakeholders”). It was mentioned that a “tech-savvy administrator is required” (VARAM team).
According to previous research, different team members may participate in different phases of co-production [48, 49]. This was the motivation for including a fine-grained method in the implementation of the CE for (optionally) assigning permissions at different levels of the co-production schema, to restrict the number of participants that can read or manipulate shared documents (resources) within different phases, objectives, and tasks. Evaluators commented that default permissions assigned to the overall co-production schema would suffice in most cases, possibly using the main categories of participants (administrator, contributors, observers) to differentiate default permission values (as mentioned by ZGZ team: “It would be useful if these different categories of users could be associated with different levels of access to the platform so that what they can do within the environment depends on the level of authorization they have.”).
Guidance through co-production schemas. On the one hand, co-production coordinators highly valued the availability in the digital platform of pre-prepared co-production schemas that recommend which phases and activities would be appropriate to co-design and co-deliver public services in certain governance contexts. On the other hand, choosing which schema fits their specific context was considered challenging (VARAM team: “it was unclear which one to choose”). This choice actually implies being able to recognize the specificities and differences between different co-production processes and schemas. This is in line with the literature findings summarized in previous section 2.2, which recognize factors like the type of stakeholders’ network, the type of service, and who is taking the initiative as impacting on the strategies, activities, and responsibilities characterizing co-production processes [9, 11, 1, 55, 83]. As an important lesson learned for the refinement of the system user interface, it emerged that the step of co-production schema selection requires an appropriate dialogue to guide coordinators' choice based on recognizable properties (e.g., type of organizations involved, level of openness of the process, etc.), in line with MEF team's suggestion: “maybe you could consider adding keywords to further simplify the choice”. In addition, the inherent flexibility of the co-production process implies that a standard co-production model hardly fits a use case without any adaptation; changes to the workplan may also occur in time, or some decisions can be taken just later on (e.g., how to organize the maintenance of the service). For this reason, PA representatives highly valued the possibility, made available by the CE, of customizing a chosen co-production schema by removing or adding phases/objectives/tasks to their plan of work. In the PA representatives' words: “The proposed schemas, although theoretically perfect, are too ambitious to be implemented in real cases” (ZGZ team). “Based on the feedback from participants, it must be possible to adjust schemas manually.” (VARAM team). “The possibility to change the title and description of objectives and tasks, as well as to delete phases/objectives/tasks not useful for our activities allowed us to create a schema suited for our co-production objectives” (MEF team).
It was also mentioned that PAs who are new to co-production may benefit from learning how similar public bodies have successfully applied co-production [49]. Needs like the following emerged: “know what was the impact, the benefit”, “know what worked and also what did not work”, “draw inspiration”, “make an analysis to know if the process is replicable in a similar context”. A catalogue of “co-production stories” could be used to draw inspiration and to understand if a past process is transferable to a similar context by carefully considering whether there is technological, organizational, and institutional fit. Furthermore, coordinators could even clone existing co-production schemas and related resources to avoid starting from scratch when creating new or revised public services. This practice would be in line with what Dolowitz and Marsh [22] call voluntary policy transfer, “the emulation and lesson drawing [which] refer to a process in which knowledge about policies, administrative arrangements, institutions and so on in one time and/or place is used in the development of policies, administrative arrangements and institutions in another time and/or place”. A digital catalogue of best practices in co-production would also help overcome the difficulty of formalization, scaling up and dissemination of the experiences related to the often informal and experimental nature of most co-production initiatives [76]. To implement this form of Open Innovation based on the promotion of replicability, the second version of the CE was extended with the possibility of cloning past processes of a team or cloning from publicly available success cases (“Stories”), with the possibility to subsequently customize the cloned process and the related resources to best fit your own co-production context.
Awareness of co-production process status. PA representatives acknowledged that face-to-face gatherings offer the optimal opportunity for coordinators to monitor stakeholders' level of involvement and to stimulate active contribution. More difficult is to get a sense of the co-production activities performed through the digital platform, such as the post-meeting elaboration of shared documents. Automatic tracking of collaborative activities performed by users was considered useful to generate summaries reporting which stakeholders contribute to what specific collaboration activities. However, more specific suggestions were made on how to support teamwork for co-production. It was mentioned, for example, that it would be useful to directly display in the interface which users have participated in each phase or activity (Figure 5), in order to keep track of participation, to possibly adjust the composition of work groups, and to provide a way to enforce accountability [78]. Visual cues that signal changes and updates to the digital representation of the co-production process since the last login were also considered essential, to let participants keep up with the latest progress. Additionally, there was also a demand for personalization: co-producers should be able to immediately see which actions require their contribution (in PA representatives' words: “which tasks need my work?”, “deadlines and alerts of tasks”).
Another important aspect pointed out by PA representatives was the need to get an intuitive view of the status of the entire process, such as a progress board summarizing the phases and tasks that have already been worked on, those that are in progress and those that are forthcoming, in order to have a sense of how the group work is progressing towards to the final goal (Figure 5). In the words of the VARAM team: “It is hard to navigate within the process, to determine where is what. The process dashboard could be more simplified, including a progress board - maybe something like Kanban?”.4
As a response to these new requirements emerged from the evaluation, in the second version of the CE the possibility of assigning resources to people and see in-app personal notifications with the assigned work was added, as well as a bespoke “Assignments” tab within the Resources view, which summarizes a personal to-do list. Besides, a new “Contributions” tab was added for each task view to allow for the visualization of who has contributed with what level of engagement for the whole task or per individual resource in the task (Figure 7).
Fig. 7.
Fig. 7. Screenshot of the “Contributions” tab available within each co-production task summarizing advancement in the task and contributions by participants.

7 Discussion

As discussed in Section 2, previous research work in public government and co-production has highlighted several benefits and barriers related to collaborative public service creation and delivery. Similarly, research in ICT-based co-production has more specifically identified opportunities and challenges for the introduction of ICT in support of collaborative public service creation. However, these findings are typically not translated into design guidelines and requirements for the implementation of digital platforms that are tailored to support the overall co-production process and are validated through an actual implementation and user evaluation study. The design, development, and evaluation of the INTERLINK Collaborative Environment detailed in the previous sections have brought forth several findings that contribute to the advancement of the field of ICT-based co-production, both in terms of the requirements that a digital platform supporting co-production should satisfy (research question RQ1) and in terms of its acceptance and actual usage in real co-production contexts (research question RQ2).
By merging the output of the requirements elicitation phase (discussed in Section 4) and the lessons learned from the user evaluation of the implemented system (discussed in Section 6), the following general guidelines can be derived to help the work of designers working on platforms for ICT-based co-production.
Convey co-production know-how. First of all, a digital platform supporting co-production needs to include an informative area that provides orientation to new users (PAs, other public and private stakeholders, citizens) and explains in practical terms how the co-creation and co-delivery of public services may unfold and how ICT contributes to enhance the process. The objective is twofold: (i) to increase awareness and interest in co-production to counterbalance the novelty that participatory approaches represent for PAs [49]; and (ii) to let potential users understand the added value of adopting a new technology, helping overcome resistance to ICT change, particularly in less technologically savvy users [4]. The need for tutorial materials, recorded demos, and live training sessions should not be underestimated.
Leverage the power of examples. Providing concrete examples of co-production experiences carried out by other public bodies is essential. PAs seek best practices and clear examples of both successful and unsuccessful experiences of other similar PAs that engaged in the co-production of public services. They are aware of the unicity that characterizes each organizational and application context. Still, checking the co-production process followed by others is of inspiration [22].
Offer a wizard or checklists. Clear guidance should be provided to coordinators on how to set off their own co-production project and take the first important decisions. What is the project about? Which governance model applies? For instance, are citizens involved just in the consultation/creation phase, while the service is to be delivered by the PA, or are both private and public actors involved in service delivery? Who is going to coordinate the activities? What type of roles are relevant? A sort of wizard or visual check-list should be implemented in the digital platform to help users go through the different questions, reflect, and prepare the ground for the actual co-production activities [49].
Balance guidance and flexibility. A diverse range of co-production pathways exists, depending on different government models, stakeholders involved, and types of services to be co-produced [9, 83, 11, 1, 55]. To translate theory into practice, PAs need to be helped to plan the path that suits them best. A digital platform for co-production should support this stage by offering a user-friendly visualization of the multi-step process that is to be performed. In our research we experimented with a tree-like schema that unfolds the co-production into a sequence of phases and nested tasks to perform. A catalogue of schemas that capture alternative co-production paths described in the literature is offered, from which PAs can choose where to start from. However, as emerged from the evaluation, it is essential to grant users the flexibility to customize the phases and tasks both at the beginning and iteratively during the process, to account for the very dynamic nature of co-production practices.
Implement accountability by design. Digital platforms for co-production should address accountability by design [13], by putting in place log mechanisms to monitor interactions with the platform, abstract meaningful actions, and associate performer users. This is to ensure that the system can effectively monitor and visualize the contribution of the different platform users, also distinguishing by roles and assigned tasks. This improves transparency and helps support trust within the network [78]. This is also essential in case coordinators deem it appropriate to associate users' contributions with incentives and rewards [74]. Furthermore, information regarding which participants contribute most to the planned actions and decisions, and how interaction revolves between members of the network, may reveal whether co-production is fair and inclusive and whether it fosters durable ties and synergies.
Make sure a shared archive is provided. One of the functionalities voted as most useful during the evaluation study was the availability of a shared workspace (for collecting documents, presentations, meeting minutes, hyperlinks to other resources, etc.), to be used within each task as a common ground for collaboration. Both a view of resources per-task and a view per-entire co-production process were considered useful, with search and order filters to facilitate the retrieval of material. This is in line with findings in previous studies that identified long-term, structured archives of shared resources as enablers of collaboration [48, 35].
Offer workplan and roles management. As was suggested by the evaluators, a digital platform supporting co-production needs to feature an easy-to-grasp visualization on the process advancement status for multiple reasons. First of all, to help keep up with deadlines and tasks to be completed, but also to increase the awareness of the effort that the whole network is putting on the co-production tasks, thus incentivizing motivation and participation [64] and reducing the impressions of discontinuity and fragmentation [65]. Distinguishing between different roles and responsibilities (e.g., coordinators and participants), emerged as a need not only from the literature [80, 45], but also in PAs' actual practices.
Enable reuse / redo. The PAs involved in our evaluation study largely took advantage of the "reusability" principle that permeates the INTERLINK CE. PAs reused the schemas suggested by the system to set up and customize their own co-production process; the template documents found inside the platform (knowledge INTERLINKERs) to prepare materials they needed for collaboration; and software modules (software INTERLINKERs) to perform some of the tasks. They also cloned co-production processes previously created and used those copies as a seed to start new processes, redoing the steps that already proved successful. To implement these different forms of reusability, the architecture of the system platform must be conceived, right from the beginning, with functionalities that allow cloning, adjusting, and redoing things, so that PAs can iteratively build their own capacity to co-production and learn from one another's experiences.

8 Conclusion and Future Work

Existing general-purpose digital collaboration platforms offer a range of tools designed to support diverse (subsets of) functionalities, including the setting of work plans, task management, the collaborative editing of documents, textual or visual discussion boards, ideas crowdsourcing, joint deliberation, or data visualization to support informed analysis and decisions, among others. However, flexibility and freedom of use come at the expense of guidance, since platforms assume that users are familiar and knowledgeable about the specific collaboration task they must perform. This is a severe limitation for PAs and related networks of stakeholders that are novice to co-production, i.e., simply wishing to involve citizens, NGOs, and private companies in virtuous collaborative processes may not guarantee project success. Background literature reports that proper bespoke support is necessary for building capacity in co-production methods; finding the right workflow of steps to follow; managing the different skills, backgrounds, and competences in heterogeneous networks of stakeholders; finding materials and examples that help bootstrap activities; lowering the financial impact of ICT use; and for periodically checking whether the legal, transparency and sustainability requirements are met. Guidance and exemplification are thus distinguishing features that are essential to provide an added value for co-production.
The research study reported in the paper focused on the problem of investigating how designers and developers should approach the implementation of technology supporting co-producers, as well as on the evaluation of systems deployed in real use cases. We started from an analysis of the problem and of the socio-technical multidimensional requirements to inform the design and development of a digital collaboration platform, where guidance and exemplification fostering reuse are pivotal. Technical requirements for digital government software architectures were integrated with knowledge on governance models and co-production processes and were validated through the analysis of three specific use cases coordinated by one central PA in Latvia, one central PA in Italy, and one local PA in Spain. The developed Collaborative Environment is based on four main pillars: (i) the availability of (an extendible set of) digital schemas representing models of co-production processes and illustrating the phases/objectives/tasks that stakeholders are suggested to go through for the collaborative design and delivery of public services; (ii) the recommendation of knowledge and software co-production enablers (named INTERLINKERs), which are reusable resources that may help bootstrap stakeholders' activities and lower the costs of public service implementation; (iii) the availability, for each co-production task, of a work space with the possibility to assign teams, create documents by instantiating INTERLINKERs or by using general purpose collaboration tools (like Google Drive and Loomio) and links to  external resources; and iv) the availability of a catalogue of INTERLINKERs, success stories, and open processes promoting reuse, replicability and adoption. The core innovation that is brought about with respect to existing general-purpose platforms for computer supported cooperative work is the operationalization of co-production processes.
An evaluation study conducted by three different PAs with the involvement of 224 stakeholders tested the usage of the platform during the execution of three co-production processes. The findings underlined the overall appreciation of the offered functionalities, with observed forms of appropriation of the tool. Participants clearly stated that the main added value that the Collaborative Environment can offer is represented by the step-by-step guidance and potential reuse (with adaptation) of ready-to-use resources. They also stressed the importance of initial training and sense making to help users familiarize themselves with the platform and reduce the cultural and technical barriers of adopting a new tool.
The evaluation study performed also sheds light on a few challenges that clearly point to future work. Firstly, there is the need to harmonize activities that are performed within the digital platform with the (possibly large) number of activities that are held physically (e.g., meetings, public gatherings), so as to add the possibility of accounting the latter within the system too. Future extensions of the platform will extend the internal data models for a clear representation of the hybrid nature of co-production processes, with functions for administrators and users to keep track of off-line activities. Secondly, co-production processes may span over months, sometimes even years. The engagement of heterogeneous networks of stakeholders for long periods of time might require incentives, both material and immaterial, in order to foster participation and active contribution. In a second round of user evaluation, we are currently testing the integration of a gamification engine that is able to track user actions inside the platform to register contributions and value them through an incentives-and-rewards system that has already proved successful in engaging citizens in sustainability actions [40, 74]. A third aspect that deserves attention is the potential initial need of PA representatives to gain a deeper understanding of whether co-production is the appropriate approach to follow, possibly benefiting and learning from the experience of other similar PAs. On this regard, in order to support the voluntary policy transfer [22], the second version of the Collaborative Environment features a catalogue of "co-production stories" for PAs to draw inspiration from, with clone facilities that help start from specifically customized co-production schemas already tested successfully by other PAs [74].

Acknowledgments

We are very grateful to all INTERLINK partners for the inspiring and intense collaboration that provided the background to this research work. Special thanks to the team of developers, particularly Julen Badiola (formerly affiliated with the University of Deusto) who set the basis of the whole Collaborative Environment tool, and other developers from the University of Deusto (Spain), Tree Technology (Spain), Fondazione Bruno Kessler (Italy), and Cloud'N'Sci (Finland) who have contributed to the implementation of the digital platform.

Footnotes

1
Single Sign-On guarantees the possibility to access several electronic services offered by the public sector with the same identification credentials, without re-entering authentication factors when more services are open.
4
In the Kanban approach, work items are represented visually on a board, allowing team members to see the state of every piece of work at any time [43].

References

[1]
John Alford. 2014. The multiple facets of co-production: Building on the work of Elinor Ostrom. Public Management Review 16 3 (2014), 299–316, DOI:
[2]
Baseer Ahmad Baheer, David Lamas, and Sónia Sousa. 2020. A systematic literature review on existing digital government architectures: State-of-the-art, challenges, and prospects. In Administrative Sciences 10, 2 (2020), 25
[3]
Giovanni Bajo, Viviana De Paola, Daniela Intravaia, Guido Pera, Umberto Rosini, and Guido Scorza. 2020. Guidelines on the acquisition and reuse of software for public administrations. Italia. Agency for Digital Italy and the Digital Transformation Team. https://docs.italia.it/italia/developers-italia/gl-acquisition-and-reuse-software-for-pa-docs/en/stabile/index.html (accessed 21st February 2023).
[4]
Miri Barak. 2018. Are digital natives open to change? Examining flexible thinking and resistance to change. Computers & Education 121 (2018), 115–123.
[5]
Gordon Baxter and Ian Sommerville. 2010. Socio-technical systems: From design methods to systems engineering. In Interacting with Computers 23, 1 (2010), 4–17
[6]
Taco Brandsen, Noortje Hoevens, Marlies Honingh, Ina Radtke, and Sawssane Bargach. 2023. Advanced governance model. Deliverable D2.2. Interlink project, Horizon 2020 - DT-GOVERNANCE-05-2020 - Grant Agreement No 959201
[7]
Taco Brandsen, Noortje Hoevens, Marlies Honingh, Ina Radtke, Chiara Leonardi, Elena Not, Pauli Misikangas, Diego López de Ipiña, Ana Ortiz de Guinea Lopez de Arana. 2022. Preliminary governance model. Deliverable D2.1. Interlink project, Horizon 2020 - DT-GOVERNANCE-05-2020 - Grant Agreement No 959201. https://zenodo.org/records/10668560
[8]
Taco Brandsen and Marlies Honingh. 2015. Distinguishing different types of coproduction: A conceptual analysis based on the classical definitions. In Public Administration Review 76, 3 (2015), 427–435. © 2015 by The American Society for Public Administration. DOI:
[9]
Taco Brandsen and Marlies Honingh. 2018. Definitions of co-production and co-creation. In: T. Brandsen, B. Verschuere, and T. Steen, (2018). Co-production and co-creation: Engaging citizens in public service delivery, 9–17. New York, London: Routledge.
[10]
Susanne Bødker. 2000. Scenarios in user-centred design --setting the stage for reflection and action. Interacting with Computers 13, 1 (2000), 61–77.
[11]
Tony Bovaird and Elke Loeffler. 2012. From engagement to co-production: The contribution of users and communities to outcomes and public value. Voluntas 23 (2012), 1119–1138.
[12]
David Boyle and Michael Harris. 2009. The challenge of co-production: How equal partnerships between professionals and the public are crucial to improving public service. NESTA, New Economics Foundation, The Lab, London.
[13]
Denis Butin, Marcos Chicote, and Daniel Le Métayer. 2013. Log design for accountability. DUMA13 - 4th International Workshop on Data Usage Management, May 2013, San Francisco, CA, USA.
[14]
John M. Carroll. 2000. Making use: Scenario-based Design of Human-Computer Interactions. MIT Press.
[15]
Ann Cavoukian. 2009. Privacy by design: The 7 foundational principles. Information and Privacy Commissioner of Ontario, Canada 5, (2009), 12.
[16]
Judith Clifton, Daniel Díaz Fuentes, Gonzalo Llamosas García. 2020. ICT-enabled co-production of public services: Barriers and enablers. A Systematic Review, Information Polity 25, 1 (2020), 25–48.
[17]
Alan Cooper, Robert Reimann, and David Cronin. 2007. About Face 3: The essentials of interaction design. Wiley.
[18]
Agostino Cortesi, Carlotta Berionni, Carini Veeckman, Chiara Leonardi, Gianluca Schiavo, Massimo Zancanaro, Marzia Cescon, Maria Sangiuliano, Dimitris Tampakis, and Manolis Falelakis. 2022. Families_share: Digital and social innovation for work and life balance. In Digital Policy, Regulation and Governance (2022).
[19]
Sharon S. Dawes and Theresa A. Pardo. 2002. Building collaborative digital government systems. In Advances in Digital Government. Advances in Database Systems W. J. McIver, A. K. Elmagarmid, (eds) 26. Springer, Boston, MA.
[20]
Eugenio De Rosa. 2017. Social innovation and ICT in social services: European experiences compared. Innovation: The European Journal of Social Science Research 30, 4 (2017), 421–432, DOI:
[21]
Alan Dix. 2007. Designing for appropriation. In Proceedings of HCI 2007 The 21st British HCI Group Annual Conference University of Lancaster, UK 21 (1–4).
[22]
David Dolowitz and David Marsh. 1996. Who learns what from whom: A review of the policy transfer literature. Political Studies 44, 2 (1996), 343–357.
[23]
Noella Edelmann and Ines Mergel. 2021. Co-production of digital public services in Austrian public administrations. Administrative Sciences 11 (2021), 22.
[24]
Noella Edelmann and Shefali Virkar. 2023. The impact of sustainability on co-creation of digital public services. Administrative Sciences 13, 2 (2023), 43.
[25]
European Commission. 2016. EU eGovernment action plan 2016-2020: Accelerating the digital transformation of government. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52016DC0179 (accessed on the 21st of February 2023)
[26]
European Commission. 2017. European interoperability framework – implementation strategy. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=COM:2017:134:FIN (accessed on the 21st of February 2023).
[27]
European Commission. 2018. Directorate-general for employment, social affairs and inclusion. Co-production: Enhancing the Role of Citizens in Governance and Service Delivery. Technical Dossier no. 4, May 2018, Publications Office, (accessed on the 12th of July 2023).
[28]
European Commission. 2023. About the Digital Europe Programme. https://ec.europa.eu/digital-building-blocks/wikis/display/DIGITAL/About+us (accessed on the 3rd November 2023)
[29]
European Union. 2014. Regulation (EU) No 910/2014 of the European Parliament and of the Council of 23 July 2014 on electronic identification and trust services for electronic transactions in the internal market and repealing Directive 1999/93/EC, https://eur-lex.europa.eu/eli/reg/2014/910/oj (accessed on the 12th of July 2023).
[30]
European Union. 2016. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation). https://eur-lex.europa.eu/eli/reg/2016/679/2016-05-04 (accessed on the 3rd of November 2023).
[31]
Mariagrazia Fugini and Mahsa Teimourikia. 2016. The role of ICT in co-production of e-government public services. In Fugini et al. (2016) Co-production in the public sector. Springer. DOI:.
[32]
Zhiyuan Fang. 2002. E-government in digital era: Concept, practice, and development. International Journal of the Computer, the Internet and Management 10, 2 (2002), 1–22.
[33]
Mila Gascó and Carola van Eijk. 2018. Case study—The Spanish project pla BUITS. Co-Production and Co-Creation. Routledge. 77–79.
[34]
Eleonora Gheduzzi, Cristina Masella, Niccolò Morelli, and Guendalina Graffigna. 2021. How to prevent and avoid barriers in co-production with family carers living in rural and remote area: An Italian case study. Research Involvement and Engagement 7, 1 (2021), 1–13.
[35]
J. Ramon Gil-Garcia and Djoko Sigit Sayogo. 2016. Government inter-organizational information sharing initiatives: Understanding the main determinants of success. Government Information Quarterly 33, 3 (2016), 572–582.
[36]
Jonathan Grudin. 1994. Computer-supported cooperative work: History and focus. Computer 27, 5 (1994), 19–26, DOI:
[37]
Marlis Honingh, Elena Bondarouk, and Taco Brandsen. 2018. Parents as co-producers in primary education. In (2018) Co-production and Co-creation: Engaging Citizens in Public Services. Routledge.
[38]
Michael Howlett and Michael Ramesh. 2017. The Achilles heels of collaboration: Overcoming critical capacity deficits in collaborative governance arrangements? Paper presented at the NUS-FPZG UNESCO Chair Workshop on The Governance of Collaboration: Co-Production, Contracting, Commissioning and Certification, Dubrovnik, Croatia.
[39]
Tomasz Janowski. 2015. Digital government evolution: From transformation to contextualization. Government Information Quarterly 32, 3 (2015), 221–236.
[40]
Raman Kazhamiakin, Annapaola Marconi, Mirko Perillo, Marco Pistore, Giuseppe Valetto, Luca Piras, Francesco Avesani, and Nicola Perri. 2015. Using gamification to incentivize sustainable urban mobility. 2015 IEEE First International Smart Cities Conference (ISC2), Guadalajara, Mexico, 1–6, DOI:
[41]
Zaheer Khan, Jens Dambruch, Jan Peters-Anders, Andreas Sackl, Anton Strasser, Peter Fröhlich, Simon Templer, and Kamran Soomro. 2017. Developing knowledge-based citizen participation platform to support smart city decision making: The smarticipate case study. Information. 8, 2 (2017), 47. 2078–2489.
[42]
Hee-Cheol Kim. 2015. Acceptability engineering: The study of user acceptance of innovative technologies. In Journal of Applied Research and Technology, (2015).
[43]
Henrik Kniberg and Mattias Skarin. 2010. Kanban and scrum-making the most of both. InfoQ.
[44]
International Standards Organisation. 2019. ISO 9241-210:2019(en) Ergonomics of human-system interaction — Part 210: Human-centred design for interactive systems. ISO, Geneva, Switzerland. https://www.iso.org/obp/ui/en/#iso:std:iso:9241:-210:ed-2:v1:en
[45]
Tazim B. Jamal and Donald Getz. 1995. Collaboration theory and community tourism planning. Annals of Tourism Research 22, 1 (1995), 186–204.
[46]
Elke Loeffler. 2018. Providing public safety and public order through co-production. Co-Production and Co-Creation. Routledge. 211–222.
[47]
Veiko Lember, Taco Brandsen, and Piret Tõnurist. 2019. The potential impacts of digital technologies on co-production and co-creation. Public Management Review 21, 11 (2019), 1665–1686.
[48]
Chiara Leonardi and Elena Not. 2022. Challenges and opportunities for ICT in co-production: A case study of public service innovation in an Italian municipality. In DG.O 2022: The 23rd Annual International Conference on Digital Government Research (DG.O 2022). Association for Computing Machinery, New York, NY, USA, 322–327.
[49]
Chiara Leonardi, Elena Not, Matteo Gerosa, and Roberta Lotti. 2023. A case study of cross-organizational co-design with public bodies: Opportunities for a collaborative platform. In The 11th International Conference on Communities and Technologies (C&T ’23), May 29–June 02, 2023, Lahti, Finland. ACM, New York, NY, USA, 6 pages.
[50]
Dennis Linders. 2011. We-government: An anatomy of citizen coproduction in the information age. In Proceedings of the 12th Annual International Digital Government Research Conference: Digital Government Innovation in Challenging Times, 167–176.
[51]
Diego López-de-Ipiña, Julen Badiola, Danile Andrés Silva, Diego Casado-Mansilla, Elena Not, Chiara Leonardi, Ana Ortiz-de-Guinea, Igone Porto. 2022. A collaborative environment to boost co-production of sustainable public services. In Proceedings of the 7th International Conference on Smart and Sustainable Technologies 2022 (https://2022.splitech.org)
[52]
Diego López de Ipiña, Roberto Carballedo, Igone Porto, Daniel Silva, Julen Badiola, Elena Not, Chiara Leonardi, Angelo Mazzoli, Michele Cappuccio, Matīss Veigurs, Ana Jiménez, Raquel Povar, Noortje Hoevens, Serge Sushkov, and Pauli Misikangas. 2022. Use-case deployment and operation report v1. INTERLINK project. Deliverable D5.3. Interlink project, Horizon 2020 - DT-GOVERNANCE-05-2020 - Grant Agreement No 959201
[53]
Albert Jacob Meijer. 2011. Networked coproduction of public services in virtual communities: From a government-centric to a community approach to public service support. Public Administration Review 71, 4 (2011), 598–607.
[54]
Albert Meijer and Wouter Boon. 2021. Digital platforms for the co-creation of public value. Policy & Politics 49, 2 (2021), 231–248.
[55]
Tina Nabatchi, A. Sancino, and Mariafrancesca Sicilia. 2017. Varieties of participation in public services: The who, when, and what of coproduction. Public Admin. Rev. 77 (2017), 766–776.
[57]
Oliver Neumann and Carina Schott. 2023. Behavioral effects of public service motivation among citizens: Testing the case of digital co-production. International Public Management Journal 26, 2 (2023), 175–198, DOI:
[58]
Jakob Nielsen. 1994. Enhancing the explanatory power of usability heuristics. Proc. ACM CHI'94 Conf. (Boston, MA, April 24-28), 152–158.
[59]
Jakob Nielsen and Rolf Molich. 1990. Heuristic evaluation of user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 249–256.
[60]
Elinor Ostrom. 1999. Crossing the great divide: Coproduction, synergy, and development, Ch. 15 in M.D. McGinnis, ed., Polycentric Governance and Development. Readings from the Workshop in Political Theory and Policy Analysis, Univ. of Michigan Press, Ann Arbor, MI.
[61]
Andrea Paletti. 2016. Co-production through ICT in the public sector: When citizens reframe the production of public services. In Digitally Supported Innovation. Springer International Publishing L. Caporello, F. Cesaroni, R. Giesecke, Missikoff, M. (eds.).
[62]
Georgios Patsoulis, Rafail Promikyridis, and Efthimios Tambouris. 2021. Integration of chatbots with knowledge graphs in egovernment: The case of getting a passport. In Proceedings of the 25th Pan-Hellenic Conference on Informatics, 425–429.
[63]
Ken Peffers, Tuure Tuunanen, Marcus A. Rothenberger, and Samir Chatterjee. 2007. A design science research methodology for information systems research. Journal of Management Information Systems 24, 3 (2007), 45–77.
[64]
Antti Pirinen. 2016. The barriers and enablers of co-design for services. International Journal of Design 10, 3 (2016), 27–42.
[65]
Antti Pirinen, Kaisa Savolainen, Sampsa Hyysalo, and Tuuli Mattelmäki. 2022. Design enters the city: Requisites and points of friction in deepening public sector design. International Journal of Design 16, 3 (2022), 1–19.
[66]
Ina Radtke, Noortje Hoevens, Taco Brandsen, and Marlies Honingh. 2023. Assessing the quality of digital coproduction: An interdisciplinary model. Administrative Sciences 13, 3 (2023), 69.
[67]
Suzanne Robertson and James Robertson. 2012. Mastering the requirements process: Getting requirements right. Addison-Wesley.
[68]
Mary Beth Rosson and John M. Carroll. 2002. Usability engineering: Scenario-based development of human-computer interaction. Morgan Kaufmann.
[69]
Joni Salminen, Soon-gyo Jung, Shammur Chowdhury, Sercan Sengün, and Bernard J. Jansen. 2020. Personas and analytics: A comparative user study of efficiency and effectiveness for a user identification task. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–13.
[70]
Gianluca Schiavo, Chiara Leonardi, and Massimo Zancanaro. 2020. Values and practices behind collaborative childcare in knowledge-based organizations. Technology Innovation Management Review 10, 5 (2020), 41–50.
[71]
Ada Scupola and Ines Merge. 2022. Co-production in digital transformation of public administration and public value creation: The case of Denmark. In Government Information Quarterly. 39, 1 (2022).
[72]
Fabio Sgaragli (Ed.). 2014. Enabling social innovation: Ecosystems for community-led territorial development. Quaderni della Fondazione Giacomo Brodolini.
[73]
Mariafrancesca Sicilia, Enrico Guarini, Alessandro Sancino, Martino Andreani, and Renato Ruffini. 2016. Public services management and co-production in multi-level governance settings. International Review of Administrative Sciences 82, 1 (2016), 8–27.
[74]
Daniel Silva, Rubén Sánchez, Diego López-de-Ipiña, Elena Not, Chiara Leonardi, Roberto Carballedo, Matteo Gerosa, and Felipe Vergara. 2023. Tackling co-delivery in co-production processes. In Proceedings of Splitech 2023, International Conference on Smart and Sustainable Technologies, Split - Bol, June 20-23, 2023.
[75]
Stian Soiland-Reyes, Peter Sefton, Leyla Jael Castro, Frederik Coppens, Daniel Garijo, Simone Leo, Marc Portier, and Paul Groth. 2022. Creating lightweight FAIR digital objects with RO-Crate. 1st International Conference on FAIR Digital Objects (FDO 2022) Research Ideas and Outcomes 8, e93937.
[76]
Maddalena Sorrentino, Mariafrancesca Sicilia, and Michael Howlett. 2018. Understanding co-production as a new public governance tool. Policy and Society 37, 3 (2018), 277–293, DOI:
[77]
Kaisa Spilling, Maija Bergström, and Kubo Creative Agency. 2021. Life and innovations. Smart Kalasatama – A district for experimentation. https://forumvirium.fi/en/publication/smartkalasatama-final-report/(accessed 16th November 2023)
[78]
Trui Steen, Taco Brandsen, and Bram Verschuere. 2018. The dark side of co-creation and co-production: Seven evils. In Co-production and Co-creation. Routledge (284–293).
[79]
Benjamin Stelzle, Anja Jannack, and Jörg Rainer Noennig. 2017. Co-design and co-decision: Decision making on collaborative design platforms. Procedia Computer Science 112 (2017), 2435–2444.
[80]
Ann C. Svendsen and Myriam Laberge. 2005. Convening stakeholder networks: A new way of thinking, being and engaging. The Journal of Corporate Citizenship 19 (2005), 91–104.
[81]
Pieter Raymaekers and Steven Van de Walle. 2019. Pleio: An open source collaboration platform for public administrations and public policy in The Netherlands. https://tropico-project.eu/cases/administration-costs-for-bureaucracy/pleio-an-open-source-collaboration-platform-for-public-administrations-and-public-policy-in-the-netherlands/(accessed 15th November 2023)
[82]
Jacob Torfing, Eva Sørensen, and Asbjørn Røiseland. 2019. Transforming the public sector into an arena for co-creation: Barriers, drivers, benefits, and ways forward. Administration & Society 51, 5 (2019), 795–825.
[83]
Jakob Trischler, Timo Dietrich, and Sharyn Rundle-Thiele. 2019. Co-Design: From expert- to user-driven ideas in public service design. Public Management Review 21, 11 (2019), 1595–1619, DOI:
[84]
Maaike van den Haak, Menno De Jong, and Peter Jan Schellens. 2003. Retrospective vs. concurrent think-aloud protocols: Testing the usability of an online library catalogue. Behaviour & Information Technology 22, 5 (2003), 339–351.
[85]
Sue Wilkinson. 1998. Focus group methodology: A review. International Journal of Social Research Methodology 1, 3 (1998), 181–203, DOI:
[86]
Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz Bonino da Silva Santos, Philip E. Bourne, Jildau Bouwman, Anthony J. Brookes, Tim Clark, Mercè Crosas, Ingrid Dillo, Olivier Dumon, Scott Edmunds, Chris T. Evelo, Richard Finkers, Alejandra Gonzalez-Beltran, Alasdair J. G. Gray, Paul Groth, Carole Goble, Jeffrey S. Grethe, Jaap Heringa, Peter A.C. 't Hoen, Rob Hooft, Tobias Kuhn, Ruben Kok, Joost Kok, Scott J. Lusher, Maryann E. Martone, Albert Mons, Abel L. Packer, Bengt Persson, Philippe Rocca-Serra, Marco Roos, Rene van Schaik, Susanna-Assunta Sansone, Erik Schultes, Thierry Sengstag, Ted Slater, George Strawn, Morris A. Swertz, Mark Thompson, Johan van der Lei, Erik van Mulligen, Jan Velterop, Andra Waagmeester, Peter Wittenburg, Katherine Wolstencroft, Jun Zhao, and Barend Mons. 2016. The FAIR guiding principles for scientific data management and stewardship. Scientific Data 3, 1 (2016), 1–9. https://www.nature.com/articles/sdata201618.pdf
[87]
Qianli Yuan. 2019. Co-production of public service and information technology: A literature review. In Proc. of 20th Annual International Conference on Digital Government Research.
[88]
Didar Zowghi and Chad Coulin. 2005. Requirements elicitation: A survey of techniques, approaches, and tools. Engineering and Managing Software Requirements, 19–46.

Cited By

View all
  • (2024)Introduction to the Special Issue on Smart Government Development and ApplicationsDigital Government: Research and Practice10.1145/36913535:3(1-9)Online publication date: 13-Sep-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Digital Government: Research and Practice
Digital Government: Research and Practice  Volume 5, Issue 3
September 2024
392 pages
EISSN:2639-0175
DOI:10.1145/3613695
Issue’s Table of Contents
This work is licensed under a Creative Commons Attribution-NoDerivs International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 September 2024
Online AM: 13 May 2024
Accepted: 25 April 2024
Revised: 05 April 2024
Received: 19 July 2023
Published in DGOV Volume 5, Issue 3

Check for updates

Author Tags

  1. ICT-enabled co-production
  2. digital collaboration environment
  3. digital platforms design
  4. co-design with public administrations
  5. public services

Qualifiers

  • Research-article

Funding Sources

  • European Commission's Horizon 2020 Programme, within the call H2020-SC6-GOVERNANCE “Governance for the Future”

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)229
  • Downloads (Last 6 weeks)149
Reflects downloads up to 04 Oct 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Introduction to the Special Issue on Smart Government Development and ApplicationsDigital Government: Research and Practice10.1145/36913535:3(1-9)Online publication date: 13-Sep-2024

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Get Access

Login options

Full Access

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media