Assessing Mapper Conflict in OpenStreetMap Using the Delphi Survey Method

The OSM community brings together a few millions of users that may assume a range of non-exclusive roles. These range (in increasing level of involvement) from potentially unknowing consumers of OSM data (including via, e.g., Google Maps), through registered users with a closer interest in the platform, through mappers who contribute new data and maintain existing data, developers who create mapping and data management tools, administrators who organize community guidelines and processes. Mappers (and above levels of user participation) are the target of this study.

Mappers can be occasional volunteer individuals who map on a needs-basis during disasters [11, 49], participants in mapathons (community-driven mapping parties) [38], or regular mappers focusing on specific regions or map content (e.g., railroads, parks, cycling infrastructure). But mappers can also be employees of public or private organizations who are paid to map [2]. Mappers from countries with higher GDP and better access to technology [50] dominate in the OSM community, as do male mappers [10].

Similar to other OPPCs, the user base of OSM is diverse and dynamic. This diversity contributed to the fast growth of OSM, but also increased the number of conflicts between mappers: from differences about semantic definitions of map features [1, 5, 37, 59] to (cartographic) vandalism [4, 44, 61, 62], and from data standards [20, 21] to governance, including collaboration processes between mappers [49] and strategies to increase mapper retention [38]. This spectrum of mapper conflict shows that OSM has now evolved from the original, relatively homogeneous group of hobbyists to an OPPC requiring more formal organizational structures. These are now established within and governed by the OSM Foundation (OSMF) and its structures, such as working groups and recognized Local Chapters by countries. Notably, the OSM Data Working Group (DWG) is tasked with the management of extreme mapper conflicts impacting on data quality.³ At this stage, OSM may have reached a point where conflict management needs to be supported more formally, and beyond mere data quality management, too [8].

Conflict is loosely defined as disagreement among group members and/or among entire groups during the problem solving process [52]. The effects of conflict have been a frequent topic of research in sociology and organizational studies, in particular focused on people working in groups, on their work processes and on the quality of their work [25, 26, 31, 34, 39]. The organizational management literature suggests that conflict management aims to adjust the amount of conflict and select the most suitable management strategy [30, 52]. The underlying assumption here is that conflict can have a positive effect on group members and lead to the development of a solution. In this sense, the conflict management differs from the conflict resolution, with the latter aiming at minimizing or eliminating the conflict [52].

Previous studies on producer conflict in OPPCs have applied conflict management theories from the organizational management literature. Producer conflict in OPPCs can be broadly defined as a disagreement that occurs between more than two producers in an online community, while the terms associated with its manifestations vary (e.g., dispute, controversy, revert, vandalism) [7, 29, 62, 63]. Several studies applied the taxonomy of type of conflict to understand the topic in which user conflict arises (i.e., task, process, relationship, affective) [3, 54] and even expanded the model to fit the context of OPPCs [12, 13, 43]. Other studies based their arguments on the model of conflict handling styles to assess the strategies that OPPC users employ to manage conflicts with each other [8, 28, 60].

Here, we take a step further and apply a research framework of conflict management process to study producer conflict in OPPCs, using OSM as a case study. We argue that using such a framework may support OPPCs in incorporating a systematic conflict management process into the operational mechanisms of the community, providing long-term benefits [8]. The conflict management process framework consists of (1) conflict diagnosis, (2) intervention design, (3) intervention implementation, and (4) intervention evaluation [52]. These steps support groups to suggest short-term solutions to an immediate conflict. They also support groups in assessing the factors that lead to repeated conflicts and systematically addressing them through conflict management tools such as a protocol, a standard operating procedure, or a communication channel [52].

Conflict diagnosis, as the first step in a conflict management process, offers a lens through which groups can approach an observed conflict. It is prompted by four principal wh-questions [51, 52]:

We explore the interpersonal mapper conflict in OSM based on these four questions via a Delphi survey. Our intention is to diagnose mapper conflict from the results of these questionnaires and to suggest design implications for the second step of the conflict management framework – intervention design. In the next section, we describe our research methodology, including the structure of the survey questionnaire and the process in detail.

We created a Delphi survey to collect perceptions of conflict in the OSM community from its mappers. The Delphi method was developed to obtain reliable consensus opinions from a group of experts via a series of questionnaires in the 1950s [16, 23, 27]. Participants in a Delphi survey are “individuals who have knowledge of the topic being investigated.” [23, p.1010]. They are meant to be experts based on their experience with the topic [16, 23]. The Delphi method has been used in multiple disciplines, from military [9], through professional development [15] to construction engineering and management [22].

The Delphi method is routinely applied to assess the current situation of a group, make a forecast, and/or set a goal for a desirable future state [16, 23, 27]. Therefore, the Delphi method presents a suitable approach to gather experiences and opinions from OPPC members, including OSM mappers, on their experiences on conflict and how these can be improved.

The advantages of the Delphi method over other types of opinion collection methods include multiple, iterative survey rounds, controlled feedback, and participants’ anonymity [16, 23]. Repeated rounds of survey questionnaires allow participants to reach consensus on the topic. As the analyst creates summaries of the results from each round, the participants are informed about how many of the participants share similar opinions. Participants are also encouraged to add, modify, and suggest the removal of answers from repeated rounds. They have also room to provide their feedback on how to structure the next round of the questionnaire. Finally, the responses of the participants are anonymized to avoid excessive influence of the participants whose answers are considered authoritative [16, 23]. We now further describe our survey questionnaire design and the recruitment of participants.

We designed the initial-round survey questionnaire based on the four wh-questions introduced in Section 2, and asked these four open-ended questions:

We itemized the answers for each question in the initial round, preserving the participants’ wording. Therefore, we minimized the influence of our own judgments and editorial interventions, following best practice [23]. Our assumption was that the survey participants, all OSM mappers, will largely share common understandings and vocabulary when describing conflict in the community. The itemized answers and the feedback of the participants became input for the design of the second-round survey questionnaire, which asked the participants to rate their agreement with the items identified in the initial-round survey (five-point Likert scale, 1-strongly disagree; 5-strongly agree). The second-round questionnaire prompted participants to suggest additions, amendments, and/or removals of items for each question.

From the results of the second-round, we identified items that reached a high level of consensus based on the median value of agreement exceeding 3.5 and the level of agreement (i.e., number of participants rating items 4 or greater) being greater than 70 percent [32]. Only items that reached consensus were included in the third (and final) round of the survey. In this third-round questionnaire, participants were asked to rate the level of importance of each selected item, based on their perceived importance for the OSM mapper community (five-point Likert scale, 1-strongly disagree; 5-strongly agree). We again provided participants with the opportunity to explain their reasoning for the ranking of the items and to contribute anecdotes and comments.⁴

The typical target number of participants in Delphi studies varies from 15 to 60 [23, 27]. If the participants come from a relatively homogeneous domain and background, the number of participants can be as low as 5 to 15 [27]. We emphasize the qualitative coverage sought here, as opposed to a statistically representative sample.

Our target participants were OSM mappers who (1) volunteer to take on an administrative role in moderating and/or managing conflict in the community; and/or those who (2) have collaborated with other mappers when mapping in OSM from 2016 onward. This period aligns with the timeline when direct editing of spatial data in OSM by non-voluntary mappers (e.g., corporations, government agencies, NGOs) in OSM increased [2].

The names, descriptions, and mailing lists of various OSM working groups are publicly available through the OSM Wiki Pages. We compiled a list of the most relevant working groups that are encountering and/or managing conflict in OSM based on their descriptions. This includes (alphabetically) the DWG, the Membership Working Group, the Operations Working Group, the OSMF Advisory Group, the OSMF Board, the OSMF License Working Group, and the OSMF Local Chapters Working Group. We first sent blind email invitations to identified mappers through group mailing lists. Following a response from an interested mapper, we explained the purpose and process of the survey, via email or teleconference. We also received additional participant recommendations from already recruited participants.

In total, 15 OSM mappers have been recruited and participated in the initial-round survey in March 2021, 14 of them continued with the second-round and 12 of them took part in the third-round survey questionnaire finalized in December 2021. We considered such participant backgrounds to be sufficiently broad and the domain sufficiently narrow to effectively cover the topic of mapper conflict in OSM within the cohort gathered.⁵ Table 1 summarizes information about participants by age group, self-identified affiliation in OSM, and their geographic area of mapping interest.⁶

The Delphi method facilitates a qualitative interpretation of the collected data. We therefore provide a brief positionality statement, enabling to contextualize our interpretation. All authors are geospatial information science researchers and belong to the wider OSM community as OSM data users and occasional mappers. We have no affiliation with the OSMF or the OSMF Working Groups, nor do we have personal relationships with any of the study participants. We are also not commissioned by any organization to do this survey. This study is part of a more extensive academic research investigating conflict management in OSM, as a contributing factor to data quality.

Conflicts among members of any OPPC reflect the current agenda of the group. As per our analysis, mapper conflict in OSM involves clashing interests among different subgroups and negative online mapper behaviors.

For example, the boundaries of the subgroups in OSM are characterized by previously known factors (i.e., gender, geographic location, level of knowledge/skill), and more recently also by professional affiliations. This reflects the progression of the OSM from a small group of like-minded users [17, 18] to a large-scale OPPC that attracts a wide community of users with different intentions [2]. Similarly, other OPPCs have experienced conflicts between contributors due to similar factors [24, 41, 42, 53] and a similar progression in their growth and development.

Most OPPCs start as a small group of people with a similar level of technical expertise and a shared vision [19, 45, 67]. Many studies demonstrate how the rise of OPPCs experiences an influx of contributors who populate products (e.g., Wikipedia articles, open source software) growing exponentially the OPPC’s reputation and usefulness [19, 33, 58], starting a cycle where the OPPC attracts more like-minded contributors but with more diversity [33, 67]. At a point where OPPCs saturate with contributors, they start witnessing increased conflict [13] and experience the need for mechanisms to control the quality of their products. Consider Wikipedia, which required the implementation of selective registration mechanisms and tools to reject entries deemed low in quality [19]. Community guidelines were implemented to set behavior expectations [19, 40]. What was not foreseen was how such quality control mechanisms could be perceived as a burden not only for newcomers but also for long-term producers, leading to reduced participation and retention [19, 58]. If ignored, the decline of an OPPC may start.

In this paper, we argue that applying conflict management theory can support stronger producer retention in OPPCs. From a governance perspective, the discussed producer (here, mapper) retention challenges faced by OPPCs are closely related to tensions between (1) the OPPC’s vision as an online community that is open to everybody and (2) the operational need to control product quality to attract more users and producers. [33, 45, 67]. Our survey design was based on the first step in a conflict management process, conflict diagnosis, and provided results that capture the snapshot of the current producer conflict in OSM (i.e., amongst OSM mappers). Numerous aspects of OSM conflict are shared with other OPPCs (e.g., geographic dispersion, diversity of contributors, introduction of code of conduct) [13, 28], while others are specific (e.g., conflicts that are specific to map editing and the local nature of mapping). Furthermore, we collected the opinions of OSM mappers on useful conflict management tools, thus linking to the next step of conflict management theory, intervention design. Our study thus indicates the potential of applying conflict management theory to other OPPCs to diagnose producer conflict and to extract customized design implications for conflict management tools.

We highlight the potential of the Delphi method as a data collection method and diagnostic framework in other OPPCs [6]. Delphi has a number of advantages over other consensus-building methods, including regular single-round survey or voting. First, Delphi captures both agreements and disagreements among participants, providing an in-depth understanding of producer conflict. It consists of multiple rounds, which allow participants to reflect on the experiences of others. An online Delphi survey can support the simultaneous participation of geographically dispersed participants, potentially more numerous than in an in-person setting [35]. Wikimedia recently used an iterative approach similar to the Delphi method to assess and manage user conflict [65]. As a user-centered approach, community rules requested through a Delphi method might be perceived as having higher legitimacy when applied during conflict management.

Throughout the survey, we see reoccurring themes among the four questions we asked the participants regarding their experience on interpersonal mapper conflict in OSM: existence of subgroups within the community and negative online mapper behaviors in mapper communication channels. From these themes, we synthesize two design implications that could help OSM mappers to manage mapper conflicts: procedural clarity and mapper interaction behavior.

Design implication on procedural clarity is focused on minimizing the ambiguity on how contributors could contribute to OPPCs in which steps. In OSM’s case, this is to address mapper conflicts among different subgroups that we found throughout the survey, as well as newcomers to contribute without fear of making mistakes and getting into unwanted conflicts with other mappers. Design implication on interactive behavior aims to encourage people to follow a code of conduct in OPPCs. In OSM, it will aim to address mapper conflicts exacerbated by negative online mapper behaviors that we found from Sections 4.2 to 4.4. We discuss examples of relevant website interface designs from other OPPCs and apply them in OSM’s context.

Our survey results show that online communication channels frequently used for conflict management by OSM mappers are fragmented into (1) public internal channels (e.g, OSM changeset comments, OSM mailing list, mapper diary/blog, and the OSM discussion forum); (2) private internal channels (e.g., direct message to a mapper via OSM mapper profile); and (3) public and private external channels (e.g., Twitter, Slack, personal email, and messenger apps). OSM mappers typically use multiple communication channels depending on the discussion topic, audience, and preference. Our design suggestions are restricted to public internal channels, as the remaining channels are impossible to monitor for compliance with the code of conduct, or because of the impossibility to customize external communication tools for OSM mapper communication. We use the OSM changeset discussion (Figure 5) as an example target for the implementation of our findings, as this channel was implemented in 2014 for the single-edit mapper conflict management [66].