Incentive Mechanism Design for Responsible Data Governance: A Large-scale Field Experiment

Technologies such as big data, data mining, and artificial intelligence are promising forces for improving data-driven decision-making. But there are also several societal concerns. For example, collecting user data through surveillance, without fair compensation and autonomy, has been a common practice in the tech industry [Zuboff 2019]. Similarly, the issue of data quality and its negative impacts often do not receive sufficient attention [Sambasivan et al. 2021]. However, with several recently proposed and enacted regulations in the European Union (EU), such as the General Data Protection Regulation (GDPR), the AI Act, the Data Governance Act (DGA), and the Digital Services Act (DSA), these practices may be about to change. The regulations stress social responsibility, trustworthiness, consumer protection, and users’ privacy, data autonomy, and informational self-determination. Data minimization, fairness, and shareability become features of future-proof good practices [European Commission 2022]. Since these legal and ethical norms cannot be incorporated into data-driven systems in hindsight, recent research suggests viewing it as a system requirement calling for responsibility by design [Abiteboul and Stoyanovich 2019]. Motivated by this, we address the problem of designing for trustworthy data collection in this article.

Specifically, we study incentive mechanism design for sharing high-quality personally measured fitness data. Instead of automatically collecting data from a tracked device, we preserve the users’ autonomy to measure and report data voluntarily. While some data providers may be intrinsically motivated to measure and report correct data, others may not be willing to do so without incentives. This may lead to low-quality data that may be difficult or impossible to improve at a later stage [Mohan and Pearl 2021]. Incentives can be manipulated by strategic misreporting, further degrading data quality. Explainable incentive-compatible mechanisms with game-theoretic guarantees [Faltings et al. 2017] are a promising way to address these problems. Using fixed incentives and an incentive-compatible peer-consistency mechanism designed to elicit subjective and unverifiable data by incentivizing truthful data reporting [Goel and Faltings 2019a], we conducted an explorative field experiment. The research questions that we aimed to explore in this experiment are (1) what is the effect of varying incentives on the quality of reported data and (2) whether the way we implemented the incentive-compatible design leads to a good user experience.

The experimental design and analysis focus on measuring and assessing data quality [Heinrich et al. 2018; Madnick et al. 2009]. For the analysis, we used a simple quality difference measure, which enabled the comparison of the fixed incentives and the incentive-compatible mechanism groups. The quality difference measure required a third group as a reference. This third group delivered proxy ground-truth observations. We find in the specific study context that the quality of the data between the two incentive groups does not differ, while data quality improves if extreme outliers are excluded from both the groups for analysis. Further, we find that the incentive-compatible mechanism provides a good user experience and compensates fairly. Based on our insights, we discuss specific directions that future studies may focus on, such as design improvements when applying the incentive-compatible mechanism.

Data collection is one of the earlier but important stages of data governance. If an overall responsible-by-design data governance is implemented, individuals are likely to act based on the applied incentives and provide trustworthy data. Our research contributes to real-world applications of data-based technologies while informing future research and complementing regulatory requirements.

The design of robust incentive-compatible mechanisms is a central theme in economics and computation. The mechanisms of interest in the context of this article are the mechanisms for information elicitation without verification. The pioneering work in this field is due to Prelec [2004] on the Bayesian Truth Serum and Miller et al. [2005] on the peer prediction method. A lot of progress [Liu and Chen 2017; Radanovic et al. 2016; Shnayder et al. 2016] has since been made to make the mechanisms suitable for practical use in a variety of scenarios like opinion feedback elicitation (e.g., product reviews on e-commerce websites), participatory sensing (e.g., pollution measurements), human computation (e.g., microwork), and so forth. Faltings and Radanovic [2017] provide a comprehensive overview. Much of the work in this field is devoted to theoretical analysis and guarantees, but there have also been a few empirical studies in this area [Faltings et al. 2014; Gao et al. 2014]. These mechanisms work for discrete signals about phenomena that can be observed by multiple agents.

One exception is the Personalized Peer Truth Serum (PPTS) of Goel and Faltings [2019a]. PPTS is a game-theoretic incentive mechanism to elicit multi-attribute personal measurements from rational agents. Personal measurements may be about a phenomenon that can be observed only personally. To the best of our knowledge, this is the only incentive mechanism in the literature that meets the requirements of our experiment, in which we ask the participants to share their physical activity measurements. The mechanism is based on the logarithmic scoring rule [Gneiting and Raftery 2007] and is a peer-consistency mechanism [Faltings and Radanovic 2017]. Peer-consistency mechanisms are incentive mechanisms to elicit correct information when there is no way to determine the correctness of the information, which is the case with subjective information related to physical activity.

The idea behind these mechanisms uses the fact that the information is often provided by multiple agents, the peers. A naive mechanism is the output agreement mechanism [Waggoner and Chen 2014]. In the output agreement, two agents may be asked to review the same product, and if they both provide the same review, they both get $1. Otherwise, they both get $0. It is obvious that this naive mechanism works only under very strong assumptions. If an agent believes that the other agent is most likely to have the same opinion, then a truth-telling equilibrium prevails. This basic idea has been improved significantly in the literature and there are now several mechanisms that make truth-telling an equilibrium even under weaker belief assumptions. In case of personal measurements such as physical activity data, the peer relationship between the agents is not clear because every agent measures and shares data about its own body (unlike products on an e-commerce website that can be used and experienced by many agents).

PPTS defines the peer relationship by clustering agents based on similarity in correlated attributes. When agents share data about multiple correlated attributes (say, X₁, X₂, X₃), the agents can be clustered based on similarity in the shared data in other attributes (say, X₂, X₃). Now, the rewards of each of the agents for the remaining attribute (X₁) can be calculated. The score of an agent for an attribute is the ratio of the likelihood of the shared value of the attribute in the cluster of the agent and the likelihood of the shared value of the attribute in the overall population. Informally:

The higher this ratio, the higher is the score of the agent. This mechanism is incentive compatible; i.e., truth-telling equilibrium prevails and other (non-truthful) equilibria are not more profitable in this mechanism [Goel and Faltings 2019a]. The score outcomes can then be scaled appropriately (as per budget constraints and fairness requirements) to calculate the rewards in euros for each agent. Depending on the structure of the collected correlated data, clustering may require large samples, which is usually the case in crowdsourcing and big data technologies.

Studies that target the reduction of dishonest behavior [Frank et al. 2017; Hussam et al. 2017; John et al. 2012, Rigol and Roth 2016], e.g., in reporting questionable research practices, show positive effectiveness of incentive mechanism designs based on peer-consistency methods. Despite their great potential, incentive designs are seldom validated nor adapted in the field and existing field studies rarely get replicated. Underlying algorithms are deemed to be too complex, so that, e.g., in John et al. [2012] they are not explained in detail. Other barriers are the large sample size that is required by these big data algorithms and the lack of suitable platforms meeting the recruitment requirements. Conventional crowdsourcing platforms might have established cultures or recruitment difficulties, while online research platforms might have a strong culture to deliver reliable and high-quality data, which would require to design additional incentives to lie [Gneezy et al. 2018] or defaults to nudge certain choices [Baillon et al. 2022]. The present study is unprecedented in its design and was therefore risky to conduct, as the chances were high that it will be a learning-from-failure study. Although we find no significant quality differences between the studied incentive mechanisms, the applied quality-dependent incentives can elicit good data and yield a good user experience. Thus, we provide a first transparent proof of concept as a practical contribution and identify dimensions that need improvement and might cause data quality problems [Madnick et al. 2009]. These dimensions cover topics from sampling through design details to feedback loops and data cleaning.

The recruitment constraints during the data collection caused most limitations to our study. Thus, even though field experiments have a relatively high internal and external validity, their success is limited by the researchers’ connections and recruitment possibilities [Roe and Just 2009]. The conventional crowdsourcing platform that we chose could not deliver, under pandemic conditions, the required large sample. The resulting different sample sizes and room for inaccuracy by design underpower the results. For example, checking for the validity of the screenshots could have been done before calculating the payoffs. Moreover, extreme outliers could have been excluded by design, e.g., by implementing reasonable ranges and asking participants to cross-check their results for accuracy before submission. The problem of the extreme outliers seems to root in participants’ negligence, inattention, and not taking the task seriously, rather than not having understood the task. What speaks for this are the descriptive statistics on explainability, which did not change in their composition after excluding extreme outliers.

Low incentives might be another problem. One way to increase incentives would be to implement the incentive design repeatedly and put individual reputation as a “high-quality data provider” at stake in a realistic setting. Feedback on the reputation could be used as part of individual progress monitoring or in social comparison. Repeated interaction would also increase the chances for learning effects and reduce the costs of initial learning invested to understand the incentive design. Implementation in a real-world setting would address the challenging problem of recruiting a large number of participants too. Moreover, it would allow establishing a new and purposive crowdsourcing culture, aligned with the incentive design and its trustworthy and responsible framing. In contrast, conventional crowdsourcing platforms often already have a disadvantageous culture, such as inattention, self-misrepresentation, high attrition, social desirability bias, and so forth that impacts data quality [Agley et al. 2022; Aguinis et al. 2021; Saravanos et al. 2021].

If not caused by recruitment anomalies on the part of the crowdsourcing platform, the reported difference in the participation rates of 25 percentage points provides some evidence for a self-selection bias in the treatment groups. The self-selection might be driven by the varying complexity of the task. Any cognitive overload due to task complexity could be reduced by repeated interaction in a realistic setting, as suggested. Supporting this suggestion, Weaver and Prelec [2013] found that truthful behavior improved as participants learned the incentive mechanism through repeated interaction, even in the absence of explicit guidance. In another algorithmic context, Biermann et al. [2022] also showed that providing feedback to participants through repeated interaction is more effective than explanation of the underlying mechanism. A future experiment may focus on, for example, the behavior of the participants in both groups as they learn through repeated interaction the incentive compatibility of the peer-consistency mechanism and the absence of incentive compatibility in case of fixed incentives.

It is worth noticing that even though we excluded outliers after and not during data collection, this procedure reduced the impact of unsophisticated cheating and inaccuracy. In order to have enough observations for the peer clustering among a group sample of 500 observations, we initially narrowed down walk time to a 15- to 20-minute range. This way, all participants also got an anchor for applying any heuristic reporting strategy, rather than completing the task. Thus, in real-world applications, it may be a good idea to use not only individual reputation as an incentive but also additional measures (e.g., including limited ground truth [Goel and Faltings 2019b]) to make sophisticated cheating too complex and costly enough to raise the overall trustworthiness of the mechanism design.

In the context of fitness data, Zhou and Zhu [2022] recently showed that presenting calorie-equivalent exercise data is effectively nudging consumers to healthier food choices. Specifically, if food labels contain precise rather than rounded exercise data, the intervention is more effective. “For instance, a chocolate bar with a calorie content of approximately 300 kcal may have a 5 km walk shown in its exercise data or, more precisely, a 4.87 km walk or a 5.13 km walk” [Zhou and Zhu 2022]. Thus, data quality matters in our context for the most various applications, and its requirements (e.g., range, scale, interpretability, feasibility, acceptability, etc.) need to be defined in advance [Heinrich et al. 2018].

Finally, the post-study survey could additionally ask whether participants were aware that they were in the respective uncontrolled group, as well as collect qualitative data via interviews or focus groups. Such feedback loops could be of key importance in repeated interaction settings in order to monitor how participants learn over time. Learning might also lead to abusing the mechanism, and thus further adjustment of the incentive design might be necessary. For monitoring continual and personalized adjustment in repeated and adaptive intervention settings, sequential randomized trials proved to be less limiting and more advantageous than traditional A/B-testing [NeCamp et al. 2019]. Again, for this purpose, real settings can be more adequate than conventional crowdsourcing platforms. Another improvement might be to initially explain the motivation behind using the quality-dependent incentives. Motivations are, for example, the consumer protection requirements by the new EU regulations or simply the aims to increase data quality and trustworthiness in the data collection procedure. Moreover, it might be helpful to explain why the mechanism is complex. For example, PPTS does not require the participants to submit proof of correctness of their data points or execution of any spot-checking or surveillance on the participants to calculate incentives, which is compensated by comparing peers based on correlated data, making cheating complex and unnecessary. On top of that, PPTS can also be used for data cleaning, not only for incentives [Goel 2020]. Data cleaning by design may require telling the participants in advance about its procedures if, for example, this procedure may affect their payoffs or behavior.

Via a large-scale framed field experiment, we demonstrate and discuss the challenges and opportunities that lie in incentive mechanism design for high-quality data sharing or collection, respecting the human-centric European responsible data governance principles. We implemented and explained the incentive mechanism in a transparent and easily comprehensive way, informing about the payoff risks of dishonest behavior and rewards for honest behavior, and leaving participation anonymous, voluntary, and untracked. We observe that incentive design can be effective in eliciting high-quality data in the context of unverifiable and personally measured fitness (physical activity) data. Moreover, we discussed the pitfalls of a challenging large-scale experiment related to design, recruitment, and analysis. In the post-study survey, participants reported they had a good user experience. Most of them felt properly instructed with a relatively high perceived effectiveness and fairness of the incentive mechanism. An ideal future experiment would include a repeated interaction, individual reputation as an additional stake, and a real-world setting by design, in addition to a larger number of participants. Our research thus contributes to improving the design and processes of future experiments and real-world applications utilizing incentive-compatible peer-consistency mechanisms for improving the quality of data.

We thank Gauthier Boeshertz from EPFL for his tireless assistance through many iterations and tests of the oTree code. Especially we thank Marc Kaufmann from the Central European University for his invaluable feedback during the analysis work. We also thank the participants of the ASFEE Conference 2022 in Lyon and colleagues for their valuable comments improving our work.

This section contains additional results.

Data and analysis codes in R are available at https://osf.io/kj95x/?view_only=f096326afa0e4506bc481d5b6b1152d1.

The data for the (Q) group was collected from November 16, 2020, to March 12, 2021; for the (F) group from March 11, 2021, to May 3, 2021; and for the (P) group from April 28, 2021, to June 30, 2021. Note that data was collected sequentially. The reason for this was limited performance met by the recruitment platform. They delivered almost 700 sequentially collected valid observations, despite the originally offered randomized data collection of a total of 1,500 observations. The participants were not qualified to be part of more than one group and groups were gender-balanced. We asked participants whether local COVID-19 measures affected their usual fitness behavior during the study but did not find it to be a confounding factor in our analysis.

In the (Q) group we had 784 persons clicking on the study link altogether. Twenty-four persons dropped out right at the welcome screen before entering any information, 115 persons were not willing to download the Walkmeter app, another 19 stated that they usually are not physically active outdoors, and one person stated to be below age 18. Since these preconditions were necessary to enter the study, these persons were not allowed to participate. Six persons dropped out when reading or hearing the informed consent, 26 never returned after reading the instructions on how to use the Walkmeter app, and 25 dropped out after reading the task description and detailed information about the earnings, out of which three persons clicked on the calculation formula. Based on the mistakes in the answers to the comprehension questions, we guess that these 25 persons had difficulties understanding the earnings calculation. At the reporting stage, 67 persons were excluded for entering invalid entries, having missing data, or not filling out the report at all. Finally, we ended up with 501 observations, which is a participation rate of 63.90%.

In the (F) group we had 111 persons clicking on the study link altogether. Fifteen persons were not willing to download the Walkmeter app, and another two stated that they usually are not physically active outdoors. At the reporting stage, four persons were excluded for entering invalid entries or not filling out the report at all. Finally, we ended up with 90 observations, which is a participation rate of 81.08%.

In the (P) group we had 170 persons clicking on the study link altogether. Five persons dropped out right at the welcome screen before entering any information, 26 persons were not willing to download the Walkmeter app, and another three stated that they usually are not physically active outdoors. Three persons dropped out when reading or hearing the informed consent. Five never returned after reading the instructions on how to use the Walkmeter app. One person dropped out after reading the task description and detailed information about the earnings. At the reporting stage, 27 persons were excluded for entering invalid entries or not filling out the report at all. Finally, we ended up with 100 observations, which is a participation rate of 58.82%. On top of that, in parts of the analysis another 23 persons were excluded for uploading invalid screenshots. We used the valid screenshots to correct any typos and small mistakes by participants who otherwise filled out the report correctly.

In our experiment, the participants did not interact with the incentive mechanism in a repeated manner. We explained the mechanism to the participants before they measured and reported their data, and they were paid after they had reported the data. Therefore, the actual payments were not directly related with the observed behavior of the participants in this experiment. However, for completeness, this section provides details on how the lotteries in the (Q) group were drawn and final payments were made according to the PPTS mechanism.

We computed the clusters for every participant and for each of the seven fitness attributes, using the values reported by the participant and the other participants on the rest of the attributes. We treated the k (=50) nearest neighbors of any participant, according to Euclidean distance metric, as their cluster. Having defined the clusters, we computed the attribute score of any participant using the logarithm-based scoring formula given in [Goel 2020]. The same process was repeated for each of the attributes and for every participant. The final score for any participant was the average of all the attribute scores for that participant. To decide lottery winners from these scores, we normalized the scores and drew lottery (averaged over several thousand draws), in which the participants were assigned probabilities of winning that were proportional to their respective final scores.

This section contains the design details of the experiment, including an extract from the instructions. The extract contains the explanation of the PPTS mechanism to the participants. For the oTree codes and the full text of the instructions, both in English and the original German version, visit the online supplementary repository: https://osf.io/kj95x/?view_only=f096326afa0e4506bc481d5b6b1152d1.

We use a scientifically validated mechanism to assess the truthfulness of your entries to every single fitness attribute as listed above, such as walk or run time, distance, pace, energy burn, and so forth. Each entry gets scored.

Truthful and accurate entries get higher scores than untruthful and inaccurate entries. We calculate the individual chance of winning from the sum of scores for every single fitness data attribute.

In order to assess the truthfulness of your single entries, for example, your entry for distance, we compare your entry with the entries of “peers.” “Peers” are those study participants who reported similar values than you to all other entries, other than distance in this example. Since all fitness data attributes are connected to each other through human physiology, “peers” are very likely to be very similar in their physiology. Peer grouping is done by an algorithm. By this, we can still assess your entries in a personalized manner, yet anonymously. The more truthful and accurate your entries are, the better the quality of the peer grouping will be.

Your entries will score higher if they are more common in your group than overall among all study participants. Thus, to maximize your scores, make your entries as truthful and accurate as you can.

Click here, if you would like to see the calculation formula.

<For distance for example we apply:where the logarithmic function (log) is an increasing function, and commonness is the likelihood of your entry in the distribution of the same entries.>

Since it is individually the best strategy to report truthfully and accurately, you can assume that also other study participants will report truthfully and accurately.

Since the fitness data is not simultaneously collected from all study participants, it is not possible to provide you real-time feedback on your scores. The algorithms applied in this study may cause minor variation in scores. However, this does not affect the general tendency of true and accurate entries getting higher scores than fake or less accurate entries.