research-article

Surrogate Scoring Rules

Authors:

Yiling ChenAuthors Info & Claims

ACM Transactions on Economics and Computation, Volume 10, Issue 3

Article No.: 12, Pages 1 - 36

https://doi.org/10.1145/3565559

Published: 15 February 2023 Publication History

Abstract

Strictly proper scoring rules (SPSR) are incentive compatible for eliciting information about random variables from strategic agents when the principal can reward agents after the realization of the random variables. They also quantify the quality of elicited information, with more accurate predictions receiving higher scores in expectation. In this article, we extend such scoring rules to settings in which a principal elicits private probabilistic beliefs but only has access to agents’ reports. We name our solution Surrogate Scoring Rules (SSR). SSR is built on a bias correction step and an error rate estimation procedure for a reference answer defined using agents’ reports. We show that, with a little information about the prior distribution of the random variables, SSR in a multi-task setting recover SPSR in expectation, as if having access to the ground truth. Therefore, a salient feature of SSR is that they quantify the quality of information despite the lack of ground truth, just as SPSR do for the setting with ground truth. As a by-product, SSR induce dominant uniform strategy truthfulness in reporting. Our method is verified both theoretically and empirically using data collected from real human forecasters.

References

[1]

Adam Altmejd, Anna Dreber, Eskil Forsell, Juergen Huber, Taisuke Imai, Magnus Johannesson, Michael Kirchler, Gideon Nave, and Colin Camerer. 2019. Predicting the replicability of social science lab experiments. PLOS One 14, 12 (2019).

[2]

Dana Angluin and Philip Laird. 1988. Learning from noisy examples. Machine Learning 2, 4 (1988), 343–370.

[3]

Pavel Atanasov, Phillip Rescober, Eric Stone, Samuel A. Swift, Emile Servan-Schreiber, Philip Tetlock, Lyle Ungar, and Barbara Mellers. 2016. Distilling the wisdom of crowds: Prediction markets vs. prediction polls. Management Science 63, 3 (2016), 691–706.

Digital Library

[4]

Glenn W. Brier. 1950. Verification of forecasts expressed in terms of probability. Monthly Weather Review 78, 1 (1950), 1–3.

[5]

Tom Bylander. 1994. Learning linear threshold functions in the presence of classification noise. In Proceedings of the 7th Annual Conference on Computational Learning Theory. ACM, 340–347.

Digital Library

[6]

Anirban Dasgupta and Arpita Ghosh. 2013. Crowdsourced judgement elicitation with endogenous proficiency. In Proceedings of the 22nd International Conference on World Wide Web. 319–330.

Digital Library

[7]

Darrell Duffie and Jun Pan. 1997. An overview of value at risk. Journal of Derivatives 4, 3 (1997), 7–49.

[8]

Alexander Frankel and Emir Kamenica. 2019. Quantifying information and uncertainty. American Economic Review 109, 10 (2019), 3650–80.

[9]

Benoît Frénay and Michel Verleysen. 2014. Classification in the presence of label noise: A survey. IEEE Transactions on Neural Networks and Learning Systems 25, 5 (2014), 845–869.

[10]

Jeffrey A. Friedman, Joshua D. Baker, Barbara A. Mellers, Philip E. Tetlock, and Richard Zeckhauser. 2018. The value of precision in probability assessment: Evidence from a large-scale geopolitical forecasting tournament. International Studies Quarterly 62, 2 (2018), 410–422.

[11]

Rafael Frongillo and Jens Witkowski. 2016. A geometric method to construct minimal peer prediction mechanisms. In 13th AAAI Conference on Artificial Intelligence.

[12]

Tilmann Gneiting and Adrian E. Raftery. 2005. Weather forecasting with ensemble methods. Science 310, 5746 (2005), 248–249.

[13]

Tilmann Gneiting and Adrian E. Raftery. 2007. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association 102, 477 (2007), 359–378.

[14]

Naman Goel and Boi Faltings. 2019. Deep Bayesian trust: A dominant and fair incentive mechanism for crowd. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 1996–2003.

Digital Library

[15]

Suzanne Hoogeveen, Alexandra Sarafoglou, and Eric-Jan Wagenmakers. 2019. Laypeople can predict which social science studies replicate. (2019).

[16]

IARPA. 2019. Hybrid Forecasting Competition. Retrieved October 13, 2022 from https://www.iarpa.gov/index.php/research-programs/hfc?id=661.

[17]

Victor Richmond Jose, Robert F. Nau, and Robert L. Winkler. 2006. Scoring Rules, Generalized Entropy and Utility Maximization. (2006). Working Paper, Fuqua School of Business, Duke University.

[18]

Radu Jurca and Boi Faltings. 2007. Collusion-resistant, incentive-compatible feedback payments. In Proceedings of the 8th ACM Conference on Electronic Commerce. ACM, 200–209.

Digital Library

[19]

Radu Jurca and Boi Faltings. 2009. Mechanisms for making crowds truthful. Journal of Artificial Intelligence Research 34 (2009), 209–253.

Digital Library

[20]

Yuqing Kong. 2020. Dominantly truthful multi-task peer prediction with a constant number of tasks. In Proceedings of the 14th Annual ACM-SIAM Symposium on Discrete Algorithms. SIAM, 2398–2411.

[21]

Yuqing Kong, Katrina Ligett, and Grant Schoenebeck. 2016. Putting peer prediction under the micro (economic) scope and making truth-telling focal. In International Conference on Web and Internet Economics. Springer, 251–264.

Digital Library

[22]

Yuqing Kong and Grant Schoenebeck. 2018. Water from two rocks: Maximizing the mutual information. In Proceedings of the 2018 ACM Conference on Economics and Computation. 177–194.

Digital Library

[23]

Yuqing Kong and Grant Schoenebeck. 2019. An information theoretic framework for designing information elicitation mechanisms that reward truth-telling. ACM Transactions on Economics and Computation 7, 1 (2019), 2.

Digital Library

[24]

Yuqing Kong, Grant Schoenebeck, Biaoshuai Tao, and Fang-Yi Yu. 2020. Information elicitation mechanisms for statistical estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 2095–2102.

[25]

Yang Liu and Mingyan Liu. 2015. An online learning approach to improving the quality of crowd-sourcing. In Proceedings of the 2015 ACM SIGMETRICS (Portland, Oregon). ACM, New York, NY, 217–230.

Digital Library

[26]

John McCarthy. 1956. Measures of the value of information. PNAS: Proceedings of the National Academy of Sciences of the United States of America 42, 9 (1956), 654–655.

[27]

Aditya Menon, Brendan Van Rooyen, Cheng Soon Ong, and Bob Williamson. 2015. Learning from corrupted binary labels via class-probability estimation. In International Conference on Machine Learning. 125–134.

[28]

Nolan Miller, Paul Resnick, and Richard Zeckhauser. 2005. Eliciting informative feedback: The peer-prediction method. Management Science 51, 9 (2005), 1359–1373.

Digital Library

[29]

Nagarajan Natarajan, Inderjit S. Dhillon, Pradeep K. Ravikumar, and Ambuj Tewari. 2013. Learning with noisy labels. In Advances in Neural Information Processing Systems. 1196–1204.

Digital Library

[30]

Matthew Parry et al. 2016. Linear scoring rules for probabilistic binary classification. Electronic Journal of Statistics 10, 1 (2016), 1596–1607.

[31]

Dražen Prelec. 2004. A Bayesian truth serum for subjective data. Science 306, 5695 (2004), 462–466.

[32]

Dražen Prelec, H. Sebastian Seung, and John McCoy. 2017. A solution to the single-question crowd wisdom problem. Nature 541, 7638 (2017), 532.

[33]

Goran Radanovic and Boi Faltings. 2013. A robust Bayesian truth serum for non-binary signals. In Proceedings of the 27th AAAI Conference on Artificial Intelligence (AAAI’13).

Digital Library

[34]

Goran Radanovic and Boi Faltings. 2014. Incentives for truthful information elicitation of continuous signals. In 28th AAAI Conference on Artificial Intelligence.

[35]

Goran Radanovic, Boi Faltings, and Radu Jurca. 2016. Incentives for effort in crowdsourcing using the peer truth serum. ACM Transactions on Intelligent Systems and Technology 7, 4 (2016), 48.

Digital Library

[36]

Blake Riley. 2014. Minimum truth serums with optional predictions. In Proceedings of the 4th Workshop on Social Computing and User Generated Content (SC’14).

[37]

Leonard J. Savage. 1971. Elicitation of personal probabilities and expectations. Journal of the American Statistical Association 66, 336 (1971), 783–801.

[38]

Grant Schoenebeck and Fang-Yi Yu. 2020. Learning and strongly truthful multi-task peer prediction: A variational approach. arXiv preprint arXiv:2009.14730 (2020).

[39]

Grant Schoenebeck and Fang-Yi Yu. 2020. Two strongly truthful mechanisms for three heterogeneous agents answering one question. In International Conference on Web and Internet Economics. Springer, 119–132.

Digital Library

[40]

Clayton Scott. 2015. A rate of convergence for mixture proportion estimation, with application to learning from noisy labels. In AISTATS.

[41]

Clayton Scott, Gilles Blanchard, Gregory Handy, Sara Pozzi, and Marek Flaska. 2013. Classification with asymmetric label noise: Consistency and maximal denoising. In COLT. 489–511.

[42]

Victor Shnayder, Arpit Agarwal, Rafael Frongillo, and David C. Parkes. 2016. Informed truthfulness in multi-task peer prediction. In Proceedings of the 2016 ACM Conference on Economics and Computation. ACM, 179–196.

Digital Library

[43]

Philip E. Tetlock, Barbara A. Mellers, Nick Rohrbaugh, and Eva Chen. 2014. Forecasting tournaments: Tools for increasing transparency and improving the quality of debate. Current Directions in Psychological Science 23, 4 (2014), 290–295.

[44]

Brendan van Rooyen and Robert C. Williamson. 2015. Learning in the presence of corruption. arXiv preprint:1504.00091 (2015).

[45]

Juntao Wang, Yang Liu, and Yiling Chen. 2019. Forecast aggregation via peer prediction. arXiv preprint arXiv:1910.03779 (2019).

[46]

Robert L. Winkler. 1969. Scoring rules and the evaluation of probability assessors. Journal of the American Statistical Association 64, 327 (1969), 1073–1078.

[47]

Jens Witkowski, Pavel Atanasov, Lyle H. Ungar, and Andreas Krause. 2017. Proper proxy scoring rules. In 31st AAAI Conference on Artificial Intelligence.

[48]

J. Witkowski and D. C. Parkes. 2012. Peer prediction without a common prior. In Proceedings of the 13th ACM Conference on Electronic Commerce (EC’12). ACM, 964–981.

Digital Library

[49]

Jens Witkowski and David C. Parkes. 2012. A robust Bayesian truth serum for small populations. In Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI’12).

[50]

Jens Witkowski and David C. Parkes. 2013. Learning the prior in minimal peer prediction. In Proceedings of the 3rd Workshop on Social Computing and User Generated Content at the ACM Conference on Electronic Commerce, Vol. 14.

Cited By

Lu YXu SZhang YKong YSchoenebeck GKleinberg RSaban DBergemann D(2024)Eliciting Informative Text Evaluations with Large Language ModelsProceedings of the 25th ACM Conference on Economics and Computation10.1145/3670865.3673532(582-612)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3670865.3673532
Kong Y(2024)Dominantly Truthful Peer Prediction Mechanisms with a Finite Number of TasksJournal of the ACM10.1145/363823971:2(1-49)Online publication date: 10-Apr-2024
https://dl.acm.org/doi/10.1145/3638239
Shan LZhang SZhang JWang ZChua TNgo CKa-Wei Lee RKumar RLauw H(2024)On Truthful Item-Acquiring Mechanisms for Reward MaximizationProceedings of the ACM Web Conference 202410.1145/3589334.3645345(25-35)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645345
Show More Cited By

Index Terms

Surrogate Scoring Rules
1. Information systems
  1. World Wide Web
    1. Web applications
      1. Crowdsourcing
        Incentive schemes
2. Theory of computation
  1. Theory and algorithms for application domains
    1. Algorithmic game theory and mechanism design
      1. Quality of equilibria

Recommendations

Surrogate Scoring Rules
EC '20: Proceedings of the 21st ACM Conference on Economics and Computation

Strictly proper scoring rules (SPSR) are incentive compatible for eliciting information about random variables from strategic agents when the principal can reward agents after the realization of the random variables. They also quantify the quality of ...
Putting Peer Prediction Under the Microeconomicscope and Making Truth-Telling Focal
WINE 2016: Proceedings of the 12th International Conference on Web and Internet Economics - Volume 10123

Peer-predictionï ź[19] is a meta-mechanism which, given any proper scoring rule, produces a mechanism to elicit prie information from self-interested agents. Formally, truth-telling is a strict Nash equilibrium of the mechanism. Unfortunately, there may ...
Optimization of Scoring Rules
EC '22: Proceedings of the 23rd ACM Conference on Economics and Computation

This paper introduces an objective for optimizing proper scoring rules. The objective is to maximize the increase in payoff of a forecaster who exerts a binary level of effort to refine a posterior belief from a prior belief. In this framework we ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Economics and Computation

ACM Transactions on Economics and Computation Volume 10, Issue 3

September 2022

115 pages

ISSN:2167-8375

EISSN:2167-8383

DOI:10.1145/3572855

Editors:
David Pennock
Rutgers University, USA
,
Ilya Segal
Stanford University, USA

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 February 2023

Online AM: 08 October 2022

Accepted: 30 September 2022

Revised: 10 September 2022

Received: 15 August 2020

Published in TEAC Volume 10, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

National Science Foundation (NSF)
Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA)
Defense Advanced Research Projects Agency (DARPA) and Space and Naval Warfare Systems Center Pacific (SSC Pacific)

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
419
Total Downloads

Downloads (Last 12 months)126
Downloads (Last 6 weeks)13

Reflects downloads up to 19 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Lu YXu SZhang YKong YSchoenebeck GKleinberg RSaban DBergemann D(2024)Eliciting Informative Text Evaluations with Large Language ModelsProceedings of the 25th ACM Conference on Economics and Computation10.1145/3670865.3673532(582-612)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3670865.3673532
Kong Y(2024)Dominantly Truthful Peer Prediction Mechanisms with a Finite Number of TasksJournal of the ACM10.1145/363823971:2(1-49)Online publication date: 10-Apr-2024
https://dl.acm.org/doi/10.1145/3638239
Shan LZhang SZhang JWang ZChua TNgo CKa-Wei Lee RKumar RLauw H(2024)On Truthful Item-Acquiring Mechanisms for Reward MaximizationProceedings of the ACM Web Conference 202410.1145/3589334.3645345(25-35)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645345
He QYan ZDiao WSun X(2024)DLC: Dynamic Loss Correction for Cross-Domain Remotely Sensed SegmentationIEEE Transactions on Geoscience and Remote Sensing10.1109/TGRS.2024.340212762(1-14)Online publication date: 2024
https://doi.org/10.1109/TGRS.2024.3402127
Liu YElkind E(2023)The importance of human-labeled data in the era of LLMsProceedings of the Thirty-Second International Joint Conference on Artificial Intelligence10.24963/ijcai.2023/802(7026-7032)Online publication date: 19-Aug-2023
https://dl.acm.org/doi/10.24963/ijcai.2023/802
Faltings BElkind E(2023)Game-theoretic mechanisms for eliciting accurate informationProceedings of the Thirty-Second International Joint Conference on Artificial Intelligence10.24963/ijcai.2023/740(6601-6609)Online publication date: 19-Aug-2023
https://dl.acm.org/doi/10.24963/ijcai.2023/740

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View full text|Download PDF

View Issue’s Table of Contents