Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/2229012.2229085acmconferencesArticle/Chapter ViewAbstractPublication PagesecConference Proceedingsconference-collections
research-article

Peer prediction without a common prior

Published: 04 June 2012 Publication History

Abstract

Reputation mechanisms at online opinion forums, such as Amazon Reviews, elicit ratings from users about their experience with different products. Crowdsourcing applications, such as image tagging on Amazon Mechanical Turk, elicit votes from users as to whether or not a job was duly completed. An important property in both settings is that the feedback received from users (agents) is truthful. The peer prediction method introduced by Miller et al. [2005] is a prominent theoretical mechanism for the truthful elicitation of reports. However, a significant obstacle to its application is that it critically depends on the assumption of a common prior amongst both the agents and the mechanism. In this paper, we develop a peer prediction mechanism for settings where the agents hold subjective and private beliefs about the state of the world and the likelihood of a positive signal given a particular state. Our shadow peer prediction mechanism exploits temporal structure in order to elicit two reports, a belief report and then a signal report, and it provides strict incentives for truthful reporting as long as the effect an agent's signal has on her posterior belief is bounded away from zero. Alternatively, this technical requirement on beliefs can be dispensed with by a modification in which the second report is a belief report rather than a signal report.

References

[1]
Friedman, D. 1983. Effective Scoring Rules for Probabilistic Forecasts. Management Science 29, 4, 447--454.
[2]
Gneiting, T. and Raftery, A. E. 2007. Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association 102, 359--378.
[3]
Jurca, R. and Faltings, B. 2006. Minimum Payments that Reward Honest Reputation Feedback. In Proceedings of the 7th ACM Conference on Electronic Commerce (EC'06).
[4]
Jurca, R. and Faltings, B. 2007. Robust Incentive-Compatible Feedback Payments. In Trust, Reputation and Security: Theories and Practice. LNAI Series, vol. 4452. Springer-Verlag, 204--218.
[5]
Jurca, R. and Faltings, B. 2008. Incentives for Expressing Opinions in Online Polls. In Proceedings of the 9th ACM Conference on Electronic Commerce (EC'08).
[6]
Jurca, R. and Faltings, B. 2009. Mechanisms for Making Crowds Truthful. Journal of Artificial Intelligence Research (JAIR) 34, 209--253.
[7]
Jurca, R. and Faltings, B. 2011. Incentives for Answering Hypothetical Questions. In Proceedings of the 1st Workshop on Social Computing and User Generated Content (SC'11).
[8]
Kalai, E. and Lehrer, E. 1993. Subjective equilibrium in repeated games. Econometrica 61, 1231--1240.
[9]
Miller, N., Resnick, P., and Zeckhauser, R. 2005. Eliciting Informative Feedback: The Peer-Prediction Method. Management Science 51, 9, 1359--1373.
[10]
P. Battigali, M. G. and Molinari, M. C. 1992. Learning convergence to equilibrium in repeated strategic interactions: An introductory survey,. Ricerche Economiche 96, 335--378.
[11]
Prelec, D. 2004. A Bayesian Truth Serum for Subjective Data. Science 306, 5695, 462--466.
[12]
Raykar, V. C., Yu, S., Zhao, L. H., Valadez, G. H., Florin, C., Bogoni, L., and Moy, L. 2010. Learning From Crowds. Journal of Machine Learning Research (JMLR) 11, 1297--1322.
[13]
Rubinstein, A. and Wolinsky, A. 1984. Rationalizable conjectural equilibrium: Between Nash and Rationalizability. Games Econom. Behavior 6, 299---311.
[14]
Selten, R. 1998. Axiomatic Characterization of the Quadratic Scoring Rule. Experimental Economics 1, 43--61.
[15]
Shaw, A. D., Horton, J. J., and Chen, D. L. 2011. Designing Incentives for Inexpert Human Raters. In Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work (CSCW '11). 275--284.
[16]
Witkowski, J. 2009. Eliciting Honest Reputation Feedback in a Markov Setting. In Proceedings of the 21st International Joint Conference on Artificial Intelligence (IJCAI'09).
[17]
Witkowski, J. 2010. Truthful Feedback for Sanctioning Reputation Mechanisms. In Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence (UAI'10).
[18]
Witkowski, J. and Parkes, D. 2012. A Robust Bayesian Truth Serum for Small Populations. In Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI'12).

Cited By

View all
  • (2024)On the Computation of Equilibria in Discrete First-Price AuctionsProceedings of the 25th ACM Conference on Economics and Computation10.1145/3670865.3673509(379-399)Online publication date: 8-Jul-2024
  • (2024)Dominantly Truthful Peer Prediction Mechanisms with a Finite Number of TasksJournal of the ACM10.1145/363823971:2(1-49)Online publication date: 10-Apr-2024
  • (2024)Identifying Wisdom (of the Crowd): A Regression ApproachJournal of Political Economy Microeconomics10.1086/733781Online publication date: 28-Oct-2024
  • Show More Cited By

Index Terms

  1. Peer prediction without a common prior

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    EC '12: Proceedings of the 13th ACM Conference on Electronic Commerce
    June 2012
    1016 pages
    ISBN:9781450314152
    DOI:10.1145/2229012
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 04 June 2012

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. information elicitation
    2. mechanism design
    3. peer prediction

    Qualifiers

    • Research-article

    Conference

    EC '12
    Sponsor:
    EC '12: ACM Conference on Electronic Commerce
    June 4 - 8, 2012
    Valencia, Spain

    Acceptance Rates

    Overall Acceptance Rate 664 of 2,389 submissions, 28%

    Upcoming Conference

    EC '25
    The 25th ACM Conference on Economics and Computation
    July 7 - 11, 2025
    Stanford , CA , USA

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)29
    • Downloads (Last 6 weeks)3
    Reflects downloads up to 18 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)On the Computation of Equilibria in Discrete First-Price AuctionsProceedings of the 25th ACM Conference on Economics and Computation10.1145/3670865.3673509(379-399)Online publication date: 8-Jul-2024
    • (2024)Dominantly Truthful Peer Prediction Mechanisms with a Finite Number of TasksJournal of the ACM10.1145/363823971:2(1-49)Online publication date: 10-Apr-2024
    • (2024)Identifying Wisdom (of the Crowd): A Regression ApproachJournal of Political Economy Microeconomics10.1086/733781Online publication date: 28-Oct-2024
    • (2023)Incentivizing honesty among competitors in collaborative learning and optimizationProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3666459(7659-7696)Online publication date: 10-Dec-2023
    • (2023)Two Strongly Truthful Mechanisms for Three Heterogeneous Agents Answering One QuestionACM Transactions on Economics and Computation10.1145/356556010:4(1-26)Online publication date: 21-Feb-2023
    • (2023)Surrogate Scoring RulesACM Transactions on Economics and Computation10.1145/356555910:3(1-36)Online publication date: 15-Feb-2023
    • (2022)Public Participation Consortium Blockchain for Smart City GovernanceIEEE Internet of Things Journal10.1109/JIOT.2021.30911519:3(2094-2108)Online publication date: 1-Feb-2022
    • (2022)Delegated expertise: Implementability with peer-monitoringGames and Economic Behavior10.1016/j.geb.2021.12.008Online publication date: Jan-2022
    • (2022)Cooperative Trustworthiness Evaluation and Trustworthy Service RatingCooperation and Integration in 6G Heterogeneous Networks10.1007/978-981-19-7648-3_16(389-416)Online publication date: 5-Oct-2022
    • (2021)Monitoring expertsTheoretical Economics10.3982/TE401716:4(1313-1350)Online publication date: 2021
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media