Abstract
Many recommendation and decision processes depend on eliciting evaluations of opportunities, products, and vendors. A scoring system is devised that induces honest reporting of feedback. Each rater merely reports a signal, and the system applies proper scoring rules to the implied posterior beliefs about another rater’s report. Honest reporting proves to be a Nash Equilibrium. The scoring schemes can be scaled to induce appropriate effort by raters and can be extended to handle sequential interaction and continuous signals. We also address a number of practical implementation issues that arise in settings such as academic reviewing and on-line recommender and reputation systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Avery, C., P. Resnick, and R. Zeckhauser, 1999. “The Market for Evaluations”, American Economic Review, 89(3) 564–584.
Breese, J., D. Heckerman, C. Kadie. 1998. Empirical Analysis of Predictive Algorithms for Collaborative Filtering. Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, Morgan Kaufmann Publisher, Madison, WI, July, 1998.
Clemen, R. 2002. Incentive Contracts and Strictly Proper Scoring Rules. Test 11(1) 195–217.
Cooke, R. M. 1991. Experts in Uncertainty: Opinion and Subjective Probability in Science. Oxford University Press, New York.
Crémer, J., R. McLean. 1985. Optimal Selling Strategies Under Uncertainty for a Discriminating Monopolist When Demands Are Interdependent. Econometrica 53(2) 345–361.
Crémer, J., R. Mc Lean. 1988. Full Extraction of Surplus in Bayesian and Dominant Strategy Auctions. Econometrica 56 (6) 1247–1257.
d’Aspremont, C., L.-A. Gérard-Varet. 1979. Incentives and Incomplete Information. Journal of Public Economics 11(1) 25–45.
d’Asprement, C.A., L.-A. Gérard-Varet. 1982. Bayesian Incentive Compatible Beliefs. Journal of Mathematical Economics 10(1) 83–103.
Dellarocas, C. 2001. Analyzing the Economic Efficiency of eBay-like Online Reputation Reporting Mechanisms. Proceedings of the 3rd ACM Conference on Electronic Commerce, Tampa, FL, October 14–16, 2001.
Friedman, D. 1983. Effective Scoring Rules for Probabilistic Forecasts. Management Science 29(4) 447–454.
Gollier, C. 2001. The Economics of Risk and Time. MITPress, Cambridge, MA.
Hanson, R. 2002. Logarithmic Market Scoring Rules for Modular Combinatoral Information Aggregation. Working Paper, George Mason University Department of Economics, http://hanson.gmu.edu/mktscore.pdf.
Jensen, F., C. Peterson. 1973. Psychological Effects of Proper Scoring Rules. Organizational Behavior and Human Performance 9(2) 307–317.
Johnson, S., N. Miller, J. Pratt, R. Zeckhauser. 2002. Efficient Design with Interdependent Valuations and an Informed Center. Kennedy School Working Paper, RWP02-025, http://ksgnotes1.harvard.edu/research/wpaper.nsf/rwp/RWP02-025.
Johnson, S., J. Pratt, R. Zeckhauser. 1990. Efficiency Despite Mutually Payoff-Relevant Private Information: The Finite Case. Econometrica 58(4) 873–900.
Kandori, M., H. Matsushima. 1998. Private Observation, Communication and Collusion. Econometrica 66(3) 627–652.
Lampe, C., P. Resnick. 2004. Slash(dot) and Burn: Distributed Moderation in a Large Online Conversation Space. CHI 2004, ACM Conference on Human Factors in Computing Systems, CHI Letters 6(1), 543–550.
Lavalle, I. 1968. On Cash Equivalents and Information Evaluation in Decisions Under Uncertainty: Part I: Basic Theory. Journal of the American Statistical Association 63(321) 252–276.
Miller, N., P. Resnick, R. Zeckhauser. 2005. A sufficient condition for correlated information in mechanism design.mimeo, Harvard University, Cambridge, MA.
Nau, R. 1985. Should Scoring Rules Be Effective? Management Science 34(5) 527–535.
Nelson, R., D. Bessler. 1989. Subjective Probabilities and Scoring Rules: Experimental Evidence. American Journal of Agricultural Economics 71(2) 363–369.
Ottaviani, M., P. N. Sørensen. 2003. Professional Advice: The Theory of Reputational Cheap Talk.Working Paper, University of Copenhagen, http://www.econ.ku.dk/sorensen/ Papers/pa.pdf.
Pratt, J., H. Raiffa, R. Schlaifer. 1965. Introduction to Statistical Decision Theory. McGraw-Hill, New York.
Prelec, D. 2001. A two-person scoring rule for subjective reports. Working Paper, Marketing Center, MITSloan School, http://mitsloan.mit.edu/vc/IPPacket.pdf.
Prendergast, C. 1993. A Theory of Yes Men. American Economic Review 83(4) 757–770.
Roth, A. 1995. Introduction to Experimental Economics. J. Kagel, A. Roth, eds. The Handbook of Experimental Economics, Princeton University Press, Princeton, NJ, 3–110.
Sarwar, B. M., G. Karypis, J. A. Konstan, J. Riedl. 2000. Analysis of Recommender Algorithms for E-Commerce, Proceedings of the ACM E-Commerce 2000 Conference, Oct. 17–20, 2000, 158–167.
Savage, L. 1954. Foundations of Statistics. Dover Publications, New York.
Savage, L. 1971. Elicitation of Personal Probabilities and Expectations. Journal of the American Statistical Association 66(336) 783–801.
Selten, R. 1998. Axiomatic Characterization of the Quadratic Scoring Rule. Experimental Economics 1(1) 43–62.
Smith, C. 1961. Consistency in Statistical Inference and Decision. Journal of the Royal Statistical Society, Series B (Methodological) 23(1) 1–37.
Staël von Holstein, C.-A. 1970. Measurement of Subjective Probability. Acta Psychologica 34(1) 146–159.
Winkler, R. 1969. Scoring Rules and the Evaluation of Probability Assessors. Journal of the American Statistical Association 64(327) 1073–1078.
Winkler, 1996. Scoring Rules and the Evaluation of Probabilities. Test 5(1) 1–60.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag London Limited
About this chapter
Cite this chapter
Miller, N., Resnick, P., Zeckhauser, R. (2009). Eliciting Informative Feedback: The Peer-Prediction Method. In: Golbeck, J. (eds) Computing with Social Trust. Human–Computer Interaction Series. Springer, London. https://doi.org/10.1007/978-1-84800-356-9_8
Download citation
DOI: https://doi.org/10.1007/978-1-84800-356-9_8
Publisher Name: Springer, London
Print ISBN: 978-1-84800-355-2
Online ISBN: 978-1-84800-356-9
eBook Packages: Computer ScienceComputer Science (R0)