article

Effective Scoring Rules for Probabilistic Forecasts

Author:

Daniel FriedmanAuthors Info & Claims

Management Science, Volume 29, Issue 4

Pages 447 - 454

https://doi.org/10.1287/mnsc.29.4.447

Published: 01 April 1983 Publication History

Abstract

This paper studies the use of a scoring rule for the elicitation of forecasts in the form of probability distributions and for the subsequent evaluation of such forecasts. Given a metric distance function on a space of probability distributions, a scoring rule is said to be effective if the forecaster's expected score is a strictly decreasing function of the distance between the elicited and "true" distributions. Two simple, well-known rules the spherical and the quadratic are shown to be effective with respect to suitable metrics. Examples and a practical application in Foreign Exchange rate forecasting are also provided.

References

[1]

DE FINETTI, B., "Methods for Discriminating Levels of Partial Knowledge Concerning a Test Item," British J. Math. and Statist. Psych., Vol. 18 (May 1965), pp. 87-123.

Crossref

Google Scholar

[2]

EPSTEIN, E. S., "A Scoring System for Probability Forecasts of Ranked Categories," J. Appl. Meteorol., Vol. 18 (December 1969), pp. 985-987.

Crossref

Google Scholar

[3]

FRIEDMAN, D., "Effective Scoring Rules for Probability Distributions," UCLA, Department of Economics Discussion Paper No. 164, November 1979.

Google Scholar

[4]

HOLT, C. A., "Elicitation of Subjective Probability Distributions and von Neumann-Morgenstern Utility Functions," Department of Economics Discussion Paper No. 79-128, University of Minnesota, 1979.

Google Scholar

[5]

KABUS, I., "You Can Bank on Uncertainty," Harvard Bus. Rev., Vol. 54 (May 1976), pp. 95-105.

Google Scholar

[6]

MATHESON, J. E. AND WINKLER, R. L., "Scoring Rules for Continuous Probability Distributions," Management Sci., Vol. 22 (1976), pp. 1087-1096.

Crossref

Google Scholar

[7]

MURPHY, A. H. AND WINKLER, R. L., "Scoring Rules in Probability Assessment and Evaluation," Acta Psych., Vol. 34 (1970), pp. 273-286.

Crossref

Google Scholar

[8]

RUDIN, W., Real and Complex Analysis, McGraw-Hill, New York, 1966.

Google Scholar

[9]

SAVAGE, L. J., "Elicitation of Personal Probabilities and Expectations," J. Amer. Statist. Assoc., Vol. 66 (1971), pp. 783-801.

Crossref

Google Scholar

[10]

STAEL VON HOLSTEIN, C.-A. S., "Measurement of Subjective Probability," Acta Psych., Vol. 34 (1970), pp. 146-159.

Crossref

Google Scholar

[11]

WINKLER, R. L., "Scoring Rules and the Evaluation of Probability Assessors," J. Amer. Statist. Assoc., Vol. 64 (September 1969), pp. 1073-78.

Crossref

Google Scholar

Cited By

View all

d'Eon GGreenwood SLeyton-Brown KWright JWooldridge MDy JNatarajan S(2024)How to evaluate behavioral modelsProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i9.28820(9636-9644)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i9.28820
Simon J(2024)Inefficient Forecasts at the SportsbookManagement Science10.1287/mnsc.2022.0045670:12(8583-8611)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1287/mnsc.2022.00456
Yu LSong JSong YErmon SRanzato MBeygelzimer ADauphin YLiang PVaughan J(2021)Pseudo-spherical contrastive divergenceProceedings of the 35th International Conference on Neural Information Processing Systems10.5555/3540261.3541973(22348-22362)Online publication date: 6-Dec-2021
https://dl.acm.org/doi/10.5555/3540261.3541973
Show More Cited By

Effective Scoring Rules for Probabilistic Forecasts
1. Applied computing
2. Mathematics of computing
  1. Probability and statistics
    1. Probabilistic reasoning algorithms

Recommendations

Should Scoring Rules be "Effective"?

A scoring rule is a reward function for eliciting or evaluating forecasts expressed as discrete or continuous probability distributions. A rule is strictly proper if it encourages the forecaster to state his true subjective probabilities, and effective ...
An Evaluation of Rules for Selecting an Extrapolation Model on Yearly Sales Forecasts

<P>Six rules are used to select among seven exponential smoothing models to pick the model that yields the lowest error on a fit to historical data. Comparisons of accuracy are made among forecasts generated by the selection rules and the individual ...
Bringing judgment into combination forecasts
Abstract
This research investigates the benefits in forecast accuracy by combining judgmental forecasts with those generated by statistical models. Our study differs from prior research efforts in this area along two important dimensions. First, two ...

Comments

Information & Contributors

Information

Published In

Management Science Volume 29, Issue 4

April 1983

137 pages

ISSN:0025-1909

Issue’s Table of Contents

Publisher

INFORMS

Linthicum, MD, United States

Publication History

Published: 01 April 1983

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 19 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

d'Eon GGreenwood SLeyton-Brown KWright JWooldridge MDy JNatarajan S(2024)How to evaluate behavioral modelsProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i9.28820(9636-9644)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i9.28820
Simon J(2024)Inefficient Forecasts at the SportsbookManagement Science10.1287/mnsc.2022.0045670:12(8583-8611)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1287/mnsc.2022.00456
Yu LSong JSong YErmon SRanzato MBeygelzimer ADauphin YLiang PVaughan J(2021)Pseudo-spherical contrastive divergenceProceedings of the 35th International Conference on Neural Information Processing Systems10.5555/3540261.3541973(22348-22362)Online publication date: 6-Dec-2021
https://dl.acm.org/doi/10.5555/3540261.3541973
Zawadzki ELahaie S(2015)Nonparametric scoring rulesProceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence10.5555/2888116.2888221(3635-3641)Online publication date: 25-Jan-2015
https://dl.acm.org/doi/10.5555/2888116.2888221
Carvalho ALarson K(2013)A consensual linear opinion poolProceedings of the Twenty-Third international joint conference on Artificial Intelligence10.5555/2540128.2540491(2518-2524)Online publication date: 3-Aug-2013
https://dl.acm.org/doi/10.5555/2540128.2540491
Witkowski JParkes DFaltings BLeyton-Brown KIpeirotis P(2012)Peer prediction without a common priorProceedings of the 13th ACM Conference on Electronic Commerce10.1145/2229012.2229085(964-981)Online publication date: 4-Jun-2012
https://dl.acm.org/doi/10.1145/2229012.2229085
Lambert NPennock DShoham YFortnow LRiedl JSandholm T(2008)Eliciting properties of probability distributionsProceedings of the 9th ACM conference on Electronic commerce10.1145/1386790.1386813(129-138)Online publication date: 8-Jul-2008
https://dl.acm.org/doi/10.1145/1386790.1386813
Nikolova ESami RMacKie-Mason JParkes DResnick P(2007)A strategic model for information marketsProceedings of the 8th ACM conference on Electronic commerce10.1145/1250910.1250956(316-325)Online publication date: 11-Jun-2007
https://dl.acm.org/doi/10.1145/1250910.1250956
Miller NResnick PZeckhauser R(2005)Eliciting Informative FeedbackManagement Science10.1287/mnsc.1050.037951:9(1359-1373)Online publication date: 1-Sep-2005
https://dl.acm.org/doi/10.1287/mnsc.1050.0379
Nau R(1985)Should Scoring Rules be "Effective"?Management Science10.1287/mnsc.31.5.52731:5(527-535)Online publication date: 1-May-1985
https://dl.acm.org/doi/10.1287/mnsc.31.5.527

Abstract

References

Cited By

Recommendations

Should Scoring Rules be "Effective"?

An Evaluation of Rules for Selecting an Extrapolation Model on Yearly Sales Forecasts

Bringing judgment into combination forecasts

Comments

Information

Published In

Publisher

Publication History

Author Tags

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

Share

Share this Publication link

Share on social media

Affiliations