research-article

Can theories be tested?: a cryptographic treatment of forecast testing

Authors:

Rafael PassAuthors Info & Claims

ITCS '13: Proceedings of the 4th conference on Innovations in Theoretical Computer Science

Pages 47 - 56

https://doi.org/10.1145/2422436.2422443

Published: 09 January 2013 Publication History

Abstract

How do we test if a weather forecaster actually knows something about whether it will rain or not? Intuitively, a "good" forecast test should be complete---namely, a forecaster knowing the distribution of Nature should be able to pass the test with high probability, and sound---an uninformed forecaster should only be able to pass the test with small probability. We provide a comprehensive cryptographic study of the feasibility of complete and sound forecast testing, introducing various notions of both completeness and soundness, inspired by the literature on interactive proofs. Our main technical result is an incompleteness theorem for our most basic notion of computationally sound and complete forecast testing: If Nature is implemented by a polynomial-time algorithm, then every complete polynomial-time test can be passed by a completely uninformed polynomial-time forecaster (i.e., a computationally-bounded "charlatan") with high probability. We additionally study alternative notions of soundness and completeness and present both positive and negative results for these notions.

References

[1]

Mihir Bellare and Oded Goldreich, On defining proofs of knowledge, CRYPTO '92, 1992, pp. 390--420.

Digital Library

[2]

J. Eric Bickel, Some comparisons among quadratic, spherical, and logarithmic scoring rules, Decision Analysis 4 (2007), no. 2, 49--65.

Digital Library

[3]

László Babai and Shlomo Moran, Arthur-Merlin games: a randomized proof system, and a hierarchy of complexity class, J. Comput. Syst. Sci. 36 (1988), no. 2, 254--276.

Digital Library

[4]

Glenn W. Brier, Verification of forecasts expressed in terms of probability, Mon. Wea. Rev. 78 (1950), no. 1, 1--3.

[5]

A. P. Dawid, The well-calibrated bayesian, Journal of the American Statistical Association 77 (1982), no. 379, pp. 605--610.

[6]

Eddie Dekel and Yossi Feinberg, Non-bayesian testing of a stochastic prediction, Review of Economic Studies 73 (2006), no. 4, 893--906.

[7]

Uriel Feige and Adi Shamir, Zero knowledge proofs of knowledge in two rounds, CRYPTO, 1989, pp. 526--544.

Digital Library

[8]

Yoav Freund and Robert E. Schapire, Adaptive game playing using multiplicative weights, Games and Economic Behavior 29 (1999), no. 1, 79--103.

[9]

Dean P. Foster and Rakesh V. Vohra, Asymptotic calibration, Biometrika 85 (1998), no. 2, 379--390.

[10]

Lance Fortnow and Rakesh V. Vohra, The complexity of forecast testing, Econometrica 77 (2009), no. 1, 93--105.

[11]

Shafi Goldwasser, Silvio Micali, and Charles Rackoff, The knowledge complexity of interactive proof systems, SIAM Journal on Computing 18 (1989), no. 1, 186--208.

Digital Library

[12]

Tilmann Gneiting and Adrian E. Raftery, Strictly proper scoring rules, prediction, and estimation, Journal of the American Statistical Association 102 (2007), 359--378.

[13]

Russell Impagliazzo and Avi Wigderson, phP = BPP if E requires exponential circuits: derandomizing the XOR lemma, Proceedings of the twenty-ninth annual ACM symposium on Theory of computing, STOC '97, ACM, 1997, pp. 220--229.

Digital Library

[14]

Ehud Lehrer, Any inspection is manipulable, Econometrica 69 (2001), no. 5, pp. 1333--1347 (English).

[15]

Silvio Micali and Rafael Pass, Precise zero knowledge, 2007.

[16]

Noam Nisan and Avi Wigderson, Hardness vs randomness, J. Comput. Syst. Sci. 49 (1994), no. 2, 149--167.

Digital Library

[17]

W Olszewski and A Sandroni, Counterfactual predictions, 2006.

[18]

Wojciech Olszewski and Alvaro Sandroni, Strategic manipulation of empirical tests, Math. Oper. Res. 34 (2009), no. 1, 57--70.

Digital Library

[19]

Alvaro Sandroni, The reproducible properties of correct forecasts, International Journal of Game Theory 32 (2003), no. 1, 151--159.

[20]

Vladimir Vovk and Glenn Shafer, Good randomized sequential probability forecasting is always possible, the game-theoretic probability and finance project, http://probabilityandfinance.com, working paper, the Journal of the Royal Statistical Society B 67 (2003), 747--763.

Cited By

Rothblum G(2023)Indistinguishable Predictions and Multi-group Fair LearningAdvances in Cryptology – EUROCRYPT 202310.1007/978-3-031-30545-0_1(3-21)Online publication date: 23-Apr-2023
https://dl.acm.org/doi/10.1007/978-3-031-30545-0_1
Dwork CKim MReingold ORothblum GYona GKhuller SVassilevska Williams V(2021)Outcome indistinguishabilityProceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing10.1145/3406325.3451064(1095-1108)Online publication date: 15-Jun-2021
https://dl.acm.org/doi/10.1145/3406325.3451064
Chung KLui EPass R(2015)From Weak to Strong Zero-Knowledge and ApplicationsTheory of Cryptography10.1007/978-3-662-46494-6_4(66-92)Online publication date: 2015
https://doi.org/10.1007/978-3-662-46494-6_4

Index Terms

Can theories be tested?: a cryptographic treatment of forecast testing
1. Theory of computation
  1. Models of computation
    1. Interactive computation

Recommendations

The complexity of forecast testing: abstract
EC '08: Proceedings of the 9th ACM conference on Electronic commerce

Consider a weather forecaster predicting a probability of rain for the next day. We consider tests that given a finite sequence of forecast predictions and outcomes will either pass or fail the forecaster. Sandroni shows that any test which passes a ...
The complexity of forecast testing

Consider a weather forecaster predicting the probability of rain for the next day. We consider tests that given a finite sequence of forecast predictions and outcomes will either pass or fail the forecaster. It is known that any test which passes a ...
How Effectively Is Defective Code Actually Tested?: An Analysis of JUnit Tests in Seven Open Source Systems
PROMISE'18: Proceedings of the 14th International Conference on Predictive Models and Data Analytics in Software Engineering

Background: Newspaper headlines still regularly report latent software defects. Such defects have often evaded testing for many years. It remains difficult to identify how well a system has been tested. It also remains difficult to assess how successful ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ITCS '13: Proceedings of the 4th conference on Innovations in Theoretical Computer Science

January 2013

594 pages

ISBN:9781450318594

DOI:10.1145/2422436

Program Chair:
Robert Kleinberg
Cornell University

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGACT: ACM Special Interest Group on Algorithms and Computation Theory

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 January 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ITCS '13

Sponsor:

SIGACT

ITCS '13: Innovations in Theoretical Computer Science

January 9 - 12, 2013

California, Berkeley, USA

Acceptance Rates

Overall Acceptance Rate 172 of 513 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
84
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)0

Reflects downloads up to 06 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Rothblum G(2023)Indistinguishable Predictions and Multi-group Fair LearningAdvances in Cryptology – EUROCRYPT 202310.1007/978-3-031-30545-0_1(3-21)Online publication date: 23-Apr-2023
https://dl.acm.org/doi/10.1007/978-3-031-30545-0_1
Dwork CKim MReingold ORothblum GYona GKhuller SVassilevska Williams V(2021)Outcome indistinguishabilityProceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing10.1145/3406325.3451064(1095-1108)Online publication date: 15-Jun-2021
https://dl.acm.org/doi/10.1145/3406325.3451064
Chung KLui EPass R(2015)From Weak to Strong Zero-Knowledge and ApplicationsTheory of Cryptography10.1007/978-3-662-46494-6_4(66-92)Online publication date: 2015
https://doi.org/10.1007/978-3-662-46494-6_4

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten