research-article

A type theory for probability density functions

Authors:

Ashish Agarwal,

Alexander GrayAuthors Info & Claims

ACM SIGPLAN Notices, Volume 47, Issue 1

Pages 545 - 556

https://doi.org/10.1145/2103621.2103721

Published: 25 January 2012 Publication History

Abstract

There has been great interest in creating probabilistic programming languages to simplify the coding of statistical tasks; however, there still does not exist a formal language that simultaneously provides (1) continuous probability distributions, (2) the ability to naturally express custom probabilistic models, and (3) probability density functions (PDFs). This collection of features is necessary for mechanizing fundamental statistical techniques. We formalize the first probabilistic language that exhibits these features, and it serves as a foundational framework for extending the ideas to more general languages. Particularly novel are our type system for absolutely continuous (AC) distributions (those which permit PDFs) and our PDF calculation procedure, which calculates PDF s for a large class of AC distributions. Our formalization paves the way toward the rigorous encoding of powerful statistical reformulations.

Supplementary Material

JPG File (popl_8b_1.jpg)

Download
15.75 KB

MP4 File (popl_8b_1.mp4)

Download
214.87 MB

References

[1]

A. Agarwal, S. Bhat, A. Gray, and I. E. Grossmann. Automating Mathematical Program Transformations. In Practical Aspects of Declarative Languages, 2010.

Digital Library

[2]

P. Audebaud and C. Paulin-Mohring. Proofs of Randomized Algorithms in Coq. In Mathematics of Program Construction, pages 49--68. Springer, 2006.

Digital Library

[3]

C. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.

Digital Library

[4]

J. Borgstram, A. D. Gordon, M. Greenberg, J. Margetson, and J. V. Gael. Measure Transformer Semantics for Bayesian Machine Learning. In European Symposium on Programming, pages 77--96, 2011.

Digital Library

[5]

H. Daumé III. HBC: Hierarchical Bayes Compiler, 2007. URL http://hal3.name/HBC.

[6]

L. Devroye. Non-Uniform Random Variate Generation, 1986.

[7]

M. Erwig and S. Kollmansberger. Functional Pearls: Probabilistic Functional Programming in Haskell. Journal of Functional Programming, 16 (01): 21--34, 2005.

Digital Library

[8]

M. Giry. A Categorical Approach to Probability Theory. Categorical Aspects of Topology and Analysis, 915: 68--85, 1981.

[9]

N. Goodman, V. Mansinghka, D. Roy, K. Bonawitz, and J. Tenenbaum. Church: A Language for Generative Models. In Uncertainty in Artificial Intelligence, 2008.

[10]

A. G. Gray, B. Fischer, J. Schumann, and W. Buntine. Automatic Derivation of Statistical Algorithms: The EM Family and Beyond. In Advances in Neural Information Processing Systems, 2003.

[11]

M. Hoyrup, C. Rojas, and K. Weihrauch. The Radon-Nikodym operator is not computable. In Computability & Complexity in Analysis, 2011.

[12]

K. Kersting and L. De Raedt. Bayesian Logic Programming: Theory and Tool. In Introduction to Statistical Relational Learning. 2007.

[13]

O. Kiselyov and C. Shan. Embedded Probabilistic Programming. In Working Conference on Domain Specific Languages. Springer, 2009.

Digital Library

[14]

D. Kozen. Semantics of Probabilistic Programs. Journal of Computer and System Sciences, 22 (3): 328--350, 1981.

[15]

T. Mhamdi, O. Hasan, and S. Tahar. On the Formalization of the Lebesgue Integration Theory in HOL. Interactive Theorem Proving, pages 387--402, 2010.

Digital Library

[16]

B. Milch, B. Marthi, S. Russell, D. Sontag, D. Ong, and A. Kolobov. BLOG: Probabilistic Models with Unknown Objects. In International Joint Conference on Artificial Intelligence, volume 19, 2005.

Digital Library

[17]

O. Nielsen. An Introduction to Integration and Measure Theory. Wiley-Interscience, 1997.

[18]

S. Park, F. Pfenning, and S. Thrun. A Probabilistic Language based upon Sampling Functions. In Principles of Programming Languages, pages 171--182. ACM New York, NY, USA, 2005.

Digital Library

[19]

A. Pfeffer. IBAL: A Probabilistic Rational Programming Language. In International Joint Conference on Artificial Intelligence, 2001.

Digital Library

[20]

N. Ramsey and A. Pfeffer. Stochastic Lambda Calculus and Monads of Probability Distributions. volume 37, pages 154--165. ACM, 2002.

Digital Library

[21]

M. Richardson and P. Domingos. Markov Logic Networks. Machine Learning, 62 (1): 107--136, 2006.

Digital Library

[22]

T. Sato and Y. Kameya. PRISM: A Symbolic-Statistical Modeling Language. In International Joint Conference on Artificial Intelligence, pages 1330--1339, 1997.

Digital Library

[23]

D. Scott. Parametric Statistical Modeling by Minimum Integrated Square Error. Technometrics, 43 (3): 274--285, 2001.

[24]

B. Silverman. Density Estimation for Statistics and Data Analysis. Chapman & Hall/CRC, 1986.

[25]

R. Solovay. A Model of Set-Theory in Which Every Set of Reals is Lebesgue Measurable. Annals of Mathematics, pages 1--56, 1970.

[26]

L. Wasserman. All of Statistics: A Concise Course in Statistical Inference. Springer, 2004.

Digital Library

Cited By

Chen MKatoen JKlinkenberg LWinkler T(2022)Does a Program Yield the Right Distribution?Computer Aided Verification10.1007/978-3-031-13185-1_5(79-101)Online publication date: 7-Aug-2022
https://dl.acm.org/doi/10.1007/978-3-031-13185-1_5
Lee WYu HRival XYang H(2019)Towards verified stochastic variational inference for probabilistic programsProceedings of the ACM on Programming Languages10.1145/33710844:POPL(1-33)Online publication date: 20-Dec-2019
https://dl.acm.org/doi/10.1145/3371084
Bornholt JMytkowicz TMcKinley KBalasubramonian RDavis AAdve S(2014) Uncertain< T > Proceedings of the 19th international conference on Architectural support for programming languages and operating systems10.1145/2541940.2541958(51-66)Online publication date: 24-Feb-2014
https://doi.org/10.1145/2541940.2541958
Show More Cited By

Index Terms

A type theory for probability density functions
1. Mathematics of computing
  1. Probability and statistics
    1. Statistical paradigms
      1. Statistical graphics
2. Theory of computation
  1. Semantics and reasoning
    1. Program reasoning
      1. Program analysis
    2. Program semantics

Recommendations

Deriving a probability density calculator (functional pearl)
ICFP 2016: Proceedings of the 21st ACM SIGPLAN International Conference on Functional Programming

Given an expression that denotes a probability distribution, often we want a corresponding density function, to use in probabilistic inference. Fortunately, the task of finding a density has been automated. It turns out that we can derive a ...
A type theory for probability density functions
POPL '12: Proceedings of the 39th annual ACM SIGPLAN-SIGACT symposium on Principles of programming languages

There has been great interest in creating probabilistic programming languages to simplify the coding of statistical tasks; however, there still does not exist a formal language that simultaneously provides (1) continuous probability distributions, (2) ...
Deriving a probability density calculator (functional pearl)
ICFP '16

Given an expression that denotes a probability distribution, often we want a corresponding density function, to use in probabilistic inference. Fortunately, the task of finding a density has been automated. It turns out that we can derive a ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGPLAN Notices

ACM SIGPLAN Notices Volume 47, Issue 1

POPL '12

January 2012

569 pages

ISSN:0362-1340

EISSN:1558-1160

DOI:10.1145/2103621

Issue’s Table of Contents

POPL '12: Proceedings of the 39th annual ACM SIGPLAN-SIGACT symposium on Principles of programming languages
January 2012
602 pages
ISBN:9781450310833
DOI:10.1145/2103656
General Chair:
John Field
Google, USA
,
Program Chair:
Michael Hicks
University of Maryland, College Park, USA

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 January 2012

Published in SIGPLAN Volume 47, Issue 1

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

25
Total Citations
View Citations
826
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)12

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chen MKatoen JKlinkenberg LWinkler T(2022)Does a Program Yield the Right Distribution?Computer Aided Verification10.1007/978-3-031-13185-1_5(79-101)Online publication date: 7-Aug-2022
https://dl.acm.org/doi/10.1007/978-3-031-13185-1_5
Lee WYu HRival XYang H(2019)Towards verified stochastic variational inference for probabilistic programsProceedings of the ACM on Programming Languages10.1145/33710844:POPL(1-33)Online publication date: 20-Dec-2019
https://dl.acm.org/doi/10.1145/3371084
Bornholt JMytkowicz TMcKinley KBalasubramonian RDavis AAdve S(2014) Uncertain< T > Proceedings of the 19th international conference on Architectural support for programming languages and operating systems10.1145/2541940.2541958(51-66)Online publication date: 24-Feb-2014
https://doi.org/10.1145/2541940.2541958
Bhat SBorgström JGordon ARusso C(2013)Deriving probability density functions from probabilistic functional programsProceedings of the 19th international conference on Tools and Algorithms for the Construction and Analysis of Systems10.1007/978-3-642-36742-7_35(508-522)Online publication date: 16-Mar-2013
https://dl.acm.org/doi/10.1007/978-3-642-36742-7_35
Klinkenberg LBlumenthal CChen MHaase DKatoen J(2024)Exact Bayesian Inference for Loopy Probabilistic Programs using Generating FunctionsProceedings of the ACM on Programming Languages10.1145/36498448:OOPSLA1(923-953)Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1145/3649844
Wang DHoffmann JReps TFreund SYahav E(2021)Sound probabilistic inference via guide typesProceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation10.1145/3453483.3454077(788-803)Online publication date: 19-Jun-2021
https://dl.acm.org/doi/10.1145/3453483.3454077
Mak COng CPaquet HWagner D(2021)Densities of Almost Surely Terminating Probabilistic Programs are Differentiable Almost EverywhereProgramming Languages and Systems10.1007/978-3-030-72019-3_16(432-461)Online publication date: 23-Mar-2021
https://doi.org/10.1007/978-3-030-72019-3_16
Wang DKahn DHoffmann J(2020)Raising expectations: automating expected cost analysis with typesProceedings of the ACM on Programming Languages10.1145/34089924:ICFP(1-31)Online publication date: 3-Aug-2020
https://dl.acm.org/doi/10.1145/3408992
Narayanan PShan C(2020)Symbolic Disintegration with a Variety of Base MeasuresACM Transactions on Programming Languages and Systems10.1145/337420842:2(1-60)Online publication date: 19-May-2020
https://dl.acm.org/doi/10.1145/3374208
Lew ACusumano-Towner MSherman BCarbin MMansinghka V(2019)Trace types and denotational semantics for sound programmable inference in probabilistic languagesProceedings of the ACM on Programming Languages10.1145/33710874:POPL(1-32)Online publication date: 20-Dec-2019
https://dl.acm.org/doi/10.1145/3371087
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents