A Domain-theoretic Approach to Statistical Programming Languages
Abstract
We give a domain-theoretic semantics to a statistical programming language, using the plain old category of dcpos, in contrast to some more sophisticated recent proposals. Remarkably, our monad of minimal valuations is commutative, which allows for program transformations that permute the order of independent random draws, as one would expect. A similar property is not known for Jones and Plotkin’s monad of continuous valuations. Instead of working with true real numbers, we work with exact real arithmetic, providing a bridge towards possible implementations (implementations by themselves are not addressed here). Rather remarkably, we show that restricting ourselves to minimal valuations does not restrict us much: All measures on the real line can be modeled by minimal valuations on the domain Iℝ⊥ of exact real arithmetic. We give three operational semantics for our language, and we show that they are all adequate with respect to the denotational semantics. We also explore quite a few examples to demonstrate that our semantics computes exactly as one would expect and to debunk the myth that a semantics based on continuous maps would not be expressive enough to encode measures with non-compact support using only measures with compact support, or to encode measures via non-continuous density functions, for instance. Our examples also include some useful, non-trivial cases of distributions on higher-order objects.
References
[1]
Samson Abramsky and Achim Jung. 1994. Domain theory. In Semantic Structures, Samson Abramsky, Dov M. Gabbay, and Thomas Stephen Edward Maibaum (Eds.), (Handbook of Logic in Computer Science, Vol. 3). Clarendon Press, Oxford, 1–168.
[2]
Nathanael Leedom Ackerman, Cameron Eric Freer, and Daniel Murphy Roy. 2011. Noncomputable conditional distributions. In Proceedings of the 26th Annual ACM/IEEE Symposium on Logic in Computer Science (LICS’11). IEEE, 107–116. DOI:
[3]
Wolfgang Adamski. 1977. \(\tau\) -smooth Borel measures on topological spaces. Mathem. Nachrich. 78 (1977), 97–107.
[4]
M. Alvarez-Manilla. 2002. Extension of valuations on locally compact sober spaces. Topol. Applic. 124 (2002), 397–433.
[5]
Mauricio Alvarez-Manilla, Abbas Edalat, and Nasser Saheb-Djahromi. 2000. An extension result for continuous valuations. J. Lond. Math. Soc. 61 (2000), 629–640.
[6]
Patrick Billingsley. 1986. Probability and Measure (3rd, 1995 ed.). John Wiley and Sons.
[7]
Richard Bird. 1998. Introduction to Functional Programming Using Haskell (2nd ed.). Prentice-Hall Europe, 98005681433 pages.
[8]
Hans-Joachim Boehm, Robert Cartwright, Mark Riggle, and Michael James O’Donnell. 1986. Exact real arithmetic: A case study in higher order programming. In Proceedings of the ACM Conference on LISP and Functional Programming. ACM Press, Cambridge, MA, 162–173.
[9]
George Edward Pelham Box and Mervin Edgar Muller. 1958. A note on the generation of random normal deviates. Ann. Math. Statist. 29, 2 (1958), 610–611. DOI:
[10]
Ugo Dal Lago and Naohiko Hoshino. 2019. The geometry of Bayesian programming. In Proceedings of the 34th Annual ACM/IEEE Symposium on Logic in Computer Science (LICS’19). IEEE, 1–13. DOI:
[11]
Matthew de Brecht. 2013. Quasi-Polish spaces. Ann. Pure Appl. Logic 164, 3 (2013), 356–381.
[12]
Matthew de Brecht, Jean Goubault-Larrecq, Xiaodong Jia, and Zhenchao Lyu. 2019. Domain-complete and LCS-complete spaces. In Proceedings of the International Symposium on Domain Theory (ISDT’19)(Electronic Notes in Theoretical Computer Science, Vol. 345), Achim Jung, Qingguo Li, Luoshan Xu, and Guo-Qiang Zhang (Eds.). Elsevier Science Publishers, 3–35. DOI:
[13]
Pietro Di Gianantonio and Pier Luca Lanzi. 2004. Algorithms for exact real arithmetic. Electron. Notes Theor. Comput. Sci. 104 (2004), 113–128.
[14]
Abbas Edalat and Martín Hötzel Escardó. 2000. Integration in real PCF. Inf. Comput. 160 (2000), 128–166. DOI:
[15]
Abbas Edalat, Martín Hötzel Escardó, and Peter Potts. 1997. Semantics of exact real arithmetic. In Proceedings of the 12th Annual IEEE Symposium on Logic in Computer Science. IEEE Computer Society, 248–257.
[16]
Abbas Edalat and Sara Negri. 1998. The generalized Riemann integral on locally compact spaces. Topol. Applic. 89, 1–2 (1998), 121–150. DOI:
[17]
Thomas Ehrhard, Michele Pagani, and Christine Tasson. 2018. Measurable cones and stable, measurable functions: A model for probabilistic higher-order programming. In Proceedings of the 45th ACM SIGPLAN Symposium on Principles of Programming Languages (POPL’18). Association for Computing Machinery, 1–28. DOI:
[18]
Yuri Leonidovich Ershov. 1997. The bounded complete hull of an \(\alpha\) -space. Theor. Comput. Sci. 175 (1997), 3–13.
[19]
Martín Hötzel Escardó. 1996. PCF extended with real numbers. Theor. Comput. Sci. 162 (1996), 79–115.
[20]
Martín Hötzel Escardó. 1996. PCF Extended with Real Numbers: A Domain-theoretic Approach to Higher-order Exact Real Number Computation. Ph.D. Dissertation. Imperial College, London, UK.
[21]
Baoying Fan, Yusong Du, Baodian Wei, and Xiao Ma. 2019. On generating exponentially distributed variates by using early rejection. In Proceedings of the 5th IEEE International Conference on Computer and Communications. IEEE, 1307–1311.
[22]
Thomas Ferguson. 1973. Bayesian analysis of some nonparametric problems. Ann. Statist. 1, 2 (1973), 209–230. DOI:
[23]
Marcelo P. Fiore and Gordon D. Plotkin. 1994. An axiomatisation of computationally adequate domain theoretic models of FPC. In Proceedings of the 9th Annual IEEE Symposium on Logic in Computer Science (LICS’94). IEEE, 92–102.
[24]
George Elmer Forsythe. 1972. Von Neumann’s comparison method for random sampling from the normal and other distributions. Math. Comput. 26, 120 (1972), 817–826.
[25]
Stefano Galatolo, Mathieu Hoyrup, and Cristóbal Rojas. 2010. Effective symbolic dynamics, random points, statistical behavior, complexity and entropy. Inf. Comput. 208, 1 (2010), 23–41.
[26]
Ronald Kay Getoor. 1990. Excessive Measures. Birkhäuser, Basel, Switzerland.
[27]
Gerhard Gierz, Karl Heinrich Hofmann, Klaus Keimel, Jimmie D. Lawson, Michael Mislove, and Dana Stewart Scott. 2003. Continuous Lattices and Domains(Encyclopedia of Mathematics and its Applications, Vol. 93). Cambridge University Press, Cambridge.
[28]
Michèle Giry. 1982. A categorical approach to probability theory. Categor. Asp. Topol. Anal. 915 (Jan. 1982), 68–85.
[29]
Noah Goodman, Vikash Mansinghka, Daniel Murphy Roy, Keith Bonawitz, and Joshua Brett Tenenbaum. 2008. Church: A language for generative models. In Proceedings of the 24th Conference on Uncertainty in Artificial Intelligence (UAI’08). AUAI Press, 220–229.
[30]
Noah D. Goodman and Andreas Stuhlmüller. 2014. The Design and Implementation of Probabilistic Programming Languages. Retrieved from http://dippl.org
[31]
Jean Goubault-Larrecq. 2013. Non-Hausdorff Topology and Domain Theory(New Mathematical Monographs, Vol. 22). Cambridge University Press, Cambridge, UK.
[32]
Jean Goubault-Larrecq. 2015. Full abstraction for non-deterministic and probabilistic extensions of PCF I: The Angelic cases. J. Logic Algeb. Meth. Program. 84, 1 (Jan. 2015), 155–184. DOI:
[33]
Jean Goubault-Larrecq. 2019. A probabilistic and non-deterministic call-by-push-value language. In Proceedings of the 34th Annual ACM/IEEE Symposium on Logic in Computer Science (LICS’19), Patricia Bouyer (Ed.). IEEE Press, 1–13. DOI:
[34]
Jean Goubault-Larrecq. 2021. Products and projective limits of continuous valuations on \(T_0\) spaces. Math. Struct. Comput. Sci. 31 (2021), 1–21. DOI:
[35]
Jean Goubault-Larrecq and Xiaodong Jia. 2021. Separating minimal valuations, point-continuous valuations, and continuous valuations. Math. Struct. Comput. Sci. 31, 6 (2021), 614–632. DOI:
[36]
Reinhold Heckmann. 1995. Spaces of Valuations. Technical Report A 09/95. FB 14 Informatik, Universität des Saarlandes, 66041 Saarbrücken, Germany.
[37]
Reinhold Heckmann. 1996. Spaces of valuations. In Papers on General Topology and Applications: Eleventh Summer Conference at the University of Southern Maine(Annals of the New York Academy of Sciences, Vol. 806), Susan Andima, Robert C. Flagg, Gerald Itzkowitz, Ralph Kopperman, Yung Kong, and Prabudh Misra (Eds.). New York Academy of Sciences, New York, 174–200.
[38]
Chris Heunen, Ohad Kammar, Sam Staton, and Hongseok Yang. 2017. A convenient category for higher-order probability theory. In Proceedings of the 32nd Annual ACM/IEEE Symposium on Logic in Computer Science (LICS’17). ACM, 1–12.
[39]
Weng Kin Ho. 2013. Exact real calculator for everyone. In Proceedings of the 1st Asian Technology Conference in Mathematics (ATCM’13), W.-C. Yang, M. Majweski, T. Alwis, and I. K. Rana (Eds.). Mathematics and Technology, LLC, 1–15.
[40]
Daniel Huang, Greg Morrisett, and Bas Spitters. 2020. Application of computable distributions to the semantics of probabilistic programs. In Foundations of Probabilistic Programming, Gilles Barthe, Joost-Pieter Katoen, and Alexandra Silva (Eds.). Cambridge University Press, Cambridge, UK, 75–120. DOI:
[41]
John Isbell. 1981. Product spaces in locales. Proc. Am. Math. Soc. 81, 1 (1981), 116–118.
[42]
Xiaodong Jia, Bert Lindenhovius, Michael Mislove, and Vladimir Zamdzhiev. 2021. Commutative monads for probabilistic programming languages. In Proceedings of the 36th Annual ACM/IEEE Symposium on Logic in Computer Science (LICS’21). IEEE Press, 1–14. DOI:
[43]
Xiaodong Jia and Michael Mislove. 2022. Completing simple valuations in K-categories. Topol. Applic. 318 (2022), 108192. DOI:
[44]
Claire Jones. 1990. Probabilistic Non-determinism. Ph.D. Dissertation. University of Edinburgh, Edinburgh.
[45]
Claire Jones and Gordon Plotkin. 1989. A probabilistic powerdomain of evaluations. In Proceedings of the 4th Annual Symposium on Logic in Computer Science. IEEE Press, 186–195.
[46]
Achim Jung and Regina Tix. 1998. The troublesome probabilistic powerdomain. In Proceedings of the 3rd Workshop on Computation and Approximation(Electronic Notes in Theoretical Computer Science, Vol. 13), Abbas Edalat, Achim Jung, Klaus Keimel, and Marta Kwiatkowska (Eds.). Springer-Verlag, Berlin, 70–91. 23 pages.
[47]
Olav Kallenberg. 2002. Foundations of Modern Probability (2nd ed.). Springer-Verlag, New York.
[48]
Ohad Kammar and Dylan McDermott. 2018. Factorisation systems for logical relations and monadic lifting in type-and-effect system semantics. Electron. Notes. Theor. Comput. Sci. 341 (2018), 239–260. DOI:
[49]
Klaus Keimel and Jimmie D. Lawson. 2008. D-completions and the \(d\) -topology. Ann. Pure Appl. Logic 159 (2008), 292–306. DOI:
[50]
Olaf Kirch. 1993. Bereiche und Bewertungen. Master’s thesis. Technische Hochschule Darmstadt.
[51]
Anders Kock. 1971. Bilinearity and Cartesian closed monads. Math. Scand. 29 (1971), 161–174.
[52]
Anders Kock. 2012. Commutative monads as a theory of distributions. Theor. Applic. Categ. 26, 4 (2012), 97–131.
[53]
Branimir Lambov. 2005. Complexity in a type-1 framework for computable analysis. In CiE 2005: New Computational Paradigms (ILLC Technical Notes (X) Series), S. B. Cooper, B. Löwe, and L. Torenvliet (Eds.). Institute for Logic, Language, and Computation, Springer, Amsterdam, 133–145.
[54]
Jimmie D. Lawson. 1997. Spaces of maximal points. Math. Struct. Comput. Sci. 7 (1997), 543–555.
[55]
David R. Lester. 1992. Vuillemin’s exact real arithmetic. In Proceedings of the Functional Programming Workshop (BCS Workshops in Computing), Rogardt Heldal, Carsten Kehler Holst, and Philip Wadler (Eds.). Springer Verlag, 225–238.
[56]
Ernest G. Manes. 1976. Algebraic Theories(Graduate Texts in Mathematics, Vol. 26). Springer-Verlag, New York.
[57]
Vikash K. Mansinghka, Daniel Selsam, and Yura N. Perov. 2014. Venture: A Higher-order Probabilistic Programming Platform with Programmable Inference. Retrieved from http://arxiv.org/abs/1404.0099
[58]
José Raymundo Marcial-Romero. 2004. Semantics of a Sequential Language for Exact Real-number Computation. Ph.D. Dissertation. University of Birmingham.
[59]
Eugenio Moggi. 1989. Computational lambda-calculus and monads. In Logic in Computer Science. IEEE Computer Society Press, 14–23.
[60]
Eugenio Moggi. 1991. Notions of computations and monads. Inf. Comput. 93, 1 (1991), 55–92.
[61]
Praveen Narayanan, Jacques Carette, Wren Romano, Chung-Chieh Shan, and Robert Zinkov. 2016. Probabilistic inference by program transformation in Hakaru. In Proceedings of the 13th International Symposium on Functional and Logic Programming (FOOPS’16). Springer, 62–79.
[62]
Sungwoo Park, Frank Pfenning, and Sebastian Thrun. 2008. A probabilistic language based on sampling functions. ACM Trans. Program. Lang. Syst. 31, 1 (2008), 1–46. DOI:
[63]
Andrew M. Pitts. 1993. Relational Properties of Domains. Technical Report 321. Cambridge University Computer Laboratory.
[64]
David Plume. 1998. A Calculator for Exact Real Number Computation. Retrieved from http://www.dcs.ed.ac.uk/home/mhe/plume/index.html
[65]
Daniel Murphy Roy. 2011. Computability, Inference and Modeling in Probabilistic Programming. Ph.D. Dissertation. MIT.
[66]
Nasser Saheb-Djahromi. 1980. CPO’s of measures for nondeterminism. Theor. Comput. Sci. 12 (1980), 19–37.
[67]
René L. Schilling. 2017. Measures, Integrals and Martingales (2nd ed.). Cambridge University Press.
[68]
Matthias Schröder and Alex Simpson. 2006. Representing probability measures using probabilistic processes. J. Complex. 22, 6 (2006), 768–782.
[69]
Holger Schulz. 2000. Type two theory of effectivity and real PCF. Electron. Notes Theor. Comput. Sci. 35 (2000), 191–203.
[70]
Dana Stewart Scott. 1969. A Type Theoretic Alternative to ISWIM, CUCH, OWHY. Manuscript, University of Oxford.
[71]
Dana Stewart Scott. 1993. A type theoretic alternative to ISWIM, CUCH, OWHY. Theor. Comput. Sci. 121 (1993), 411–440.
[72]
Chung-Chieh Shan and Norman Ramsey. 2017. Exact Bayesian inference by symbolic disintegration. In Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming Languages (POPL’17). Association for Computing Machinery, 130–144.
[73]
Michael Sharpe. 1988. General Theory of Markov Processes. Academic Press, Cambridge.
[74]
Alex Simpson. 1998. Lazy functional algorithms for exact real functionals. In Mathematical Foundations of Computer Science 1998(Lecture Notes in Computer Science, Vol. 1450). Springer Verlag, 456–464.
[75]
Sam Staton. 2017. Commutative semantics for probabilistic programming. In Proceedings of the 26th European Symposium on Programming (ESOP’17) (Lecture Notes in Computer Science, Vol. 10201), Hongseok Yang (Ed.). Springer Verlag, 855–879. DOI:
[76]
Sam Staton, Hongseok Yang, Chris Heunen, Ohad Kammar, and Frank Wood. 2016. Semantics for probabilistic programming: Higher-order functions, continuous distributions, and soft constraints. In Proceedings of the 31st Annual ACM/IEEE Symposium on Logic in Computer Science (LICS’16). ACM, New York, 525–534.
[77]
Thomas Streicher. 2002. Mathematical Foundations of Functional Programming. Retrieved from http://www.mathematik.tu-darmstadt.de/streicher/MGFP/mgfp.ps.gz
[78]
Philipp Sünderhauf. 1995. A faithful computational model of the real numbers. Theor. Comput. Sci. 151, 1 (1995), 277–294. DOI:
[79]
Regina Tix. 1995. Stetige Bewertungen auf topologischen Räumen. Master’s thesis. Technische Hochschule Darmstadt.
[80]
Leonida Tonelli. 1909. Sull’integrazione per parti. Atti della Accademia Nazionale dei Lincei 18, 2 (1909), 246–253.
[81]
Matthijs Vákár, Ohad Kammar, and Sam Staton. 2019. A domain theory for statistical probabilistic programming. In Proceedings of the 46th ACM SIGPLAN Symposium on Principles of Programming Languages (POPL’19). 1–29. DOI:
[82]
Steven J. Vickers. 2011. A Monad of Valuation Locales. Retrieved from https://www.cs.bham.ac.uk/sjv/Riesz.pdf
[83]
Klaus Weihrauch. 1987. Computability. Springer Verlag, Berlin.
[84]
Klaus Weihrauch. 1999. Computability on the probability measures on the Borel sets of the unit interval. Theor. Comput. Sci. 219 (1999), 421–437.
[85]
Frank Wood, Jan Willem van de Meent, and Vikash Mansinghka. 2014. A new approach to probabilistic programming inference. In Proceedings of the 17th International Conference on Artificial Intelligence and Statistics (AISTATS’14)(Proceedings of Machine Learning Research, Vol. 33), Samuel Kaski and Jukka Corander (Eds.). PMLR, 1024–1032.
Index Terms
- A Domain-theoretic Approach to Statistical Programming Languages
Recommendations
A domain theory for statistical probabilistic programming
We give an adequate denotational semantics for languages with recursive higher-order types, continuous probability distributions, and soft constraints. These are expressive languages for building Bayesian models of the kinds used in computational ...
Observationally-induced Effect Monads: Upper and Lower Powerspace Constructions
Alex Simpson has suggested to use an observationally-induced approach towards modelling computational effects in denotational semantics. The principal idea is that a single observation algebra is used for defining the computational type structure. He ...
A Set-Theoretic Representation of Algebraic L-domains
Theory and Applications of Models of ComputationAbstractIn this paper, we aim to establish a concrete representation, as a family of sets, for every algebraic L-domain. We generalize the notion of a topped algebraic intersection structure to a locally algebraic intersection structure. Just as topped ...
Comments
Information & Contributors
Information
Published In
October 2023
388 pages
ISSN:0004-5411
EISSN:1557-735X
DOI:10.1145/3627674
- Editor:
- Venkatesan Guruswami
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 October 2023
Online AM: 19 August 2023
Accepted: 26 July 2023
Revised: 06 July 2023
Received: 15 October 2021
Published in JACM Volume 70, Issue 5
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- NSFC
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 477Total Downloads
- Downloads (Last 12 months)477
- Downloads (Last 6 weeks)18
Reflects downloads up to 09 Aug 2024
Other Metrics
Citations
Cited By
View allView Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in