Compiling Probabilistic Programs for Variable Elimination with Information Flow
Authors: 
Jianlin Li, Eric Wang, 
Yizhou ZhangAuthors Info & Claims
Jianlin Li, Eric Wang, Yizhou ZhangAuthors Info & ClaimsArticle No.: 218, Pages 1755 - 1780
Abstract
A key promise of probabilistic programming is the ability to specify rich models using an expressive program- ming language. However, the expressive power that makes probabilistic programming languages enticing also poses challenges to inference, so much so that specialized approaches to inference ban language features such as recursion. We present an approach to variable elimination and marginal inference for probabilistic programs featuring bounded recursion, discrete distributions, and sometimes continuous distributions. A compiler eliminates probabilistic side effects, using a novel information-flow type system to factorize probabilistic computations and hoist independent subcomputations out of sums or integrals. For a broad class of recursive programs with dynamically recurring substructure, the compiler effectively decomposes a global marginal-inference problem, which may otherwise be intractable, into tractable subproblems. We prove the compilation correct by showing that it preserves denotational semantics. Experiments show that the compiled programs subsume widely used PTIME algorithms for recursive models and that the compilation time scales with the size of the inference problems. As a separate contribution, we develop a denotational, logical-relations model of information-flow types in the novel measure-theoretic setting of probabilistic programming; we use it to prove noninterference and consequently the correctness of variable elimination.
References
[1]
Martín Abadi, Anindya Banerjee, Nevin Heintze, and Jon G. Riecke. 1999. A core calculus of dependency. In ACM SIGPLAN Symp. on Principles of Programming Languages (POPL). https://doi.org/10.1145/292540.292555
[2]
Torben Amtoft and Anindya Banerjee. 2020. A theory of slicing for imperative probabilistic programs. ACM Tran. on Programming Languages and Systems (TOPLAS) 42, 2 (April 2020 ). https://doi.org/10.1145/3372895
[3]
Eric Atkinson, Charles Yuan, Guillaume Baudart, Louis Mandel, and Michael Carbin. 2022. Semi-symbolic inference for eficient streaming probabilistic programming. Proc. of the ACM on Programming Languages (PACMPL) 6, OOPSLA2 (Oct. 2022 ). https://doi.org/10.1145/3563347
[4]
Martin Avanzini, Gilles Barthe, and Ugo Dal Lago. 2021. On continuation-passing transformations and expected cost analysis. Proc. of the ACM on Programming Languages (PACMPL) 5, ICFP (Aug. 2021 ). https://doi.org/10.1145/3473592
[5]
J. K. Baker. 1979. Trainable grammars for speech recognition. The Journal of the Acoustical Society of America 65, S1 ( 1979 ). https://doi.org/10.1121/1.2017061
[6]
Jialu Bao, Simon Docherty, Justin Hsu, and Alexandra Silva. 2021. A bunched logic for conditional independence. In ACM/IEEE Symp. on Logic In Computer Science (LICS). https://doi.org/10.1109/LICS52264. 2021.9470712
[7]
Gilles Barthe, Justin Hsu, and Kevin Liao. 2019. A probabilistic separation logic. Proc. of the ACM on Programming Languages (PACMPL) 4, POPL (Dec. 2019 ). https://doi.org/10.1145/3371123
[8]
Guillaume Baudart, Louis Mandel, Eric Atkinson, Benjamin Sherman, Marc Pouzet, and Michael Carbin. 2020. Reactive probabilistic programming. In ACM SIGPLAN Conf. on Programming Language Design and Implementation (PLDI). https://doi.org/10.1145/3385412.3386009
[9]
Richard E. Bellman. 1957. Dynamic Programming. Princeton University Press.
[10]
Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D. Goodman. 2019. Pyro: Deep universal probabilistic programming. Journal of Machine Learning Research (JMLR) 20, 1 ( 2019 ). arXiv: 1810.09538
[11]
Johannes Borgström, Ugo Dal Lago, Andrew D. Gordon, and Marcin Szymczak. 2016. A lambda-calculus foundation for universal probabilistic programming. In ACM SIGPLAN Conf. on Functional Programming (ICFP). https://doi.org/10. 1145/2951913.2951942
[12]
Johannes Borgström, Andrew D. Gordon, Michael Greenberg, James Margetson, and Jurgen Van Gael. 2011. Measure transformer semantics for Bayesian machine learning. In European Symp. on Programming (ESOP). https://doi.org/10. 1007/978-3-642-19718-5_5
[13]
Bob Carpenter, Andrew Gelman, Matthew D. Hofman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. 2017. Stan: A probabilistic programming language. Journal of Statistical Software 76, 1 ( 2017 ). https://doi.org/10.18637/jss.v076.i01
[14]
Mark Chavira and Adnan Darwiche. 2008. On probabilistic inference by weighted model counting. Artificial Intelligence 172, 6 ( 2008 ). https://doi.org/10.1016/j.artint. 2007. 11.002
[15]
David Chiang, Colin McDonald, and Chung-chieh Shan. 2023. Exact recursive probabilistic programming. Proc. of the ACM on Programming Languages (PACMPL) 7, OOPSLA1 (April 2023 ). https://doi.org/10.1145/3586050
[16]
David Chiang and Darcey Riley. 2020. Factor graph grammars. In Conf. on Neural Information Processing Systems (NeurIPS). arXiv: 2010.12048
[17]
Guillaume Claret, Sriram K. Rajamani, Aditya V. Nori, Andrew D. Gordon, and Johannes Borgström. 2013. Bayesian inference using data flow analysis. In Proc. of the 9th Joint Meeting of the European Software Engineering Conf. and the
[18]
Oliver Danvy and Andrzex Filinski. 1992. Representing control: a study of the CPS transformation. Mathematical Structures in Computer Science 2, 4 ( 1992 ). https://doi.org/10.1017/S0960129500001535
[19]
Adnan Darwiche. 2009. Modeling and Reasoning with Bayesian Networks. Cambridge University Press. https: //doi.org/10.1017/CBO9780511811357
[20]
Dorothy E. Denning and Peter J. Denning. 1977. Certification of programs for secure information flow. ACM (CACM) 20, 7 ( July 1977 ). https://doi.org/10.1145/359636.359712
[21]
Dorothy Elizabeth Robling Denning. 1982. Cryptography and Data Security. Addison-Wesley Reading.
[22]
Simon Duane, A.D. Kennedy, Brian J. Pendleton, and Duncan Roweth. 1987. Hybrid Monte Carlo. Physics Letters B 195, 2 ( 1987 ). https://doi.org/10.1016/ 0370-2693 ( 87 ) 91197-X
[23]
Roy Frostig, Matthew Johnson, and Chris Leary. 2018. Compiling machine learning programs via highllevel tracing. https://mlsys.org/Conferences/doc/2018/146.pdf
[24]
Timon Gehr, Sasa Misailovic, and Martin Vechev. 2016. PSI: Exact symbolic inference for probabilistic programs. In Int'l Conf. on Computer Aided Verification (CAV). https://doi.org/10.1007/978-3-319-41528-4_4
[25]
Timon Gehr, Samuel Stefen, and Martin Vechev. 2020. PSI: Exact inference for higher-order probabilistic programs. In ACM SIGPLAN Conf. on Programming Language Design and Implementation (PLDI). https://doi.org/10.1145/3385412. 3386006
[26]
J. A. Goguen and J. Meseguer. 1982. Security policies and security models. In IEEE Symp. on Security and Privacy. https://doi.org/10.1109/SP. 1982.10014
[27]
Noah D. Goodman and Andreas Stuhlmüller. 2014. The design and implementation of probabilistic programming languages. http://dippl.org.
[28]
Maria I. Gorinova, Andrew D. Gordon, Charles Sutton, and Matthijs Vákár. 2021. Conditional independence by typing. ACM Tran. on Programming Languages and Systems (TOPLAS) 44, 1 (Dec. 2021 ). https://doi.org/10.1145/3490421 arXiv: 2010.11887
[29]
Chris Heunen, Ohad Kammar, Sam Staton, and Hongseok Yang. 2017. A convenient category for higher-order probability theory. In ACM/IEEE Symp. on Logic In Computer Science (LICS). https://doi.org/10.5555/3329995.3330072 arXiv: 1701.02547
[30]
Steven Holtzen, Guy Van den Broeck, and Todd Millstein. 2020. Scaling exact inference for discrete probabilistic programs. Proc. of the ACM on Programming Languages (PACMPL) 4, OOPSLA (Nov. 2020 ). https://doi.org/10.1145/ 3428208 arXiv: 2005.09089
[31]
Matthew D. Homan and Andrew Gelman. 2014. The no-U-turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research (JMLR) 15, 1 (Jan. 2014 ). https://doi.org/10.5555/2627435.2638586
[32]
Chung-Kil Hur, Aditya V. Nori, Sriram K. Rajamani, and Selva Samuel. 2014. Slicing probabilistic programs. In ACM SIGPLAN Conf. on Programming Language Design and Implementation (PLDI). https://doi.org/10.1145/2594291.2594303
[33]
Daphne Koller, David McAllester, and Avi Pfefer. 1997. Efective Bayesian inference for stochastic programs. In Proc. of the 14th National Conf. on Artificial Intelligence and the 9thConf. on Innovative Applications of Artificial Intelligence (AAAI/IAAI).
[34]
Satoshi Kura. 2023. Higher-order weakest precondition transformers via a CPS transformation. ( 2023 ). arXiv: 2301.09997
[35]
Alexander K. Lew, Matin Ghavamizadeh, Martin C. Rinard, and Vikash K. Mansinghka. 2023. Probabilistic programming with stochastic probabilities. Proc. of the ACM on Programming Languages (PACMPL) 7, PLDI ( June 2023 ). https: //doi.org/10.1145/3591290
[36]
Jianlin Li, Leni Ven, Pengyuan Shi, and Yizhou Zhang. 2023. Type-preserving, dependence-aware guide generation for sound, efective amortized probabilistic inference. Proc. of the ACM on Programming Languages (PACMPL) POPL. https://doi.org/10.1145/3571243
[37]
Jianlin Li, Eric Wang, and Yizhou Zhang. 2024. Compiling probabilistic programs for variable elimination with information flow (artifact). https://doi.org/10.5281/zenodo.10951893
[38]
Jianlin Li, Eric Wang, and Yizhou Zhang. 2024. Compiling Probabilistic Programs for Variable Elimination with Information Flow (Extended Version). Technical Report CS-2024-03. School of Computer Science, University of Waterloo. https://cs.uwaterloo.ca/sites/ca.computer-science/files/uploads/files/cs-2024-03.pdf
[39]
John M. Li, Amal Ahmed, and Steven Holtzen. 2023. Lilac: A modal separation logic for conditional probability. Proc. of the ACM on Programming Languages (PACMPL) 7, PLDI ( June 2023 ). https://doi.org/10.1145/3591226 arXiv: 2304.01339
[40]
Christopher Manning and Hinrich Schütze. 1999. Foundations of Statistical Natural Language Processing. MIT Press.
[41]
Andrew McCallum, Karl Schultz, and Sameer Singh. 2009. FACTORIE: Probabilistic programming via imperatively defined factor graphs. In Conf. on Neural Information Processing Systems (NIPS).
[42]
Lawrence Murray, Daniel Lundén, Jan Kudlicka, David Broman, and Thomas Schön. 2018. Delayed sampling and automatic Rao-Blackwellization of probabilistic programs. In Int'l Conf. on Artificial Intelligence and Statistics (AISTATS). arXiv:1708.07787
[43]
Andrew C. Myers. 1999. JFlow: Practical mostly-static information flow control. In ACM SIGPLAN Symp. on Principles of Programming Languages (POPL). https://doi.org/10.1145/292540.292561
[44]
Andrew C. Myers and Barbara Liskov. 1997. A decentralized model for information flow control. In Proc. of the 16th ACM Symp. on Operating Systems Principles (SOSP). https://doi.org/10.1145/268998.266669
[45]
Praveen Narayanan, Jacques Carette, Wren Romano, Chung-chieh Shan, and Robert Zinkov. 2016. Probabilistic inference by program transformation in Hakaru (system description). In Functional and Logic Programming. https: //doi.org/10.1007/978-3-319-29604-3_5
[46]
Radford M. Neal. 2001. Annealed importance sampling. Statistics and Computing 11 ( 2001 ). https://doi.org/10.1023/A: 1008923215028
[47]
Fritz Obermeyer, Eli Bingham, Martin Jankowiak, Neeraj Pradhan, Justin Chiu, Alexander Rush, and Noah Goodman. 2019. Tensor variable elimination for plated factor graphs. In Int'l Conf. on Machine Learning (ICML). arXiv: 1902.03210
[48]
Judea Pearl. 1988. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann. https://doi.org/10.1016/B978-0-08-051489-5. 50001-1
[49]
Avi Pfefer. [n.d.]. IBAL: An expressive, functional probabilistic modeling language. ([n. d.]). https://www.cs.tufts.edu/ ~nr/cs257/archive/avi-pfefer/ibal-journal.pdf
[50]
Avi Pfefer. 2001. IBAL: A probabilistic rational programming language. In Int'l Joint Conf. on Artificial Intelligence (IJCAI).
[51]
Avi Pfefer. 2007. The design and implementation of IBAL: A general-purpose probabilistic language. In Introduction to Statistical Relational Learning. The MIT Press. https://doi.org/10.7551/mitpress/7432.003.0016
[52]
Avi Pfefer. 2016. Practical Probabilistic Programming. Manning Publications.
[53]
Du Phan, Neeraj Pradhan, and Martin Jankowiak. 2019. Composable efects for flexible and accelerated probabilistic programming in NumPyro. ( 2019 ). arXiv: 1912.11554
[54]
Lawrence R. Rabiner and Biing-Hwang Juang. 1986. An introduction to hidden Markov models. IEEE ASSP Magazine 3, 1 ( 1986 ). https://doi.org/10.1109/MASSP. 1986.1165342
[55]
Vineet Rajani and Deepak Garg. 2018. Types for information flow control: Labeling granularity and semantic models. In IEEE Computer Security Foundations Symp. (CSF). https://doi.org/10.1109/CSF. 2018.00024
[56]
Christian P. Robert and George Casella. 1999. Monte Carlo Statistical Methods. Springer. https://doi.org/10.1007/978-1-4757-4145-2
[57]
Fredrik Ronquist, Jan Kudlicka, Viktor Senderov, Johannes Borgström, Nicolas Lartillot, Daniel Lundén, Lawrence Murray, Thomas B. Schön, and David Broman. 2021. Universal probabilistic programming ofers a powerful approach to statistical phylogenetics. Communications Biology 4, 1 ( 2021 ). https://doi.org/10.1038/s42003-021-01753-7
[58]
Feras A. Saad, Martin C. Rinard, and Vikash K. Mansinghka. 2021. SPPL: Probabilistic programming with fast exact symbolic inference. In ACM SIGPLAN Conf. on Programming Language Design and Implementation (PLDI). https://doi.org/10.1145/3453483.3454078
[59]
Andrei Sabelfeld and Andrew C. Myers. 2003. Language-based information-flow security. IEEE Journal on Selected Areas in Communications 21, 1 ( 2003 ). https://doi.org/10.1109/JSAC. 2002.806121
[60]
Andrei Sabelfeld and David Sands. 2001. A PER model of secure information flow in sequential programs. Higher-Order and Symbolic Computation 14 ( 2001 ). https://doi.org/10.1023/A:1011553200337
[61]
Adam Ścibior, Ohad Kammar, Matthijs Vákár, Sam Staton, Hongseok Yang, Yufei Cai, Klaus Ostermann, Sean K. Moss, Chris Heunen, and Zoubin Ghahramani. 2017. Denotational validation of higher-order Bayesian inference. Proc. of the ACM on Programming Languages (PACMPL) 2, POPL (Dec. 2017 ). https://doi.org/10.1145/3158148
[62]
Andreas Stuhlmüller and Noah D. Goodman. 2012. A dynamic programming algorithm for inference in recursive probabilistic programs. ( 2012 ). arXiv: 1206.3555
[63]
Matthijs Vákár, Ohad Kammar, and Sam Staton. 2019. A domain theory for statistical probabilistic programming. Proc. of the ACM on Programming Languages (PACMPL) 3, POPL ( 2019 ). https://doi.org/10.1145/3290349
[64]
Rajan Walia, Praveen Narayanan, Jacques Carette, Sam Tobin-Hochstadt, and Chung-chieh Shan. 2019. From high-level inference algorithms to eficient code. Proc. of the ACM on Programming Languages (PACMPL) 3, ICFP ( July 2019 ). https://doi.org/10.1145/3341702
[65]
Mitchell Wand, Ryan Culpepper, Theophilos Giannakopoulos, and Andrew Cobb. 2018. Contextual equivalence for a probabilistic language with continuous random variables and recursion. Proc. of the ACM on Programming Languages (PACMPL) 2, ICFP ( 2018 ). https://doi.org/10.1145/3236782
[66]
Di Wang, Jan Hofmann, and Thomas Reps. 2021. Sound probabilistic inference via guide types. In ACM SIGPLAN Conf. on Programming Language Design and Implementation (PLDI). arXiv:2104.03598
[67]
Nevin Lianwen Zhang and David Poole. 1994. A simple approach to Bayesian network computations. In Proc. of the 10th Canadian Conference on Artificial Intelligence.
[68]
Yizhou Zhang and Nada Amin. 2022. Reasoning about “reasoning about reasoning”: Semantics and contextual equivalence for probabilistic programs with nested queries and recursion. Proc. of the ACM on Programming Languages (PACMPL) 6, POPL ( 2022 ). https://doi.org/10.1145/3498677
Index Terms
- Compiling Probabilistic Programs for Variable Elimination with Information Flow
Recommendations
Compiling Markov chain Monte Carlo algorithms for probabilistic modeling
PLDI 2017: Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and ImplementationThe problem of probabilistic modeling and inference, at a high-level, can be viewed as constructing a (model, query, inference) tuple, where an inference algorithm implements a query on a model. Notably, the derivation of inference algorithms can be a ...
Compiling Markov chain Monte Carlo algorithms for probabilistic modeling
PLDI '17The problem of probabilistic modeling and inference, at a high-level, can be viewed as constructing a (model, query, inference) tuple, where an inference algorithm implements a query on a model. Notably, the derivation of inference algorithms can be a ...
Compiling Stan to generative probabilistic languages and extension to deep probabilistic programming
PLDI 2021: Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and ImplementationStan is a probabilistic programming language that is popular in the statistics community, with a high-level syntax for expressing probabilistic models. Stan differs by nature from generative probabilistic programming languages like Church, Anglican, or ...
Comments
Information & Contributors
Information
Published In
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 20 June 2024
Published in PACMPL Volume 8, Issue PLDI
Check for updates
Badges
Author Tags
Qualifiers
- Research-article
Funding Sources
- NSERC
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 255Total Downloads
- Downloads (Last 12 months)255
- Downloads (Last 6 weeks)73
Reflects downloads up to 22 Sep 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in