Smart choices and the selection monad
Article No.: 81, Pages 1 - 14
Abstract
Describing systems in terms of choices and their resulting costs and rewards promises to free algorithm designers and programmers from specifying how to make those choices. In implementations, the choices can be realized by optimization or machine-learning methods.
We study this approach from a programming-language perspective. We define a small language that supports decision-making abstraction, rewards, and probabilities. We give a globally optimizing operational semantics, and, using the selection monad for decision-making, three denotational semantics with auxiliary monads for reward and probability; the three model various correlations between returned values and expected rewards. We show the two kinds of semantics coincide by proving adequacy theorems; we show that observational equivalence is characterized by semantic equality (at basic types) by proving full abstraction theorems; and we discuss program equations.
References
[1]
Martín Abadi and Gordon D. Plotkin. Smart choices and the selection monad. CoRR, abs/2007.08926, 2020.
[2]
Philip Amortila, Doina Precup, Prakash Panangaden, and Marc G. Bellemare. A distributional analysis of sampling-based reinforcement learning algorithms. In Silvia Chiappa and Roberto Calandra, editors, The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26--28 August 2020, Online [Palermo, Sicily, Italy], volume 108 of Proceedings of Machine Learning Research, pages 4357--4366. PMLR, 2020.
[3]
Richard Bellman. Dynamic Programming. Princeton University Press, Princeton, 1957.
[4]
Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul A. Szerlip, Paul Horsfall, and Noah D. Goodman. Pyro: Deep universal probabilistic programming. J. Mach. Learn. Res., 20:28:1--28:6, 2019.
[5]
Richard Blute, Thomas Ehrhard, and Christine Tasson. A convenient differential category. arXiv preprint arXiv:1006.3140, 2010.
[6]
Joe Bolt, Jules Hedges, and Philipp Zahn. Sequential games and nondeterministic selection functions. CoRR, abs/1811.06810, 2018.
[7]
C. Boutilier, R. Reiter, M. Soutchanski, and S. Thrun. Decision-theoretic, high-level robot programming in the situation calculus. In Proceedings of the AAAI National Conference on Artificial Intelligence. AAAI, 2000.
[8]
David Budden, Matteo Hessel, John Quan, and Steven Kapturowski. RLax: Reinforcement Learning in JAX, 2020. https://github.com/deepmind/rlax.
[9]
Vladimir Bychkovsky. Spiral: Self-tuning services via realtime machine learning, 2018. Blog post here.
[10]
Victor Carbune, Thierry Coppey, Alexander N. Daryin, Thomas Deselaers, Nikhil Sarda, and Jay Yagnik. Predicted variables in programming. CoRR, abs/1810.00619, 2018.
[11]
Kai-Wei Chang, He He, Stéphane Ross, Hal Daumé III, and John Langford. A credit assignment compiler for joint prediction. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5--10, 2016, Barcelona, Spain, pages 1705--1713, 2016.
[12]
Thomas Ehrhard and Laurent Regnier. The differential lambda-calculus. Theo. Comp. Sci., 309(1--3):1--41, 2003.
[13]
Martín Escardó. Constructive decidability of classical continuity. Math. Struct. Comput. Sci., 25(7):1578--1589, 2015.
[14]
Martin Escardó and Paulo Oliva. Sequential games and optimal strategies. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 467(2130):1519--1545, 2011.
[15]
Martín Escardó and Paulo Oliva. The Herbrand functional interpretation of the double negation shift. J. Symb. Log., 82(2):590--607, 2017.
[16]
Martín Hötzel Escardó and Paulo Oliva. Selection functions, bar recursion and backward induction. Math. Struct. Comput. Sci., 20(2):127--168, 2010.
[17]
Martín Hötzel Escardó and Paulo Oliva. The Peirce translation. Ann. Pure Appl. Log., 163(6):681--692, 2012.
[18]
Martín Hötzel Escardó, Paulo Oliva, and Thomas Powell. System T and the product of selection functions. In Marc Bezem, editor, Computer Science Logic, 25th International Workshop / 20th Annual Conference of the EACSL, CSL 2011, September 12--15, 2011, Bergen, Norway, Proceedings, volume 12 of LIPIcs, pages 233--247. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2011.
[19]
Matthias Felleisen and Daniel P. Friedman. Control operators, the secd-machine, and the λ-calculus. In Martin Wirsing, editor, Formal Description of Programming Concepts - III: Proceedings of the IFIP TC 2/WG 2.2 Working Conference on Formal Description of Programming Concepts - III, Ebberup, Denmark, 25--28 August 1986, pages 193--222. North-Holland, 1987.
[20]
Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes. In UAI, volume 4, pages 162--169, 2004.
[21]
Andrzej Filinski. Representing monads. In Proceedings of the 21st ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL '94, page 446--457. Association for Computing Machinery, 1994.
[22]
Jules Hedges. The selection monad as a CPS transformation. CoRR, abs/1503.06061, 2015.
[23]
Ohad Kammar, Sam Lindley, and Nicolas Oury. Handlers in action. ACM SIGPLAN Notices, 48(9):145--158, 2013.
[24]
Klaus Keimel and Gordon D. Plotkin. Mixed powerdomains for probability and nondeterminism. Log. Methods Comput. Sci., 13(1), 2017.
[25]
Andreas Kriegl and Peter W Michor. The convenient setting of global analysis, volume 53. American Mathematical Soc., 1997.
[26]
Paul Blain Levy, John Power, and Hayo Thielecke. Modelling environments in call-by-value programming languages. Inf. Comput., 185(2):182--210, 2003.
[27]
Aliaume Lopez and Alex Simpson. Basic operational preorders for algebraic effects in general, and for combined probability and nondeterminism in particular. In Dan R. Ghica and Achim Jung, editors, 27th EACSL Annual Conference on Computer Science Logic, CSL 2018, September 4--7, 2018, Birmingham, UK, volume 119 of LIPIcs, pages 29:1--29:17. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2018.
[28]
Eugenio Moggi. Computational lambda-calculus and monads. In Proceedings of the Fourth Annual Symposium on Logic in Computer Science (LICS '89), Pacific Grove, California, USA, June 5--8, 1989, pages 14--23. IEEE Computer Society, 1989.
[29]
Dave Moore and Maria I. Gorinova. Effect handling for composable program transformations in edward2. CoRR, abs/1811.06150, 2018.
[30]
Gordon Plotkin and John Power. Adequacy for algebraic effects. In Furio Honsell and Marino Miculan, editors, Foundations of Software Science and Computation Structures, pages 1--24, Berlin, Heidelberg, 2001. Springer Berlin Heidelberg.
[31]
Gordon D. Plotkin and John Power. Algebraic operations and generic effects. Applied Categorical Structures, 11(1):69--94, 2003.
[32]
Gordon D. Plotkin and Matija Pretnar. Handling algebraic effects. Log. Methods Comput. Sci., 9(4), 2013.
[33]
Scott Sanner. Relational dynamic influence diagram language (rddl): Language description, 2011. Official language of the uncertainty track of the Seventh International Planning Competition.
[34]
Marshall Harvey Stone. Postulates for the barycentric calculus. Annali di Matematica Pura ed Applicata, 29(1):25--30, 1949.
[35]
Tim Vieira, Matthew Francis-Landau, Nathaniel Wesley Filardo, Farzad Khorasani, and Jason Eisner. Dyna: Toward a self-optimizing declarative language for machine learning applications. In Proceedings of the 1st ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2017, page 8--17. Association for Computing Machinery, 2017.
[36]
Li-yao Xia, Yannick Zakowski, Paul He, Chung-Kil Hur, Gregory Malecha, Benjamin C. Pierce, and Steve Zdancewic. Interaction trees: representing recursive and impure programs in coq. Proc. ACM Program. Lang., 4(POPL):51:1--51:32, 2020.
Index Terms
- Smart choices and the selection monad
Index terms have been assigned to the content through auto-classification.
Recommendations
An equational notion of lifting monad
Category theory and computer scienceWe introduce the notion of an equational lifting monad: a commutative strong monad satisfying one additional equation (valid for monads arising from partial map classifiers). We prove that any equational lifting monad has a representation by a partial ...
The Coinductive Resumption Monad
Resumptions appear in many forms as a convenient abstraction, such as in semantics of concurrency and as a programming pattern. In this paper we introduce generalised resumptions in a category-theoretic, coalgebraic context and show their basic ...
Identifying All Preorders on the Subdistribution Monad
The countable valuation monad, the countable distribution monad, and the countable subdistribution monad are often used in the coalgebraic treatment of discrete probabilistic transition systems. We identify preorders on them using a technique based on ...
Comments
Information & Contributors
Information
Published In
Sponsors
In-Cooperation
- EACSL: European Association for Computer Science Logic
- IEEE-CS: Computer Society
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 24 November 2021
Check for updates
Qualifiers
- Research-article
Conference
LICS '21
Sponsor:
LICS '21: 36th Annual ACM/IEEE Symposium on Logic in Computer Science
June 29 - July 2, 2021
Rome, Italy
Acceptance Rates
Overall Acceptance Rate 215 of 622 submissions, 35%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 31Total Downloads
- Downloads (Last 12 months)8
- Downloads (Last 6 weeks)3
Reflects downloads up to 17 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in