Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article
Open access

Data-driven lemma synthesis for interactive proofs

Published: 31 October 2022 Publication History

Abstract

Interactive proofs of theorems often require auxiliary helper lemmas to prove the desired theorem. Existing approaches for automatically synthesizing helper lemmas fall into two broad categories. Some approaches are goal-directed, producing lemmas specifically to help a user make progress from a given proof state, but they have limited expressiveness in terms of the lemmas that can be produced. Other approaches are highly expressive, able to generate arbitrary lemmas from a given grammar, but they are completely undirected and hence not amenable to interactive usage.
In this paper, we develop an approach to lemma synthesis that is both goal-directed and expressive. The key novelty is a technique for reducing lemma synthesis to a data-driven program synthesis problem, whereby examples for synthesis are generated from the current proof state. We also describe a technique to systematically introduce new variables for lemma synthesis, as well as techniques for filtering and ranking candidate lemmas for presentation to the user. We implement these ideas in a tool called lfind, which can be run as a Coq tactic. In an evaluation on four benchmark suites, lfind produces useful lemmas in 68% of the cases where a human prover used a lemma to make progress. In these cases lfind synthesizes a lemma that either enables a fully automated proof of the original goal or that matches the human-provided lemma.

References

[1]
1995. Circuits. https://github.com/coq-contribs/circuits
[2]
2003. Coq-of-Ocaml. https://github.com/foobar-land/coq-of-ocaml
[3]
Aws Albarghouthi, Sumit Gulwani, and Zachary Kincaid. 2013. Recursive program synthesis. In International Conference on Computer Aided Verification. 934–950.
[4]
Angello Astorga, P Madhusudan, Shambwaditya Saha, Shiyu Wang, and Tao Xie. 2019. Learning stateful preconditions modulo a test generator. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation. 775–787.
[5]
R Aubin. 1976. Mechanising Structural Induction.
[6]
Kshitij Bansal, Sarah M. Loos, Markus N. Rabe, Christian Szegedy, and Stewart Wilcox. 2019. HOList: An Environment for Machine Learning of Higher-Order Theorem Proving (extended version). CoRR, abs/1904.03241 (2019), arxiv:1904.03241. arxiv:1904.03241
[7]
Jasmin Christian Blanchette, Sascha Böhme, and Lawrence C. Paulson. 2011. Extending Sledgehammer with SMT Solvers. In Automated Deduction – CADE-23, Nikolaj Bjørner and Viorica Sofronie-Stokkermans (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg. 116–130. isbn:978-3-642-22438-6
[8]
Robert S Boyer and J Strother Moore. 1979. A Computational Logic. ACM Monograph Series.
[9]
Alan Bundy, Andrew Stevens, Frank Van Harmelen, Andrew Ireland, and Alan Smaill. 1993. Rippling: A heuristic for guiding inductive proofs. Artificial intelligence, 62, 2 (1993), 185–253.
[10]
Jacqueline Castaing. 1985. How to Facilitate the Proof of Theorems by Using the Induction-matching, and by Generalization. In IJCAI.
[11]
Harsh Raju Chamarthi, Peter C. Dillinger, Matt Kaufmann, and Panagiotis Manolios. 2011. Integrating Testing and Interactive Theorem Proving. In Proceedings 10th International Workshop on the ACL2 Theorem Prover and its Applications, ACL2 2011, Austin, Texas, USA, November 3-4, 2011, David Hardin and Julien Schmaltz (Eds.) (EPTCS, Vol. 70). 4–19. arxiv:1110.4473
[12]
Adam Chlipala. 2013. Certified Programming with Dependent Types: A Pragmatic Introduction to the Coq Proof Assistant. MIT Press.
[13]
Koen Claessen and John Hughes. 2000. QuickCheck: a lightweight tool for random testing of Haskell programs. In 5th ACM SIGPLAN International Conference on Functional Programming (ICFP) (ICFP). ACM, 268–279. http://www.eecs.northwestern.edu/~robby/courses/395-495-2009-fall/quick.pdf
[14]
Koen Claessen, Moa Johansson, Dan Rosén, and Nicholas Smallbone. 2013. Automating Inductive Proofs Using Theory Exploration. In Automated Deduction - CADE-24 - 24th International Conference on Automated Deduction, Lake Placid, NY, USA, June 9-14, 2013. Proceedings, Maria Paola Bonacina (Ed.) (Lecture Notes in Computer Science, Vol. 7898). Springer, 392–406. isbn:978-3-642-38573-5
[15]
Koen Claessen, Nicholas Smallbone, and John Hughes. 2010. QuickSpec: Guessing Formal Specifications Using Testing. In TAP@TOOLS, Gordon Fraser 0001 and Angelo Gargantini (Eds.) (Lecture Notes in Computer Science, Vol. 6143). Springer, 6–21. isbn:978-3-642-13976-5
[16]
Ł ukasz Czajka and Cezary Kaliszyk. 2018. Hammer for Coq: Automation for Dependent Type Theory. Journal of Automated Reasoning, 61, 1 (2018), 01 Jun, 423–453. issn:1573-0670 https://doi.org/10.1007/s10817-018-9458-4
[17]
Leonardo de Moura, Soonho Kong, Jeremy Avigad, Floris van Doorn, and Jakob von Raumer. 2015. The Lean Theorem Prover (System Description). In Automated Deduction - CADE-25, Amy P. Felty and Aart Middeldorp (Eds.). Springer International Publishing, Cham. 378–388. isbn:978-3-319-21401-6
[18]
Lucas Dixon and Jacques Fleuriot. 2003. IsaPlanner: A prototype proof planner in Isabelle. In International Conference on Automated Deduction. 279–283.
[19]
Michael D. Ernst. 2000. Dynamically Discovering Likely Program Invariants. University of Washington Department of Computer Science and Engineering. Seattle, Washington.
[20]
P Ezudheen, Daniel Neider, Deepak D’Souza, Pranav Garg, and P Madhusudan. 2018. Horn-ICE learning for synthesizing invariants and contracts. Proceedings of the ACM on Programming Languages, 2, OOPSLA (2018), 1–25.
[21]
John K. Feser, Swarat Chaudhuri, and Isil Dillig. 2015. Synthesizing Data Structure Transformations from Input-Output Examples. SIGPLAN Not., 50, 6 (2015), jun, 229–239. issn:0362-1340 https://doi.org/10.1145/2813885.2737977
[22]
Jean-Christophe Filliâtre, Hugo Herbelin, Bruno Barras, Bruno Barras, Samuel Boutin, Eduardo Giménez, Samuel Boutin, Gérard Huet, César Muñoz, Cristina Cornes, Cristina Cornes, Judicaël Courant, Judicael Courant, Chetan Murthy, Chetan Murthy, Catherine Parent, Catherine Parent, Christine Paulin-mohring, Christine Paulin-mohring, Amokrane Saibi, Amokrane Saibi, Benjamin Werner, and Benjamin Werner. 1997. The Coq Proof Assistant - Reference Manual Version 6.1.
[23]
Emily First, Yuriy Brun, and Arjun Guha. 2020. TacTok: Semantics-Aware Proof Synthesis. In Object-oriented Programming, Systems, Languages, and Applications.
[24]
Jonathan Frankle, Peter-Michael Osera, David Walker, and Steve Zdancewic. 2016. Example-Directed Synthesis: A Type-Theoretic Interpretation. SIGPLAN Not., 51, 1 (2016), jan, 802–815. issn:0362-1340 https://doi.org/10.1145/2914770.2837629
[25]
Emilio Jesús Gallego Arias, Karl Palmskog, and Vasily Pestun. 2020. SerAPI:Machine-Friendly, Data-Centric Serialization for Coq. https://github.com/ejgallego/coq-serapi
[26]
Pranav Garg, Christof Löding, P Madhusudan, and Daniel Neider. 2014. ICE: A robust framework for learning invariants. In International Conference on Computer Aided Verification. 69–87.
[27]
Pranav Garg, Daniel Neider, P. Madhusudan, and Dan Roth. 2016. Learning invariants using decision trees and implication counterexamples. In Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL 2016, St. Petersburg, FL, USA, January 20 - 22, 2016, Rastislav Bodík and Rupak Majumdar (Eds.). ACM, 499–512. isbn:978-1-4503-3549-2 http://dl.acm.org/citation.cfm?id=2837614
[28]
Thibault Gauthier, Cezary Kaliszyk, and Josef Urban. 2017. TacticToe: Learning to Reason with HOL4 Tactics. In LPAR-21. 21st International Conference on Logic for Programming, Artificial Intelligence and Reasoning, Thomas Eiter and David Sands (Eds.) (EPiC Series in Computing, Vol. 46). EasyChair, 125–143. issn:2398-7340 https://doi.org/10.29007/ntlb
[29]
Jónathan Heras, Ekaterina Komendantskaya, Moa Johansson, and Ewen Maclean. 2013. Proof-Pattern Recognition and Lemma Discovery in ACL2. In Logic for Programming, Artificial Intelligence, and Reasoning - 19th International Conference, LPAR-19, Stellenbosch, South Africa, December 14-19, 2013. Proceedings, Kenneth L. McMillan, Aart Middeldorp, and Andrei Voronkov (Eds.) (Lecture Notes in Computer Science, Vol. 8312). Springer, 389–406. isbn:978-3-642-45220-8
[30]
Jane Thurmann Hesketh. 1992. Using Middle-Out Reasoning to Guide Inductive Theorem Proving. Ph.D. Dissertation. University of Edinburgh.
[31]
Daniel Huang, Prafulla Dhariwal, Dawn Song, and Ilya Sutskever. 2019. GamePad: A Learning Environment for Theorem Proving. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. https://openreview.net/forum?id=r1xwKoR9Y7
[32]
B Hummel. 1990. Generation of induction axioms and generalisation.
[33]
Andrew Ireland and Alan Bundy. 1996. Productive use of failure in inductive proof. In Automated Mathematical Induction. Springer, 79–111.
[34]
Moa Johansson. 2019. Lemma Discovery for Induction - A Survey. In Intelligent Computer Mathematics - 12th International Conference, CICM 2019, Prague, Czech Republic, July 8-12, 2019, Proceedings, Cezary Kaliszyk, Edwin C. Brady, Andrea Kohlhase, and Claudio Sacerdoti Coen (Eds.) (Lecture Notes in Computer Science, Vol. 11617). Springer, 125–139. isbn:978-3-030-23249-8; 978-3-030-23250-4
[35]
Moa Johansson, Lucas Dixon, and Alan Bundy. 2010. Dynamic Rippling, Middle-Out Reasoning and Lemma Discovery. In Verification, Induction, Termination Analysis - Festschrift for Christoph Walther on the Occasion of His 60th Birthday, Simon Siegler and Nathan Wasser (Eds.) (Lecture Notes in Computer Science, Vol. 6463). Springer, 102–116. isbn:978-3-642-17171-0
[36]
Moa Johansson, Lucas Dixon, and Alan Bundy. 2011. Conjecture Synthesis for Inductive Theories. J. Autom. Reason, 47, 3 (2011), 251–289.
[37]
Moa Johansson, Dan Rosén, Nicholas Smallbone, and Koen Claessen. 2014. Hipster: Integrating Theory Exploration in a Proof Assistant. In Intelligent Computer Mathematics - International Conference, CICM 2014, Coimbra, Portugal, July 7-11, 2014. Proceedings, Stephen M. Watt, James H. Davenport, Alan P. Sexton, Petr Sojka, and Josef Urban (Eds.) (Lecture Notes in Computer Science, Vol. 8543). Springer, 108–122. isbn:978-3-319-08433-6
[38]
Cezary Kaliszyk and Josef Urban. 2015. HOL(y)Hammer: Online ATP Service for HOL Light. Mathematics in Computer Science, 9, 1 (2015), 01 Mar, 5–22. issn:1661-8289 https://doi.org/10.1007/s11786-014-0182-0
[39]
Cezary Kaliszyk and Josef Urban. 2015. MizAR 40 for Mizar 40. Journal of Automated Reasoning, 55, 3 (2015), 01 Oct, 245–256. issn:1573-0670 https://doi.org/10.1007/s10817-015-9330-8
[40]
Deepak Kapur and Mahadevan Subramaniam. 1996. Lemma Discovery in Automated Induction. In Automated Deduction - CADE-13, 13th International Conference on Automated Deduction, New Brunswick, NJ, USA, July 30 - August 3, 1996, Proceedings, Michael A. McRobbie and John K. Slaney (Eds.) (Lecture Notes in Computer Science, Vol. 1104). Springer, 538–552. isbn:3-540-61511-3
[41]
Matt Kaufmann and J S. Moore. 1997. An Industrial Strength Theorem Prover for a Logic Based on Common Lisp. IEEE Transactions on Software Engineering, 23, 4 (1997), April, 203–213.
[42]
Justin Lubin, Nick Collins, Cyrus Omar, and Ravi Chugh. 2020. Program Sketching with Live Bidirectional Evaluation. Proc. ACM Program. Lang., 4, ICFP (2020), Article 109, aug, 29 pages. https://doi.org/10.1145/3408991
[43]
Anders Miltner, Adrian Trejo Nuñez, Ana Brendel, Swarat Chaudhuri, and Isil Dillig. 2022. Bottom-up Synthesis of Recursive Functional Programs Using Angelic Execution. Proc. ACM Program. Lang., 6, POPL (2022), Article 21, jan, 29 pages. https://doi.org/10.1145/3498682
[44]
Anders Miltner, Saswat Padhi, David Walker, and Todd Millstein. 2020. Data-driven inference of representation invariants. In Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation.
[45]
Omar Montaño-Rivas, Roy L. McCasland, Lucas Dixon, and Alan Bundy. 2012. Scheme-based theorem discovery and concept invention. Expert Syst. Appl, 39, 2 (2012), 1637–1646.
[46]
Peter-Michael Osera and Steve Zdancewic. 2015. Type-and-example-directed program synthesis. ACM SIGPLAN Notices, 50, 6 (2015), 619–630.
[47]
Saswat Padhi, Rahul Sharma, and Todd Millstein. 2016. Data-driven precondition inference with learned features. In Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation. 42–56.
[48]
Aditya Paliwal, Sarah M. Loos, Markus N. Rabe, Kshitij Bansal, and Christian Szegedy. 2020. Graph Representations for Higher-Order Logic and Theorem Proving. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020. AAAI Press, 2967–2974. https://ojs.aaai.org/index.php/AAAI/article/view/5689
[49]
Zoe Paraskevopoulou, Aaron Eline, and Leonidas Lampropoulos. 2022. Computing correctly with inductive relations. In Proceedings of the 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation. 966–980.
[50]
Zoe Paraskevopoulou, Cătălin Hriţcu, Maxime Dénès, Leonidas Lampropoulos, and Benjamin Pierce. 2015. Foundational Property-Based Testing. 9236, https://doi.org/10.1007/978-3-319-22102-1_22
[51]
Lawrence C. Paulson. 1993. Natural Deduction as Higher-Order Resolution. CoRR, cs.LO/9301104 (1993), arxiv:cs.LO/9301104
[52]
Andrew Reynolds and Viktor Kuncak. 2015. Induction for SMT Solvers. In VMCAI, Deepak D’Souza, Akash Lal, and Kim Guldstrand Larsen (Eds.) (Lecture Notes in Computer Science, Vol. 8931). Springer, 80–98. isbn:978-3-662-46080-1
[53]
Alex Sanchez-Stern, Yousef Alhessi, Lawrence K. Saul, and Sorin Lerner. 2020. Generating correctness proofs with neural networks. In Proceedings of the 4th ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL@PLDI 2020, London, UK, June 15, 2020, Koushik Sen and Mayur Naik (Eds.). ACM, 1–10. https://doi.org/10.1145/3394450.3397466
[54]
Taro Sekiyama, Akifumi Imanishi, and Kohei Suenaga. 2017. Towards Proof Synthesis Guided by Neural Machine Translation for Intuitionistic Propositional Logic. CoRR, abs/1706.06462 (2017), arxiv:1706.06462. arxiv:1706.06462
[55]
Armando Solar-Lezama. 2009. The Sketching Approach to Program Synthesis. In Programming Languages and Systems, 7th Asian Symposium, APLAS 2009, Seoul, Korea, December 14-16, 2009. Proceedings, Zhenjiang Hu (Ed.) (Lecture Notes in Computer Science, Vol. 5904). Springer, 4–13. isbn:978-3-642-10671-2
[56]
William Sonnex, Sophia Drossopoulou, and Susan Eisenbach. 2012. Zeno: An automated prover for properties of recursive data structures. In International Conference on Tools and Algorithms for the Construction and Analysis of Systems. 407–421.
[57]
Daniel Whalen. 2016. Holophrasm: a neural Automated Theorem Prover for higher-order logic. arxiv:1608.02644.
[58]
Kaiyu Yang and Jia Deng. 2019. Learning to Prove Theorems via Interacting with Proof Assistants. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.) (Proceedings of Machine Learning Research, Vol. 97). PMLR, 6984–6994. http://proceedings.mlr.press/v97/yang19a.html
[59]
Weikun Yang, Grigory Fedyukovich, and Aarti Gupta. 2019. Lemma synthesis for automating induction over algebraic data types. In International Conference on Principles and Practice of Constraint Programming. 600–617.
[60]
He Zhu, Stephen Magill, and Suresh Jagannathan. 2018. A data-driven CHC solver. ACM SIGPLAN Notices, 53, 4 (2018), 707–721.

Cited By

View all
  • (2024)CCLemma: E-Graph Guided Lemma Discovery for Inductive Equational ProofsProceedings of the ACM on Programming Languages10.1145/36746538:ICFP(818-844)Online publication date: 15-Aug-2024
  • (2023)Complete First-Order Reasoning for Properties of Functional ProgramsProceedings of the ACM on Programming Languages10.1145/36228357:OOPSLA2(1063-1092)Online publication date: 16-Oct-2023

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Programming Languages
Proceedings of the ACM on Programming Languages  Volume 6, Issue OOPSLA2
October 2022
1932 pages
EISSN:2475-1421
DOI:10.1145/3554307
Issue’s Table of Contents
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 October 2022
Published in PACMPL Volume 6, Issue OOPSLA2

Permissions

Request permissions for this article.

Check for updates

Badges

Author Tags

  1. Data-Driven Synthesis
  2. Interactive Theorem Proving
  3. Lemma Synthesis
  4. Type-Directed Synthesis

Qualifiers

  • Research-article

Funding Sources

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)247
  • Downloads (Last 6 weeks)40
Reflects downloads up to 18 Aug 2024

Other Metrics

Citations

Cited By

View all
  • (2024)CCLemma: E-Graph Guided Lemma Discovery for Inductive Equational ProofsProceedings of the ACM on Programming Languages10.1145/36746538:ICFP(818-844)Online publication date: 15-Aug-2024
  • (2023)Complete First-Order Reasoning for Properties of Functional ProgramsProceedings of the ACM on Programming Languages10.1145/36228357:OOPSLA2(1063-1092)Online publication date: 16-Oct-2023

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Get Access

Login options

Full Access

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media