research-article

Public Access

Program synthesis using conflict-driven learning

Authors:

Osbert Bastani,

Isil DilligAuthors Info & Claims

ACM SIGPLAN Notices, Volume 53, Issue 4

Pages 420 - 435

https://doi.org/10.1145/3296979.3192382

Published: 11 June 2018 Publication History

Abstract

We propose a new conflict-driven program synthesis technique that is capable of learning from past mistakes. Given a spurious program that violates the desired specification, our synthesis algorithm identifies the root cause of the conflict and learns new lemmas that can prevent similar mistakes in the future. Specifically, we introduce the notion of equivalence modulo conflict and show how this idea can be used to learn useful lemmas that allow the synthesizer to prune large parts of the search space. We have implemented a general-purpose CDCL-style program synthesizer called Neo and evaluate it in two different application domains, namely data wrangling in R and functional programming over lists. Our experiments demonstrate the substantial benefits of conflict-driven learning and show that Neo outperforms two state-of-the-art synthesis tools, Morpheus and Deepcoder, that target these respective domains.

Supplementary Material

WEBM File (p420-feng.webm)

Download
128.16 MB

References

[1]

Aws Albarghouthi, Sumit Gulwani, and Zachary Kincaid. 2013. Recursive Program Synthesis. In Proc. International Conference on Computer Aided Verification . Springer, 934–950.

[2]

Rajeev Alur, Rastislav Bodik, Garvit Juniwal, Milo MK Martin, Mukund Raghothaman, Sanjit A Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, and Abhishek Udupa. 2013. Syntax-guided synthesis. In Proc. Formal Methods in Computer-Aided Design. IEEE, 1–8.

[3]

Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2017. Deepcoder: Learning to write programs. In Proc. International Conference on Learning Representations . OpenReview.

[4]

Daniel Le Berre and Anne Parrain. 2010. The Sat4j library, release 2.2. Journal on Satisfiability, Boolean Modeling and Computation (2010), 59–6.

[5]

Armin Biere, Marijn Heule, Hans van Maaren, and Toby Walsh. 2009. Conflict-driven clause learning SAT solvers. Handbook of Satisfiability, Frontiers in Artificial Intelligence and Applications (2009), 131–153.

[6]

Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: An efficient SMT solver. In Proc. Tools and Algorithms for Construction and Analysis of Systems . Springer, 337–340.

Digital Library

[7]

Yu Feng, Ruben Martins, Osbert Bastani, and Isil Dillig. 2018. Neo. http://utopia-group.github.io/neo/ .

[8]

Yu Feng, Ruben Martins, Jacob Van Geffen, Isil Dillig, and Swarat Chaudhuri. 2017. Component-based synthesis of table consolidation and transformation tasks from examples. In Proc. Conference on Programming Language Design and Implementation . ACM, 422–436.

Digital Library

[9]

Yu Feng, Ruben Martins, Yuepeng Wang, Isil Dillig, and Thomas Reps. 2017. Component-Based Synthesis for Complex APIs. In Proc. Symposium on Principles of Programming Languages . ACM, 599–612.

Digital Library

[10]

John K. Feser, Swarat Chaudhuri, and Isil Dillig. 2015. Synthesizing Data Structure Transformations from Input-output Examples. In Proc. Conference on Programming Language Design and Implementation . ACM, 229–239.

Digital Library

[11]

Harald Ganzinger, George Hagen, Robert Nieuwenhuis, Albert Oliveras, and Cesare Tinelli. 2004. DPLL( T): Fast Decision Procedures. In Proc. International Conference on Computer Aided Verification . Springer, 175–188.

[12]

Adrià Gascón, Ashish Tiwari, Brent Carmer, and Umang Mathur. 2017. Look for the Proof to Find the Program: Decorated-Component-Based Program Synthesis. In Proc. International Conference on Computer Aided Verification . Springer, 86–103.

[13]

Sumit Gulwani. 2011. Automating string processing in spreadsheets using input-output examples. In Proc. Symposium on Principles of Programming Languages . ACM, 317–330.

Digital Library

[14]

Sumit Gulwani, Susmit Jha, Ashish Tiwari, and Ramarathnam Venkatesan. 2011. Synthesis of loop-free programs. In Proc. Conference on Programming Language Design and Implementation . ACM, 62–73.

Digital Library

[15]

Tihomir Gvero, Viktor Kuncak, Ivan Kuraj, and Ruzica Piskac. 2013. Complete completion using types and weights. In Proc. Conference on Programming Language Design and Implementation . ACM, 27–38.

Digital Library

[16]

Susmit Jha, Sumit Gulwani, Sanjit A. Seshia, and Ashish Tiwari. 2010. Oracle-guided component-based program synthesis. In Proc. International Conference on Software Engineering . ACM/IEEE, 215–224.

Digital Library

[17]

David Mandelin, Lin Xu, Rastislav Bodík, and Doug Kimelman. 2005. Jungloid mining: helping to navigate the API jungle. In Proc. Conference on Programming Language Design and Implementation . ACM, 48–61.

Digital Library

[18]

Joao Marques-Silva and Karem A. Sakallah. 1999. GRASP: A Search Algorithm for Propositional Satisfiability. IEEE Trans. Computers 48, 5 (1999), 506–521.

Digital Library

[19]

Aditya Menon, Omer Tamuz, Sumit Gulwani, Butler Lampson, and Adam Kalai. 2013. A machine learning framework for programming by example. In Proc. International Conference on Machine Learning. Proceedings of Machine Learning Research, 187–195.

Digital Library

[20]

Arvind Neelakantan, Quoc V Le, Martin Abadi, Andrew McCallum, and Dario Amodei. 2017. Learning a natural language interface with neural programmer. In Proc. International Conference on Learning Representations . OpenReview.

[21]

Arvind Neelakantan, Quoc V Le, and Ilya Sutskever. 2016. Neural programmer: Inducing latent programs with gradient descent. In Proc. International Conference on Learning Representations . OpenReview.

[22]

Peter-Michael Osera and Steve Zdancewic. 2015. Type-and-exampledirected program synthesis. In Proc. Conference on Programming Language Design and Implementation . ACM, 619–630.

Digital Library

[23]

Nadia Polikarpova, Ivan Kuraj, and Armando Solar-Lezama. 2016. Program synthesis from polymorphic refinement types. Proc. Conference on Programming Language Design and Implementation (2016), 522–538.

Digital Library

[24]

Veselin Raychev, Pavol Bielik, Martin Vechev, and Andreas Krause. 2016. Learning programs from noisy data. In Proc. Symposium on Principles of Programming Languages . ACM, 761–774.

Digital Library

[25]

Veselin Raychev, Martin Vechev, and Eran Yahav. 2014. Code completion with statistical language models. In Proc. Conference on Programming Language Design and Implementation . ACM, 419–428.

Digital Library

[26]

Rishabh Singh. 2016. Blinkfill: Semi-supervised programming by example for syntactic string transformations. Proceedings of the VLDB Endowment 9, 10 (2016), 816–827.

Digital Library

[27]

Rishabh Singh and Sumit Gulwani. 2016. Transforming spreadsheet data types using examples. In Proc. Symposium on Principles of Programming Languages . ACM, 343–356.

Digital Library

[28]

Calvin Smith and Aws Albarghouthi. 2016. MapReduce program synthesis. In Proc. Conference on Programming Language Design and Implementation . ACM, 326–340.

Digital Library

[29]

Armando Solar-Lezama. 2008. Program synthesis by sketching. University of California, Berkeley.

Digital Library

[30]

Armando Solar-Lezama. 2009. The Sketching Approach to Program Synthesis. In Proc. Asian Symposium on Programming Languages and Systems . Springer, 4–13.

Digital Library

[31]

Armando Solar-Lezama, Rodric M. Rabbah, Rastislav Bodík, and Kemal Ebcioglu. 2005. Programming by sketching for bit-streaming programs. In Proc. Conference on Programming Language Design and Implementation . ACM, 281–294.

Digital Library

[32]

Armando Solar-Lezama, Liviu Tancau, Rastislav Bodik, Sanjit Seshia, and Vijay Saraswat. 2006. Combinatorial Sketching for Finite Programs. In Proc. International Conference on Architectural Support for Programming Languages and Operating Systems . ACM, 404–415.

Digital Library

[33]

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems . 3104–3112.

Digital Library

[34]

Ashish Tiwari, Adria Gascón, and Bruno Dutertre. 2015. Program synthesis using dual interpretation. In Proc. International Conference on Automated Deduction . Springer, 482–497.

[35]

Abhishek Udupa, Arun Raghavan, Jyotirmoy V Deshmukh, Sela MadorHaim, Milo MK Martin, and Rajeev Alur. 2013. TRANSIT: specifying protocols with concolic snippets. Proc. Conference on Programming Language Design and Implementation (2013), 287–296.

Digital Library

[36]

Chenglong Wang, Alvin Cheung, and Rastislav Bodik. 2017. Synthesizing highly expressive SQL queries from input-output examples. In Proc. Conference on Programming Language Design and Implementation . ACM, 452–466.

Digital Library

[37]

Xinyu Wang, Isil Dillig, and Rishabh Singh. 2017. Synthesis of Data Completion Scripts using Finite Tree Automata. In Proc. International Conference on Object-Oriented Programming, Systems, Languages, and Applications . ACM, 62:1–62:26.

Digital Library

[38]

Xinyu Wang, Isil Dillig, and Rishabh Singh. 2018. Program Synthesis using Abstraction Refinement. In Proc. Symposium on Principles of Programming Languages . ACM, 63:1–63:30.

Digital Library

[39]

Navid Yaghmazadeh, Christian Klinger, Isil Dillig, and Swarat Chaudhuri. 2016. Synthesizing transformations on hierarchically structured data. In Proc. Conference on Programming Language Design and Implementation . ACM, 508–521.

Digital Library

[40]

Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig. 2017. SQLizer: Query Synthesis from Natural Language. In Proc. International Conference on Object-Oriented Programming, Systems, Languages, and Applications . ACM, 63:1–63:26.

Digital Library

[41]

Lintao Zhang, Conor F. Madigan, Matthew W. Moskewicz, and Sharad Malik. 2001. Efficient Conflict Driven Learning in Boolean Satisfiability Solver. In Proc. of International Conference on Computer-Aided Design. IEEE Computer Society, 279–285.

Digital Library

[42]

Sai Zhang and Yuyin Sun. 2013. Automatically synthesizing sql queries from input-output examples. In Proc. International Conference on Automated Software Engineering . IEEE, 224–234.

Digital Library

Cited By

Wei YWang ZLiu JDing YZhang LSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)MagicoderProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694228(52632-52657)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3694228
Hong QAiken A(2024)Recursive Program Synthesis using ParamorphismsProceedings of the ACM on Programming Languages10.1145/36563818:PLDI(102-125)Online publication date: 20-Jun-2024
https://doi.org/10.1145/3656381
Patton NRahmani KMissula MBiswas JDillig I(2024)Programming-by-Demonstration for Long-Horizon Robot TasksProceedings of the ACM on Programming Languages10.1145/36328608:POPL(512-545)Online publication date: 5-Jan-2024
https://dl.acm.org/doi/10.1145/3632860
Show More Cited By

Index Terms

Program synthesis using conflict-driven learning
1. Software and its engineering
  1. Software creation and management
    1. Software development techniques
      1. Automatic programming
  2. Software notations and tools
    1. Context specific languages
      1. Programming by example
2. Theory of computation
  1. Semantics and reasoning
    1. Program reasoning
      1. Program specifications

Recommendations

Program synthesis using conflict-driven learning
PLDI 2018: Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation

We propose a new conflict-driven program synthesis technique that is capable of learning from past mistakes. Given a spurious program that violates the desired specification, our synthesis algorithm identifies the root cause of the conflict and learns ...
Program synthesis using abstraction refinement

We present a new approach to example-guided program synthesis based on counterexample-guided abstraction refinement. Our method uses the abstract semantics of the underlying DSL to find a program P whose abstract behavior satisfies the examples. However, ...
Algorithmic program synthesis: introduction

Program synthesis is a process of producing an executable program from a specification. Algorithmic synthesis produces the program automatically, without an intervention from an expert. While classical compilation falls under the definition of ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGPLAN Notices

ACM SIGPLAN Notices Volume 53, Issue 4

PLDI '18

April 2018

834 pages

ISSN:0362-1340

EISSN:1558-1160

DOI:10.1145/3296979

Editor:
Matthew Fluet
Rodchester Institude of Technology

Issue’s Table of Contents

PLDI 2018: Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation
June 2018
825 pages
ISBN:9781450356985
DOI:10.1145/3192366
General Chair:
Jeffrey S. Foster
University of Maryland at College Park, USA
,
Program Chair:
Dan Grossman
University of Washington, USA

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 June 2018

Published in SIGPLAN Volume 53, Issue 4

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

91
Total Citations
View Citations
2,692
Total Downloads

Downloads (Last 12 months)386
Downloads (Last 6 weeks)47

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wei YWang ZLiu JDing YZhang LSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)MagicoderProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694228(52632-52657)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3694228
Hong QAiken A(2024)Recursive Program Synthesis using ParamorphismsProceedings of the ACM on Programming Languages10.1145/36563818:PLDI(102-125)Online publication date: 20-Jun-2024
https://doi.org/10.1145/3656381
Patton NRahmani KMissula MBiswas JDillig I(2024)Programming-by-Demonstration for Long-Horizon Robot TasksProceedings of the ACM on Programming Languages10.1145/36328608:POPL(512-545)Online publication date: 5-Jan-2024
https://dl.acm.org/doi/10.1145/3632860
Dou QWang HTang HPan HZhang S(2024)Query Reverse Engineering of Pre-deleted Uncorrelated OperatorsData Mining and Big Data10.1007/978-981-97-0844-4_4(45-58)Online publication date: 22-Feb-2024
https://doi.org/10.1007/978-981-97-0844-4_4
Miltner AWang ZChaudhuri SDillig I(2024)Relational Synthesis of Recursive Programs via Constraint Annotated Tree AutomataComputer Aided Verification10.1007/978-3-031-65633-0_3(41-63)Online publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1007/978-3-031-65633-0_3
Li YParsert JPolgreen E(2024)Guiding Enumerative Program Synthesis with Large Language ModelsComputer Aided Verification10.1007/978-3-031-65630-9_15(280-301)Online publication date: 25-Jul-2024
https://doi.org/10.1007/978-3-031-65630-9_15
Thakkar ASands NPetrou GAlur RNaik MRaghothaman M(2023)Mobius: Synthesizing Relational Queries with Recursive and Invented PredicatesProceedings of the ACM on Programming Languages10.1145/36228477:OOPSLA2(1394-1417)Online publication date: 16-Oct-2023
https://dl.acm.org/doi/10.1145/3622847
Wu JWei LJiang YCheung SRen LXu C(2023)Programming by Example Made EasyACM Transactions on Software Engineering and Methodology10.1145/360718533:1(1-36)Online publication date: 7-Jul-2023
https://dl.acm.org/doi/10.1145/3607185
Wang JGupta AWang C(2023)Synthesizing MILP Constraints for Efficient and Robust OptimizationProceedings of the ACM on Programming Languages10.1145/35912987:PLDI(1896-1919)Online publication date: 6-Jun-2023
https://dl.acm.org/doi/10.1145/3591298
Barnaby CChen QSamanta RDillig I(2023)ImageEye: Batch Image Processing using Program SynthesisProceedings of the ACM on Programming Languages10.1145/35912487:PLDI(686-711)Online publication date: 6-Jun-2023
https://dl.acm.org/doi/10.1145/3591248
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents