research-article

Open access

Augmented example-based synthesis using relational perturbation properties

Authors:

Sasa Misailovic,

Roopsha SamantaAuthors Info & Claims

Proceedings of the ACM on Programming Languages, Volume 4, Issue POPL

Article No.: 56, Pages 1 - 24

https://doi.org/10.1145/3371124

Published: 20 December 2019 Publication History

Abstract

Example-based specifications for program synthesis are inherently ambiguous and may cause synthesizers to generate programs that do not exhibit intended behavior on unseen inputs. Existing synthesis techniques attempt to address this problem by either placing a domain-specific syntactic bias on the hypothesis space or heavily relying on user feedback to help resolve ambiguity.

We present a new framework to address the ambiguity/generalizability problem in example-based synthesis. The key feature of our framework is that it places a semantic bias on the hypothesis space using "relational perturbation properties" that relate the perturbation/change in a program output to the perturbation/change in a program input. An example of such a property is permutation invariance: the program output does not change when the elements of the program input (array) are permuted. The framework is portable across multiple domains and synthesizers and is based on two core steps: (1) automatically augment the set of user-provided examples by "applying" relational perturbation properties and (2) use a generic example-based synthesizer to generate a program consistent with the augmented set of examples. Our framework can be instantiated with three different user interfaces, with varying degrees of user engagement to help infer relevant relational perturbation properties. This includes an interface in which the user only provides examples and our framework automatically infers relevant properties. We implement our framework in a tool SKETCHAX specialized to the SKETCH synthesizer and demonstrate that SKETCHAX is effective in significantly boosting the performance of SKETCH for all three user interfaces.

Supplementary Material

WEBM File (a56-an.webm)

Download
79.45 MB

References

[1]

2019. Syntax-Guided Synthesis Competition. https://sygus.org/ .

[2]

2019. Sketch Benchmarks Repositories. https://bitbucket.org/gatoatigrado/sketch-frontend/src .

[3]

Rajeev Alur, Rastislav Bodík, Garvit Juniwal, Milo M. K. Martin, Mukund Raghothaman, Sanjit A. Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, and Abhishek Udupa. 2013. Syntax-Guided Synthesis. In Formal Methods in Computer-Aided Design (FMCAD) . 1–8.

[4]

Matej Balog, Alexander L. Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2017. DeepCoder: Learning to Write Programs. In International Conference on Learning Representations (ICLR).

[5]

Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, T. H. Tse, and Zhi Quan Zhou. 2018. Metamorphic Testing: A Review of Challenges and Opportunities. Comput. Surveys 51, 1 (2018), 4:1–4:27.

[6]

Alessandro Cimatti, Anders Franzén, Alberto Griggio, Roberto Sebastiani, and Cristian Stenico. 2010. Satisfiability Modulo the Theory of Costs: Foundations and Applications. In Tools and Algorithms for the Construction and Analysis of Systems (TACAS) . 99–113.

[7]

Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. 2017. RobustFill: Neural Program Learning under Noisy I/O. In International Conference on Machine Learning (ICML). 990–998.

[8]

Dana Drachsler-Cohen, Sharon Shoham, and Eran Yahav. 2017. Synthesis with Abstract Examples. In Computer Aided Verification (CAV) . 254–278.

[9]

John K. Feser, Swarat Chaudhuri, and Isil Dillig. 2015. Synthesizing Data Structure Transformations from Input-Output Examples. In Programming Language Design and Implementation (PLDI). 229–239.

[10]

Sumit Gulwani. 2011. Automating String Processing in Spreadsheets using Input-Output Examples. In Principles of Programming Languages (POPL) . 317–330.

[11]

Sumit Gulwani. 2016. Programming by Examples: Applications, Algorithms, and Ambiguity Resolution. In International Joint Conference on Automated Reasoning (IJCAR) . 9–14.

Digital Library

[12]

Sumit Gulwani, William R. Harris, and Rishabh Singh. 2012. Spreadsheet Data Manipulation using Examples. Commun. ACM 55, 8 (2012), 97–105.

Digital Library

[13]

Sumit Gulwani, Oleksandr Polozov, and Rishabh Singh. 2017. Program Synthesis. Foundations and Trends in Programming Languages 4, 1-2 (2017), 1–119.

[14]

Jeevana Priya Inala and Rishabh Singh. 2017. WebRelate: Integrating Web Data with Spreadsheets using Examples. Proceedings of the ACM on Programming Languages (PACMPL) 2, POPL (2017), 2:1–2:28.

Digital Library

[15]

Susmit Jha, Sumit Gulwani, Sanjit A. Seshia, and Ashish Tiwari. 2010. Oracle-guided Component-based Program Synthesis. In International Conference on Software Engineering (ICSE). 215–224.

[16]

Upulee Kanewala, James M Bieman, and Asa Ben-Hur. 2016. Predicting Metamorphic Relations for Testing Scientific Software: A Machine Learning Approach using Graph Kernels. Software Testing, Verification & Reliability 26, 3 (2016), 245–269.

[17]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems (NIPS) 25. 1097–1105.

[18]

Alan Leung, John Sarracino, and Sorin Lerner. 2015. Interactive Parser Synthesis by Example. In Programming Language Design and Implementation (PLDI) . 565–574.

[19]

Henry Lieberman. 2000. Programming by Example: Introduction. Commun. ACM 43, 3 (2000), 72–74.

Digital Library

[20]

Henry Lieberman. 2001. Your Wish is My Command: Programming by Example. Morgan Kaufmann.

[21]

Mikaël Mayer, Gustavo Soares, Maxim Grechkin, Vu Le, Mark Marron, Oleksandr Polozov, Rishabh Singh, Benjamin G. Zorn, and Sumit Gulwani. 2015. User Interaction Models for Disambiguation in Programming by Example. In Symposium on User Interface Software & Technology (UIST) . 291–301.

Digital Library

[22]

Emilio Parisotto, Abdel-rahman Mohamed, Rishabh Singh, Lihong Li, Dengyong Zhou, and Pushmeet Kohli. 2017. NeuroSymbolic Program Synthesis. In International Conference on Learning Representations (ICLR).

[23]

Hila Peleg, Sharon Shoham, and Eran Yahav. 2018. Programming Not Only by Example. In International Conference on Software Engineering (ICSE) . 1114–1124.

Digital Library

[24]

Veselin Raychev, Pavol Bielik, Martin Vechev, and Andreas Krause. 2016. Learning Programs from Noisy Data. In Principles of Programming Languages (POPL) . 761–774.

[25]

Patrice Y. Simard, David Steinkraus, and John C. Platt. 2003. Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis. In International Conference on Document Analysis and Recognition (ICDAR). 958–962.

[26]

Rishabh Singh. 2016. BlinkFill: Semi-supervised Programming by Example for Syntactic String Transformations. Proceedings of the VLDB Endowment 9, 10 (2016), 816–827.

Digital Library

[27]

Rishabh Singh and Sumit Gulwani. 2012. Synthesizing Number Transformations from Input-Output Examples. In Computer Aided Verification (CAV) . 634–651.

[28]

Rishabh Singh and Sumit Gulwani. 2015. Predicting a Correct Program in Programming by Example. In Computer Aided Verificationi (CAV) . 398–414.

[29]

Rishabh Singh and Armando Solar-Lezama. 2011. Synthesizing Data Structure Manipulations from Storyboards. In Foundations of Software Engineeringi (ESEC/FSE) . 289–299.

[30]

Calvin Smith and Aws Albarghouthi. 2016. MapReduce Program Synthesis. In Programming Language Design and Implementation (PLDI) . 326–340.

[31]

Armando Solar-Lezama, Liviu Tancau, Rastislav Bodik, Sanjit Seshia, and Vijay Saraswat. 2006. Combinatorial Sketching for Finite Programs. In Architectural Support for Programming Languages and Operating Systems (ASPLOS). 404–415.

[32]

Chenglong Wang, Alvin Cheung, and Rastislav Bodík. 2017. Synthesizing Highly Expressive SQL Queries from Input-Output Examples. In Programming Language Design and Implementation (PLDI). 452–466.

[33]

Yuepeng Wang, Xinyu Wang, and Isil Dillig. 2018. Relational Program Synthesis. Proceedings of the ACM on Programming Languages (PACMPL) 2, OOPSLA (2018), 155:1–155:27.

Digital Library

[34]

Jie Zhang, Junjie Chen, Dan Hao, Yingfei Xiong, Bing Xie, Lu Zhang, and Hong Mei. 2014. Search-based Inference of Polynomial Metamorphic Relations. In Automated Software Engineering (ASE). 701–712.

Cited By

Wang BBaluta TKolluri ASaxena PSpinellis DGousios GChechik MDi Penta M(2021)SynGuar: guaranteeing generalization in programming by exampleProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3468264.3468621(677-689)Online publication date: 20-Aug-2021
https://dl.acm.org/doi/10.1145/3468264.3468621
Wang JSung CRaghothaman MWang C(2021)Data-Driven Synthesis of Provably Sound Side Channel AnalysesProceedings of the 43rd International Conference on Software Engineering10.1109/ICSE43902.2021.00079(810-822)Online publication date: 22-May-2021
https://dl.acm.org/doi/10.1109/ICSE43902.2021.00079
Gao XBarke SRadhakrishna ASoares GGulwani SLeung ANagappan NTiwari A(2020)Feedback-driven semi-supervised synthesis of program transformationsProceedings of the ACM on Programming Languages10.1145/34282874:OOPSLA(1-30)Online publication date: 13-Nov-2020
https://doi.org/10.1145/3428287
Show More Cited By

Index Terms

Augmented example-based synthesis using relational perturbation properties
1. Software and its engineering
  1. Software notations and tools
    1. General programming languages
  2. Software organization and properties
    1. Software functional properties
      1. Correctness

Recommendations

Bottom-up synthesis of recursive functional programs using angelic execution

We present a novel bottom-up method for the synthesis of functional recursive programs. While bottom-up synthesis techniques can work better than top-down methods in certain settings, there is no prior technique for synthesizing recursive programs from ...
Can reactive synthesis and syntax-guided synthesis be friends?
SPLASH Companion 2021: Companion Proceedings of the 2021 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity

While reactive synthesis and syntax-guided synthesis (SyGuS) have seen enormous progress in recent years, combining the two approaches has remained a challenge. In this work, we present the synthesis of reactive programs from Temporal Stream Logic ...
Can reactive synthesis and syntax-guided synthesis be friends?
PLDI 2022: Proceedings of the 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation

While reactive synthesis and syntax-guided synthesis (SyGuS) have seen enormous progress in recent years, combining the two approaches has remained a challenge. In this work, we present the synthesis of reactive programs from Temporal Stream Logic ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Programming Languages

Proceedings of the ACM on Programming Languages Volume 4, Issue POPL

January 2020

1984 pages

EISSN:2475-1421

DOI:10.1145/3377388

Issue’s Table of Contents

Copyright © 2019 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 December 2019

Published in PACMPL Volume 4, Issue POPL

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
812
Total Downloads

Downloads (Last 12 months)96
Downloads (Last 6 weeks)15

Reflects downloads up to 26 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang BBaluta TKolluri ASaxena PSpinellis DGousios GChechik MDi Penta M(2021)SynGuar: guaranteeing generalization in programming by exampleProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3468264.3468621(677-689)Online publication date: 20-Aug-2021
https://dl.acm.org/doi/10.1145/3468264.3468621
Wang JSung CRaghothaman MWang C(2021)Data-Driven Synthesis of Provably Sound Side Channel AnalysesProceedings of the 43rd International Conference on Software Engineering10.1109/ICSE43902.2021.00079(810-822)Online publication date: 22-May-2021
https://dl.acm.org/doi/10.1109/ICSE43902.2021.00079
Gao XBarke SRadhakrishna ASoares GGulwani SLeung ANagappan NTiwari A(2020)Feedback-driven semi-supervised synthesis of program transformationsProceedings of the ACM on Programming Languages10.1145/34282874:OOPSLA(1-30)Online publication date: 13-Nov-2020
https://doi.org/10.1145/3428287
Gavran IDarulova EMajumdar R(2020)Interactive synthesis of temporal specifications from examples and natural languageProceedings of the ACM on Programming Languages10.1145/34282694:OOPSLA(1-26)Online publication date: 13-Nov-2020
https://dl.acm.org/doi/10.1145/3428269

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Figures

Tables

Media

View Issue’s Table of Contents