research-article

Open access

Relational verification using reinforcement learning

Authors:

Osbert Bastani,

Isil DilligAuthors Info & Claims

Proceedings of the ACM on Programming Languages, Volume 3, Issue OOPSLA

Article No.: 141, Pages 1 - 30

https://doi.org/10.1145/3360567

Published: 10 October 2019 Publication History

Abstract

Relational verification aims to prove properties that relate a pair of programs or two different runs of the same program. While relational properties (e.g., equivalence, non-interference) can be verified by reducing them to standard safety, there are typically many possible reduction strategies, only some of which result in successful automated verification. Motivated by this problem, we propose a novel relational verification algorithm that learns useful reduction strategies using reinforcement learning. Specifically, we show how to formulate relational verification as a Markov Decision Process (MDP) and use reinforcement learning to synthesize an optimal policy for the underlying MDP. The learned policy is then used to guide the search for a successful verification strategy. We have implemented this approach in a tool called Coeus and evaluate it on two benchmark suites. Our evaluation shows that Coeus solves significantly more problems within a given time limit compared to multiple baselines, including two state-of-the-art relational verification tools.

Supplementary Material

a141-chen (a141-chen.webm)

Presentation at OOPSLA '19

Download
101.27 MB

References

[1]

Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2016. Deepcoder: Learning to write programs. In ICLR.

[2]

Gilles Barthe, Juan Manuel Crespo, and César Kunz. 2011. Relational verification using product programs. In International Symposium on Formal Methods. Springer, 200–214.

[3]

Gilles Barthe, Juan Manuel Crespo, and César Kunz. 2016. Product programs and relational program logics. Journal of Logical and Algebraic Methods in Programming 85, 5 (2016), 847–859.

[4]

Gilles Barthe, Pedro R D’Argenio, and Tamara Rezk. 2004. Secure information flow by self-composition. In Computer Security Foundations Workshop, 2004. Proceedings. 17th IEEE. IEEE, 100–114.

[5]

Gilles Barthe, Boris Köpf, Federico Olmedo, and Santiago Zanella Béguelin. 2012. Probabilistic relational reasoning for differential privacy. In ACM SIGPLAN Notices, Vol. 47. ACM, 97–110.

Digital Library

[6]

Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. 2018a. Verifiable reinforcement learning via policy extraction. In NIPS.

[7]

Osbert Bastani, Rahul Sharma, Alex Aiken, and Percy Liang. 2017. Synthesizing program input grammars. In PLDI, Vol. 52. ACM, 95–110.

Digital Library

[8]

Osbert Bastani, Rahul Sharma, Alex Aiken, and Percy Liang. 2018b. Active learning of points-to specifications. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation. ACM, 678–692.

Digital Library

[9]

Nels E Beckman and Aditya V Nori. 2011. Probabilistic, modular and scalable inference of typestate specifications. In PLDI, Vol. 46. ACM, 211–221.

Digital Library

[10]

Nick Benton. 2004. Simple relational correctness proofs for static analyses and program transformations. In ACM SIGPLAN Notices, Vol. 39. ACM, 14–25.

Digital Library

[11]

Pavol Bielik, Veselin Raychev, and Martin Vechev. 2016. PHOG: probabilistic model for code. In International Conference on Machine Learning. 2933–2942.

[12]

Pavol Bielik, Veselin Raychev, and Martin Vechev. 2017. Learning a static analyzer from data. In International Conference on Computer Aided Verification. Springer, 233–253.

[13]

Nikolaj Bjørner, Arie Gurfinkel, Ken McMillan, and Andrey Rybalchenko. 2015. Horn clause solvers for program verification. In Fields of Logic and Computation II. Springer, 24–51.

[14]

Jia Chen, Yu Feng, and Isil Dillig. 2017. Precise Detection of Side-Channel Vulnerabilities using Quantitative Cartesian Hoare Logic. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. ACM, 875–890.

Digital Library

[15]

Lonnie Chrisman. 1992. Reinforcement learning with perceptual aliasing: The perceptual distinctions approach. In AAAI, Vol. 1992. Citeseer, 183–188.

[16]

Lazaro Clapp, Osbert Bastani, Saswat Anand, and Alex Aiken. 2016. Minimizing GUI event traces. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 422–434.

Digital Library

[17]

Michael R. Clarkson and Fred B. Schneider. 2010. Hyperproperties. Journal of Computer Security 18, 6 (Sept. 2010), 1157–1210.

Digital Library

[18]

Emanuele De Angelis, Fabio Fioravanti, Alberto Pettorossi, and Maurizio Proietti. 2016a. Horn Clause Transformation for Program Verification. Technical Report.

[19]

Emanuele De Angelis, Fabio Fioravanti, Alberto Pettorossi, and Maurizio Proietti. 2016b. Relational verification through Horn clause transformation. In International Static Analysis Symposium. Springer, 147–169.

[20]

Leonardo de Moura and Nikolaj Bjørner. 2008. Z3: An Efficient SMT Solver. In Tools and Algorithms for the Construction and Analysis of Systems. Springer Berlin Heidelberg, Berlin, Heidelberg, 337–340.

Digital Library

[21]

Marco Eilers, Peter Müller, and Samuel Hitz. 2018. Modular Product Programs. In European Symposium on Programming. Springer, 502–529.

[22]

Dennis Felsing, Sarah Grebing, Vladimir Klebanov, Philipp Rümmer, and Mattias Ulbrich. 2014. Automating Regression Verification. In Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering. 349–360.

Digital Library

[23]

Yu Feng, Ruben Martins, Osbert Bastani, and Isil Dillig. 2018. Program synthesis using conflict-driven learning. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation. ACM, 420–435.

Digital Library

[24]

Yu Feng, Ruben Martins, Jacob Van Geffen, Isil Dillig, and Swarat Chaudhuri. 2017. Component-based synthesis of table consolidation and transformation tasks from examples. In PLDI, Vol. 52. ACM, 422–436.

[25]

Cormac Flanagan and K. Rustan M. Leino. 2001. Houdini, an Annotation Assistant for ESC/Java. In Proceedings of the International Symposium of Formal Methods Europe on Formal Methods for Increasing Software Productivity (FME ’01). Springer-Verlag, Berlin, Heidelberg, 500–517.

[26]

Patrice Godefroid, Hila Peleg, and Rishabh Singh. 2017. Learn&fuzz: Machine learning for input fuzzing. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering. IEEE Press, 50–59.

Digital Library

[27]

Joseph A Goguen and José Meseguer. 1982. Security policies and security models. In Security and Privacy, 1982 IEEE Symposium on. IEEE, 11–11.

[28]

Xiaoxiao Guo, Satinder Singh, Honglak Lee, Richard L Lewis, and Xiaoshi Wang. 2014. Deep learning for real-time Atari game play using offline Monte-Carlo tree search planning. In Advances in neural information processing systems. 3338–3346.

[29]

Arie Gurfinkel, Temesghen Kahsai, and Jorge A. Navas. 2015. SeaHorn: A Framework for Verifying C Programs (Competition Contribution). In Tools and Algorithms for the Construction and Analysis of Systems, Christel Baier and Cesare Tinelli (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 447–450.

[30]

Stefan Heule, Eric Schkufza, Rahul Sharma, and Alex Aiken. 2016. Stratified synthesis: automatically learning the x86-64 instruction set. In PLDI, Vol. 51. ACM, 237–250.

Digital Library

[31]

Geoffrey Irving, Christian Szegedy, Alexander A Alemi, Niklas Eén, François Chollet, and Josef Urban. 2016. Deepmath-deep sequence models for premise selection. In Advances in Neural Information Processing Systems. 2235–2243.

[32]

Ashwin Kalyan, Abhishek Mohta, Oleksandr Polozov, Dhruv Batra, Prateek Jain, and Sumit Gulwani. 2018. Neural-Guided Deductive Search for Real-Time Program Synthesis from Examples. In ICLR.

[33]

Anvesh Komuravelli, Arie Gurfinkel, and Sagar Chaki. 2016. SMT-based Model Checking for Recursive Programs. Formal Methods in System Design 48, 3 (June 2016), 175–205.

Digital Library

[34]

Ted Kremenek, Paul Twohey, Godmar Back, Andrew Ng, and Dawson Engler. 2006. From uncertainty to belief: Inferring the specification within. In Proceedings of the 7th symposium on Operating systems design and implementation. 161–176.

[35]

Shuvendu K Lahiri, Chris Hawblitzel, Ming Kawaguchi, and Henrique Rebêlo. 2012. Symdiff: A language-agnostic semantic diff tool for imperative programs. In International Conference on Computer Aided Verification. Springer, 712–717.

Digital Library

[36]

Shuvendu K Lahiri, Kenneth L McMillan, Rahul Sharma, and Chris Hawblitzel. 2013. Differential assertion checking. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. ACM, 345–355.

Digital Library

[37]

Woosuk Lee, Kihong Heo, Rajeev Alur, and Mayur Naik. 2018. Accelerating search-based program synthesis using learned probabilistic models. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation. ACM, 436–449.

Digital Library

[38]

Xavier Leroy. 2009. Formal Verification of a Realistic Compiler. Commun. ACM 52, 7 (July 2009), 107–115.

Digital Library

[39]

Percy Liang, Omer Tripp, and Mayur Naik. 2011. Learning minimal abstractions. In POPL, Vol. 46. ACM, 31–42.

Digital Library

[40]

Ben Liblit, Mayur Naik, Alice X Zheng, Alex Aiken, and Michael I Jordan. 2005. Scalable statistical bug isolation. 40, 6 (2005), 15–26.

[41]

Benjamin Livshits, Aditya V Nori, Sriram K Rajamani, and Anindya Banerjee. 2009. Merlin: specification inference for explicit information flow problems, Vol. 44. ACM, 75–86.

[42]

Ravi Mangal, Xin Zhang, Aditya V Nori, and Mayur Naik. 2015. A user-guided approach to program analysis. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. ACM, 462–473.

Digital Library

[43]

R Andrew McCallum. 1993. Overcoming incomplete perception with utile distinction memory. In Proceedings of the Tenth International Conference on Machine Learning. 190–196.

[44]

William H Montgomery and Sergey Levine. 2016. Guided policy search via approximate mirror descent. In Advances in Neural Information Processing Systems. 4008–4016.

[45]

Dmitry Mordvinov and Grigory Fedyukovich. 2017. Synchronizing constrained Horn clauses. LPAR, EPiC Series in Computing. EasyChair (2017).

[46]

Saswat Padhi, Rahul Sharma, and Todd Millstein. 2016. Data-driven precondition inference with learned features. In PLDI, Vol. 51. ACM, 42–56.

Digital Library

[47]

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NIPS-W.

[48]

Amir Pnueli, Michael Siegel, and Eli Singerman. 1998. Translation Validation. In Proceedings of the 4th International Conference on Tools and Algorithms for Construction and Analysis of Systems (TACAS ’98). Springer-Verlag, Berlin, Heidelberg, 151–166.

Digital Library

[49]

Dan Quinlan and Chunhua Liao. 2011. The ROSE Source-to-Source Compiler Infrastructure. In Cetus Users and Compiler Infrastructure Workshop, in conjunction with PACT 2011.

[50]

Mukund Raghothaman, Sulekha Kulkarni, Kihong Heo, and Mayur Naik. 2018. User-guided program reasoning using Bayesian inference. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation. ACM, 722–735.

Digital Library

[51]

Veselin Raychev, Pavol Bielik, and Martin Vechev. 2016a. Probabilistic model for code with decision trees. In OOPSLA, Vol. 51. ACM, 731–747.

Digital Library

[52]

Veselin Raychev, Pavol Bielik, Martin Vechev, and Andreas Krause. 2016b. Learning programs from noisy data. In POPL, Vol. 51. ACM, 761–774.

Digital Library

[53]

Veselin Raychev, Martin Vechev, and Andreas Krause. 2015. Predicting program properties from big code. In POPL, Vol. 50. ACM, 111–124.

Digital Library

[54]

Veselin Raychev, Martin Vechev, and Eran Yahav. 2014. Code completion with statistical language models. In PLDI, Vol. 49. ACM, 419–428.

Digital Library

[55]

Eric Schkufza, Rahul Sharma, and Alex Aiken. 2013. Stochastic superoptimization. In ASPLOS, Vol. 41. ACM, 305–316.

Digital Library

[56]

Eric Schkufza, Rahul Sharma, and Alex Aiken. 2014. Stochastic optimization of floating-point programs with tunable precision. In PLDI, Vol. 49. ACM, 53–64.

Digital Library

[57]

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. 2015. Trust region policy optimization. In International Conference on Machine Learning. 1889–1897.

[58]

Rahul Sharma and Alex Aiken. 2014. From invariant checking to invariant inference using randomized search. In CAV.

[59]

Rahul Sharma, Saurabh Gupta, Bharath Hariharan, Alex Aiken, and Aditya V Nori. 2013. Verification as learning geometric concepts. In International Static Analysis Symposium. Springer, 388–411.

[60]

Xujie Si, Hanjun Dai, Mukund Raghothaman, Mayur Naik, and Le Song. 2018a. Learning loop invariants for program verification. In Advances in Neural Information Processing Systems. 7762–7773.

[61]

Xujie Si, Yuan Yang, Hanjun Dai, Mayur Naik, and Le Song. 2018b. Learning a Meta-Solver for Syntax-Guided Program Synthesis. In ICLR.

[62]

David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. 2016. Mastering the Game of Go with Deep Neural Networks and Tree Search. Nature 529, 7587 (Jan. 2016), 484–489.

[63]

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. 2017. Mastering the game of go without human knowledge. Nature 550, 7676 (2017), 354.

[64]

Gagandeep Singh, Markus Püschel, and Martin Vechev. 2018. Fast Numerical Program Analysis with Reinforcement Learning. In International Conference on Computer Aided Verification. Springer, 211–229.

[65]

Marcelo Sousa and Isil Dillig. 2016. Cartesian hoare logic for verifying k-safety properties. In Proc. Conference on Programming Language Design and Implementation. 57–69.

Digital Library

[66]

Marcelo Sousa, Isil Dillig, and Shuvendu Lahiri. 2018. Verifying Semantic Conflict-Freedom in Three-Way Program Merges. arXiv preprint arXiv:1802.06551 (2018).

[67]

Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press.

Digital Library

[68]

Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. 2000. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems. 1057–1063.

[69]

Tachio Terauchi and Alex Aiken. 2005. Secure Information Flow As a Safety Problem. In Proceedings of the 12th International Conference on Static Analysis (SAS’05). 352–367.

Digital Library

[70]

Geoffrey Towell and Jude W. Shavlik. 1992. Interpretation of Artificial Neural Networks: Mapping Knowledge-Based Neural Networks into Rules. In Advances in Neural Information Processing Systems 4, J. E. Moody, S. J. Hanson, and R. P. Lippmann (Eds.). Morgan-Kaufmann, 977–984.

[71]

Mingzhe Wang, Yihe Tang, Jian Wang, and Jia Deng. 2017. Premise selection for theorem proving by deep graph embedding. In Advances in Neural Information Processing Systems. 2786–2796.

[72]

Wenhan Xiong, Thien Hoang, and William Yang Wang. 2017. Deeppath: A reinforcement learning method for knowledge graph reasoning. In EMNLP.

[73]

Hongseok Yang. 2007. Relational separation logic. Theoretical Computer Science 375, 1-3 (2007), 308–334.

Digital Library

[74]

Anna Zaks and Amir Pnueli. 2008. Covac: Compiler validation by program analysis of the cross-product. In FM 2008: Formal Methods. Springer, 35–51.

Digital Library

Cited By

Lubin JFerguson JYe KYim JChasins S(2024)Equivalence by Canonicalization for Synthesis-Backed RefactoringProceedings of the ACM on Programming Languages10.1145/36564538:PLDI(1879-1904)Online publication date: 20-Jun-2024
https://dl.acm.org/doi/10.1145/3656453
Miltner AWang ZChaudhuri SDillig I(2024)Relational Synthesis of Recursive Programs via Constraint Annotated Tree AutomataComputer Aided Verification10.1007/978-3-031-65633-0_3(41-63)Online publication date: 26-Jul-2024
https://doi.org/10.1007/978-3-031-65633-0_3
Wang CFan GYao PPan FZhang CGrundy JPollock LPenta M(2023)Verifying Data Constraint Equivalence in FinTech SystemsProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00117(1329-1341)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00117
Show More Cited By

Index Terms

Relational verification using reinforcement learning
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Formal software verification
2. Theory of computation
  1. Theory and algorithms for application domains
    1. Machine learning theory
      1. Reinforcement learning

Recommendations

Safety Verification of Cyber-Physical Systems with Reinforcement Learning Control
Special Issue ESWEEK 2019, CASES 2019, CODES+ISSS 2019 and EMSOFT 2019

This paper proposes a new forward reachability analysis approach to verify safety of cyber-physical systems (CPS) with reinforcement learning controllers. The foundation of our approach lies on two efficient, exact and over-approximate reachability ...
Relational Reinforcement Learning

Relational reinforcement learning is presented, a learning technique that combines reinforcement learning with relational learning or inductive logic programming. Due to the use of a more expressive representation language to represent states, actions ...
Learning with whom to communicate using relational reinforcement learning
AAMAS '09: Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems - Volume 2

Relational reinforcement learning (RRL) has emerged in the machine learning community as a new promising subfield of reinforcement learning (RL) (e.g. [1]). It upgrades RL techniques by using relational representations for states, actions and learned ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Programming Languages

Proceedings of the ACM on Programming Languages Volume 3, Issue OOPSLA

October 2019

2077 pages

EISSN:2475-1421

DOI:10.1145/3366395

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 October 2019

Published in PACMPL Volume 3, Issue OOPSLA

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSF
DARPA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
930
Total Downloads

Downloads (Last 12 months)119
Downloads (Last 6 weeks)14

Reflects downloads up to 18 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lubin JFerguson JYe KYim JChasins S(2024)Equivalence by Canonicalization for Synthesis-Backed RefactoringProceedings of the ACM on Programming Languages10.1145/36564538:PLDI(1879-1904)Online publication date: 20-Jun-2024
https://dl.acm.org/doi/10.1145/3656453
Miltner AWang ZChaudhuri SDillig I(2024)Relational Synthesis of Recursive Programs via Constraint Annotated Tree AutomataComputer Aided Verification10.1007/978-3-031-65633-0_3(41-63)Online publication date: 26-Jul-2024
https://doi.org/10.1007/978-3-031-65633-0_3
Wang CFan GYao PPan FZhang CGrundy JPollock LPenta M(2023)Verifying Data Constraint Equivalence in FinTech SystemsProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00117(1329-1341)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00117
Liu JChen YTan BDillig IFeng Y(2022)Learning Contract Invariants Using Reinforcement LearningProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3556962(1-11)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3551349.3556962
Wang JWang C(2022)Learning to Synthesize Relational InvariantsProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3556942(1-12)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3551349.3556942
Zhang GXu YShen XDillig I(2021)UDF to SQL translation through compositional lazy inductive synthesisProceedings of the ACM on Programming Languages10.1145/34854895:OOPSLA(1-26)Online publication date: 15-Oct-2021
https://dl.acm.org/doi/10.1145/3485489
Lopes NLee JHur CLiu ZRegehr JFreund SYahav E(2021)Alive2: bounded translation validation for LLVMProceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation10.1145/3453483.3454030(65-79)Online publication date: 19-Jun-2021
https://dl.acm.org/doi/10.1145/3453483.3454030
DE ANGELIS EFIORAVANTI FGALLAGHER JHERMENEGILDO MPETTOROSSI APROIETTI M(2021)Analysis and Transformation of Constrained Horn Clauses for Program VerificationTheory and Practice of Logic Programming10.1017/S147106842100021122:6(974-1042)Online publication date: 15-Nov-2021
https://doi.org/10.1017/S1471068421000211
Mariano BChen YFeng YLahiri SDillig IGrundy JLe Goues CLo D(2020)Demystifying loops in smart contractsProceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering10.1145/3324884.3416626(262-274)Online publication date: 21-Dec-2020
https://dl.acm.org/doi/10.1145/3324884.3416626

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents