research-article

Validating SMT solvers via semantic fusion

Authors:

Dominik Winterer,

Zhendong SuAuthors Info & Claims

PLDI 2020: Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation

Pages 718 - 730

https://doi.org/10.1145/3385412.3385985

Published: 11 June 2020 Publication History

Abstract

We introduce Semantic Fusion, a general, effective methodology for validating Satisfiability Modulo Theory (SMT) solvers. Our key idea is to fuse two existing equisatisfiable (i.e., both satisfiable or unsatisfiable) formulas into a new formula that combines the structures of its ancestors in a novel manner and preserves the satisfiability by construction. This fused formula is then used for validating SMT solvers.

We realized Semantic Fusion as YinYang, a practical SMT solver testing tool. During four months of extensive testing, YinYang has found 45 confirmed, unique bugs in the default arithmetic and string solvers of Z3 and CVC4, the two state-of-the-art SMT solvers. Among these, 41 have already been fixed by the developers. The majority (29/45) of these bugs expose critical soundness issues. Our bug reports and testing effort have been well-appreciated by SMT solver developers.

References

[1]

2019. Using the GNU Compiler Collection (GCC): Gcov. Retrieved 2019-10-30 from https://gcc.gnu.org/onlinedocs/gcc/Gcov.html

[2]

Clark Barrett, Christopher L. Conway, Morgan Deters, Liana Hadarean, Dejan Jovanović, Tim King, Andrew Reynolds, and Cesare Tinelli. 2011.

[3]

CVC4. In CAV. 171–177.

[4]

Clark Barrett, Pascal Fontaine, and Cesare Tinelli. 2019. The Satisfiability Modulo Theories Library (SMT-LIB). Retrieved 2019-10-30 from www.SMT-LIB.org

[5]

Clark Barrett, Aaron Stump, and Cesare Tinelli. 2010. The SMT-LIB Standard: Version 2.0. In SMT.

[6]

Dmitry Blotsky, Federico Mora, Murphy Berzish, Yunhui Zheng, Ifaz Kabir, and Vijay Ganesh. 2018. StringFuzz: A fuzzer for string solvers. In CAV. 45–51.

[7]

Robert Brummayer and Armin Biere. 2009. Boolector: An Efficient SMT Solver for Bit-Vectors and Arrays. In TACAS. 174–177.

[8]

Robert Brummayer and Armin Biere. 2009. Fuzzing and deltadebugging SMT solvers. In SMT. 1–5.

[9]

Robert Brummayer, Florian Lonsing, and Armin Biere. 2010. Automated Testing and Debugging of SAT and QBF Solvers. In SAT. 44–57.

[10]

Alexandra Bugariu and Peter Müller. 2020. Automatically Testing String Solvers. In ICSE.

[11]

Alexandra Bugariu, Valentin Wüstholz, Maria Christakis, and Peter Müller. 2018. Automatically testing implementations of numerical abstract domains. In ASE. 768–778.

[12]

Cristian Cadar and Alastair Donaldson. 2016. Analysing the Program Analyser. In ICSE. 765–768.

[13]

Cristian Cadar, Daniel Dunbar, and Dawson R. Engler. 2008. KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs. In OSDI. 209–224.

Digital Library

[14]

Tsong Y Chen, Shing C Cheung, and Shiu Ming Yiu. 1998. Metamorphic testing: a new approach for generating next test cases. Technical Report.

[15]

The International SMT Competition. 2019. SMT-COMP. Retrieved 2019-10-30 from https://smt-comp.github.io/2019/index.html

[16]

Leonardo de Moura and Nikolaj Bjørner. 2008. Z3: An Efficient SMT Solver. In TACAS.

[17]

Rob DeLine and Rustan Leino. 2005. BoogiePL: A Typed Procedural Language for Checking Object-Oriented Programs. Technical Report.

[18]

David Detlefs, Greg Nelson, and James B. Saxe. 2005. Simplify: A Theorem Prover for Program Checking. JACM (2005), 365–473.

[19]

Alastair F Donaldson, Hugues Evrard, Andrei Lascu, and Paul Thomson. 2017. Automated testing of graphics shader compilers. In OOPSLA.

[20]

Vijay Ganesh, Dmitry Blotsky, Federico Mora, Ifaz Kabir, Murphy Berzish, and Yunhui Zheng. 2019. StringFuzz. Retrieved 2019-10-30 from http://stringfuzz.dmitryblotsky.com/

[21]

Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: directed automated random testing. In PLDI. 213–223.

[22]

Timotej Kapus and Cristian Cadar. 2017. Automatic testing of symbolic execution engines via program generation and differential testing. In ASE. 590–600.

[23]

Christian Klinger, Maria Christakis, and Valentin Wüstholz. 2019. Differentially testing soundness and precision of program analyzers. In ISSTA. 239–250.

[24]

Vu Le, Mehrdad Afshari, and Zhendong Su. 2014. Compiler validation via equivalence modulo inputs. In PLDI. 216–226.

[25]

Vu Le, Chengnian Sun, and Zhendong Su. 2015. Finding deep compiler bugs via guided stochastic program mutation. In OOPSLA. 386–399.

[26]

Christopher Lidbury, Andrei Lascu, Nathan Chong, and Alastair F Donaldson. 2015. Many-core compiler fuzzing. In PLDI. 65–76.

[27]

Aina Niemetz, Mathias Preiner, and Armin Biere. 2017. Model-based API testing for SMT solvers. In SMT.

[28]

Felix Pauck, Eric Bodden, and Heike Wehrheim. 2018. Do Android taint analysis tools keep their promises?. In ESEC/FSE. 331–341.

[29]

Lina Qiu, Yingying Wang, and Julia Rubin. 2018. Analyzing the analyzers: FlowDroid/IccTA, AmanDroid, and DroidSafe. In ISSTA. 176–186.

[30]

John Regehr, Yang Chen, Pascal Cuoq, Eric Eide, Chucky Ellison, and Xuejun Yang. 2012. Test-case Reduction for C Compiler Bugs. In PLDI. 335–346.

[31]

Sergio Segura, Gordon Fraser, Ana B Sanchez, and Antonio Ruiz-Cortés. 2016. A survey on metamorphic testing. TSE (2016), 805–824.

[32]

Armando Solar-Lezama. 2008. Program Synthesis by Sketching. Ph.D. Dissertation. EECS Dept., UC Berkeley.

Digital Library

[33]

Chengnian Sun, Vu Le, and Zhendong Su. 2016. Finding compiler bugs via live code mutation. In OOPSLA. 849–863.

[34]

Emina Torlak and Rastislav Bodik. 2014. A lightweight symbolic virtual machine for solver-aided host languages. In PLDI. 530–541.

[35]

Jingyue Wu, Gang Hu, Yang Tang, and Junfeng Yang. 2013. Effective dynamic detection of alias analysis errors. In ESEC/FSE. 279–289.

[36]

Chengyu Zhang, Ting Su, Yichen Yan, Fuyuan Zhang, Geguang Pu, and Zhendong Su. 2019. Finding and understanding bugs in software model checkers. In ESEC/FSE. 763–773.

Cited By

Mathur UMestel DViswanathan M(2025)The Decision Problem for Regular First Order TheoriesProceedings of the ACM on Programming Languages10.1145/37048709:POPL(986-1012)Online publication date: 9-Jan-2025
https://dl.acm.org/doi/10.1145/3704870
Feng NMarsso LChechik MFilkov VRay BZhou M(2024)Diagnosis via Proofs of Unsatisfiability for First-Order Logic with Relational ObjectsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695522(1521-1532)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695522
Wu GCao WYao YWei HChen TMa XFilkov VRay BZhou M(2024)LLM Meets Bounded Model Checking: Neuro-symbolic Loop Invariant InferenceProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695014(406-417)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695014
Show More Cited By

Index Terms

Validating SMT solvers via semantic fusion
1. Software and its engineering
  1. Software organization and properties
    1. Software functional properties
      1. Correctness
      2. Formal methods

Recommendations

Fuzzing SMT solvers via two-dimensional input space exploration
ISSTA 2021: Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis

Satisfiability Modulo Theories (SMT) solvers serve as the core engine of many techniques, such as symbolic execution. Therefore, ensuring the robustness and correctness of SMT solvers is critical. While fuzzing is an efficient and effective method for ...
Generative type-aware mutation for testing SMT solvers

We propose Generative Type-Aware Mutation, an effective approach for testing SMT solvers. The key idea is to realize generation through the mutation of expressions rooted with parametric operators from the SMT-LIB specification. Generative Type-Aware ...
On the unusual effectiveness of type-aware operator mutations for testing SMT solvers

We propose type-aware operator mutation, a simple, but unusually effective approach for testing SMT solvers. The key idea is to mutate operators of conforming types within the seed formulas to generate well-typed mutant formulas. These mutant formulas ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

PLDI 2020: Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation

June 2020

1174 pages

ISBN:9781450376136

DOI:10.1145/3385412

General Chair:
Alastair F. Donaldson
Imperial College London, UK
,
Program Chair:
Emina Torlak
University of Washington, USA

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 June 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Distinguished Paper

Author Tags

Qualifiers

Research-article

Conference

PLDI '20

Sponsor:

SIGPLAN

PLDI '20: 41st ACM SIGPLAN International Conference on Programming Language Design and Implementation

June 15 - 20, 2020

London, UK

Acceptance Rates

Overall Acceptance Rate 406 of 2,067 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

48
Total Citations
View Citations
1,383
Total Downloads

Downloads (Last 12 months)222
Downloads (Last 6 weeks)31

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Mathur UMestel DViswanathan M(2025)The Decision Problem for Regular First Order TheoriesProceedings of the ACM on Programming Languages10.1145/37048709:POPL(986-1012)Online publication date: 9-Jan-2025
https://dl.acm.org/doi/10.1145/3704870
Feng NMarsso LChechik MFilkov VRay BZhou M(2024)Diagnosis via Proofs of Unsatisfiability for First-Order Logic with Relational ObjectsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695522(1521-1532)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695522
Wu GCao WYao YWei HChen TMa XFilkov VRay BZhou M(2024)LLM Meets Bounded Model Checking: Neuro-symbolic Loop Invariant InferenceProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695014(406-417)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695014
Winterer DSu Z(2024)Validating SMT Solvers for Correctness and Performance via Grammar-Based EnumerationProceedings of the ACM on Programming Languages10.1145/36897958:OOPSLA2(2378-2401)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689795
Zhang CSu Z(2024)SMT2Test: From SMT Formulas to Effective Test CasesProceedings of the ACM on Programming Languages10.1145/36897198:OOPSLA2(222-245)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689719
Mikek BZhang Q(2024)SMT Theory Arbitrage: Approximating Unbounded Constraints using Bounded TheoriesProceedings of the ACM on Programming Languages10.1145/36563878:PLDI(246-271)Online publication date: 20-Jun-2024
https://dl.acm.org/doi/10.1145/3656387
Xia CPaltenghi MLe Tian JPradel MZhang LRoychoudhury APaiva AAbreu RStorey M(2024)Fuzz4All: Universal Fuzzing with Large Language ModelsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639121(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639121
Ma PJi ZYao PWang SRen KRoychoudhury APaiva AAbreu RStorey M(2024)Enabling Runtime Verification of Causal Discovery Algorithms with Automated Conditional Independence ReasoningProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3623348(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3623348
Jiang MZheng XChang RZhou YLuo X(2024)Examiner-Pro: Testing Arm Emulators Across Different PrivilegesIEEE Transactions on Software Engineering10.1109/TSE.2024.340690050:11(2786-2806)Online publication date: Nov-2024
https://doi.org/10.1109/TSE.2024.3406900
Gao MWang HXu C(2024)Testing Constraint Checking Implementations via Principled Metamorphic Transformations2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER60148.2024.00096(884-895)Online publication date: 12-Mar-2024
https://doi.org/10.1109/SANER60148.2024.00096
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents