research-article

Fuzzing SMT solvers via two-dimensional input space exploration

Authors:

Charles ZhangAuthors Info & Claims

ISSTA 2021: Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis

Pages 322 - 335

https://doi.org/10.1145/3460319.3464803

Published: 11 July 2021 Publication History

Abstract

Satisfiability Modulo Theories (SMT) solvers serve as the core engine of many techniques, such as symbolic execution. Therefore, ensuring the robustness and correctness of SMT solvers is critical. While fuzzing is an efficient and effective method for validating the quality of SMT solvers, we observe that prior fuzzing work only focused on generating various first-order formulas as the inputs but neglected the algorithmic configuration space of an SMT solver, which leads to under-reporting many deeply-hidden bugs. In this paper, we present Falcon, a fuzzing technique that explores both the formula space and the configuration space. Combining the two spaces significantly enlarges the search space and makes it challenging to detect bugs efficiently. We solve this problem by utilizing the correlations between the two spaces to reduce the search space, and introducing an adaptive mutation strategy to boost the search efficiency. During six months of extensive testing, Falcon finds 518 confirmed bugs in CVC4 and Z3, two state-of-the-art SMT solvers, 469 of which have already been fixed. Compared to two state-of-the-art fuzzers, Falcon detects 38 and 44 more bugs and improves the coverage by a large margin in 24 hours of testing.

References

[1]

Carlos Ansótegui, Meinolf Sellmann, and Kevin Tierney. 2009. A Gender-Based Genetic Algorithm for the Automatic Configuration of Algorithms. In Principles and Practice of Constraint Programming-CP 2009, 15th International Conference, CP 2009, Lisbon, Portugal, September 20-24, 2009, Proceedings (Lecture Notes in Computer Science, Vol. 5732 ), Ian P. Gent (Ed.). Springer, 142-157. https://doi.org/ 10.1007/978-3-642-04244-7_14

[2]

Cornelius Aschermann, Tommaso Frassetto, Thorsten Holz, Patrick Jauernig, Ahmad-Reza Sadeghi, and Daniel Teuchert. 2019. NAUTILUS: Fishing for Deep Bugs with Grammars. In 26th Annual Network and Distributed System Security Symposium, NDSS 2019, San Diego, California, USA, February 24-27, 2019. The Internet Society. https://www.ndss-symposium.org/ndss-paper/nautilus-fishingfor-deep-bugs-with-grammars/

[3]

Thanassis Avgerinos, Alexandre Rebert, Sang Kil Cha, and David Brumley. 2014. Enhancing Symbolic Execution with Veritesting. In Proceedings of the 36th International Conference on Software Engineering (Hyderabad, India) ( ICSE 2014). ACM, New York, NY, USA, 1083-1094. https://doi.org/10.1145/2568225.2568293

Digital Library

[4]

Mislav Balunovic, Pavol Bielik, and Martin T. Vechev. 2018. Learning to Solve SMT Formulas. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (Eds.). 10338-10349. https://proceedings.neurips.cc/paper/2018/hash/ 68331f0427b551b68e911eebe35233b-Abstract.html

[5]

Eduard Baranov, Axel Legay, and Kuldeep S. Meel. 2020. Baital: an adaptive weighted sampling approach for improved t-wise coverage. In ESEC/FSE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020, Prem Devanbu, Myra B. Cohen, and Thomas Zimmermann (Eds.). ACM, 1114-1126. https://doi.org/10.1145/3368089.3409744

Digital Library

[6]

Clark Barrett, Christopher Conway, Morgan Deters, Liana Hadarean, Dejan Jovanović, Tim King, Andrew Reynolds, and Cesare Tinelli. 2011. CVC4. In Proceedings of the 23rd International Conference on Computer Aided Verification (Snowbird, UT) (CAV'11). Springer-Verlag, Berlin, Heidelberg, 171-177. http://dl.acm.org/citation.cfm?id= 2032305. 2032319

Digital Library

[7]

Clark Barrett, Aaron Stump, and Cesare Tinelli. 2010. The satisfiability modulo theories library (SMT-LIB). www. SMT-LIB. org 15 ( 2010 ), 18-52.

[8]

Murphy Berzish, Vijay Ganesh, and Yunhui Zheng. 2017. Z3str3: A String Solver with Theory-Aware Heuristics. In Proceedings of the 17th Conference on Formal Methods in Computer-Aided Design (Vienna, Austria) ( FMCAD '17). FMCAD Inc, Austin, Texas, 55-59. https://doi.org/10.5555/3168451.3168468

[9]

Karthikeyan Bhargavan, Barry Bond, Antoine Delignat-Lavaud, Cédric Fournet, Chris Hawblitzel, Catalin Hritcu, Samin Ishtiaq, Markulf Kohlweiss, Rustan Leino, Jay R. Lorch, Kenji Maillard, Jianyang Pan, Bryan Parno, Jonathan Protzenko, Tahina Ramananandro, Ashay Rane, Aseem Rastogi, Nikhil Swamy, Laure Thompson, Peng Wang, Santiago Zanella Béguelin, and Jean Karim Zinzindohoue. 2017. Everest: Towards a Verified, Drop-in Replacement of HTTPS. In 2nd Summit on Advances in Programming Languages, SNAPL 2017, May 7-10, 2017, Asilomar, CA, USA (LIPIcs, Vol. 71 ), Benjamin S. Lerner, Rastislav Bodík, and Shriram Krishnamurthi (Eds.). Schloss Dagstuhl-Leibniz-Zentrum für Informatik, 1 : 1-1 : 12. https://doi.org/10.4230/LIPIcs.SNAPL. 2017.1

[10]

Nikolaj Bjørner, Vijay Ganesh, Raphaël Michel, and Margus Veanes. 2012. An SMT-LIB format for sequences and regular expressions. SMT 12 ( 2012 ), 76-86.

[11]

Tim Blazytko, Moritz Contag, Cornelius Aschermann, and Thorsten Holz. 2017. Syntia: Synthesizing the Semantics of Obfuscated Code. In 26th USENIX Security Symposium, USENIX Security 2017, Vancouver, BC, Canada, August 16-18, 2017, Engin Kirda and Thomas Ristenpart (Eds.). USENIX Association, 643-659. https://www.usenix.org/conference/usenixsecurity17/technical-sessions/ presentation/blazytko

[12]

Dmitry Blotsky, Federico Mora, Murphy Berzish, Yunhui Zheng, Ifaz Kabir, and Vijay Ganesh. 2018. StringFuzz: A Fuzzer for String Solvers. In Computer Aided Verification-30th International Conference, CAV 2018, Held as Part of the Federated Logic Conference, FloC 2018, Oxford, UK, July 14-17, 2018, Proceedings, Part II (Lecture Notes in Computer Science, Vol. 10982 ), Hana Chockler and Georg Weissenbacher (Eds.). Springer, 45-51. https://doi.org/10.1007/978-3-319-96142-2_6

[13]

Malik Bouchet, Byron Cook, Bryant Cutler, Anna Druzkina, Andrew Gacek, Liana Hadarean, Ranjit Jhala, Brad Marshall, Daniel Peebles, Neha Rungta, Cole Schlesinger, Chriss Stephens, Carsten Varming, and Andy Warfield. 2020. Block public access: trust safety verification of access control policies. In ESEC/FSE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020, Prem Devanbu, Myra B. Cohen, and Thomas Zimmermann (Eds.). ACM, 281-291. https://doi.org/10.1145/3368089.3409728

Digital Library

[14]

Robert Brummayer and Armin Biere. 2009. Fuzzing and Delta-Debugging SMT Solvers. In Proceedings of the 7th International Workshop on Satisfiability Modulo Theories (Montreal, Canada) ( SMT '09). Association for Computing Machinery, New York, NY, USA, 1-5. https://doi.org/10.1145/1670412.1670413

Digital Library

[15]

Roberto Bruttomesso, Edgar Pek, Natasha Sharygina, and Aliaksei Tsitovich. 2010. The OpenSMT Solver. In Tools and Algorithms for the Construction and Analysis of Systems, 16th International Conference, TACAS 2010, Held as Part of the Joint European Conferences on Theory and Practice of Software, ETAPS 2010, Paphos, Cyprus, March 20-28, 2010. Proceedings (Lecture Notes in Computer Science, Vol. 6015 ), Javier Esparza and Rupak Majumdar (Eds.). Springer, 150-153. https://doi.org/10.1007/978-3-642-12002-2_12

Digital Library

[16]

Alexandra Bugariu and Peter Müller. 2020. Automatically testing string solvers. In ICSE '20: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June-19 July, 2020, Gregg Rothermel and Doo-Hwan Bae (Eds.). ACM, 1459-1470. https://doi.org/10.1145/3377811.3380398

Digital Library

[17]

Cristian Cadar, Daniel Dunbar, Dawson R Engler, et al. 2008. KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation (San Diego, California) ( OSDI'08). USENIX Association, Berkeley, CA, USA, 209-224. http://dl.acm.org/citation.cfm?id= 1855741. 1855756

Digital Library

[18]

Junjie Chen, Guancheng Wang, Dan Hao, Yingfei Xiong, Hongyu Zhang, and Lu Zhang. 2019. History-Guided Configuration Diversification for Compiler TestProgram Generation. In 34th IEEE/ACM International Conference on Automated Software Engineering, ASE 2019, San Diego, CA, USA, November 11-15, 2019. IEEE, 305-316. https://doi.org/10.1109/ASE. 2019.00037

Digital Library

[19]

Jürgen Christ, Jochen Hoenicke, and Alexander Nutz. 2012. SMTInterpol: An Interpolating SMT Solver. In Model Checking Software-19th International Workshop, SPIN 2012, Oxford, UK, July 23-24, 2012. Proceedings (Lecture Notes in Computer Science, Vol. 7385 ), Alastair F. Donaldson and David Parker (Eds.). Springer, 248-254. https://doi.org/10.1007/978-3-642-31759-0_19

Digital Library

[20]

Myra B. Cohen, Matthew B. Dwyer, and Jiangfan Shi. 2006. Coverage and adequacy in software product line testing. In Proceedings of the 2006 Workshop on Role of Software Architecture for Testing and Analysis, held in conjunction with the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2006 ), ROSATEA 2006, Portland, Maine, USA, July 17-20, 2006, Robert M. Hierons and Henry Muccini (Eds.). ACM, 53-63. https://doi.org/10.1145/1147249.1147257

Digital Library

[21]

Byron Cook. 2018. Formal Reasoning About the Security of Amazon Web Services. In Computer Aided Verification-30th International Conference, CAV 2018, Held as Part of the Federated Logic Conference, FloC 2018, Oxford, UK, July 14-17, 2018, Proceedings, Part I (Lecture Notes in Computer Science, Vol. 10981 ), Hana Chockler and Georg Weissenbacher (Eds.). Springer, 38-47. https://doi.org/10.1007/978-3-319-96145-3_3

[22]

Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: An eficient SMT solver. In Proceedings of the Theory and Practice of Software, 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (Budapest, Hungary) ( TACAS'08/ETAPS'08). Springer-Verlag, Berlin, Heidelberg, 337-340. http://dl.acm.org/citation.cfm?id= 1792734. 1792766

[23]

Leonardo de Moura and Grant Olney Passmore. 2013. The Strategy Challenge in SMT Solving. ( 2013 ), 15-44.

[24]

Kalyanmoy Deb and Debayan Deb. 2014. Analysing Mutation Schemes for RealParameter Genetic Algorithms. Int. J. Artif. Intell. Soft Comput. 4, 1 (Feb. 2014 ), 1-28. https://doi.org/10.1504/IJAISC. 2014.059280

Digital Library

[25]

Bruno Dutertre. 2014. Yices2.2. In Proceedings of the 16th International Conference on Computer Aided Verification-Volume 8559. Springer-Verlag, Berlin, Heidelberg, 737-744. https://doi.org/10.1007/978-3-319-08867-9_49

Digital Library

[26]

Bruno Dutertre and Leonardo Mendonça de Moura. 2006. A Fast LinearArithmetic Solver for DPLL(T). In Computer Aided Verification, 18th International Conference, CAV 2006, Seattle, WA, USA, August 17-20, 2006, Proceedings (Lecture Notes in Computer Science, Vol. 4144 ), Thomas Ball and Robert B. Jones (Eds.). Springer, 81-94. https://doi.org/10.1007/11817963_11

Digital Library

[27]

Martin Eberlein, Yannic Noller, Thomas Vogel, and Lars Grunske. 2020. Evolutionary Grammar-Based Fuzzing. In Search-Based Software Engineering, Aldeida Aleti and Annibale Panichella (Eds.). Springer International Publishing, Cham, 105-120. https://doi.org/10.1007/978-3-030-59762-7_8

Digital Library

[28]

M Eddington. 2008. Peach fuzzer. URl: http://www. peachfuzzer. com/(visited on 06/21/ 2017 ) ( 2008 ).

[29]

Hassan Eldib, Chao Wang, and Patrick Schaumont. 2014. SMT-Based Verification of Software Countermeasures against Side-Channel Attacks. In Tools and Algorithms for the Construction and Analysis of Systems-20th International Conference, TACAS 2014, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2014, Grenoble, France, April 5-13, 2014. Proceedings (Lecture Notes in Computer Science, Vol. 8413 ), Erika Ábrahám and Klaus Havelund (Eds.). Springer, 62-77. https://doi.org/10.1007/978-3-642-54862-8_5

[30]

Vijay Ganesh and David L Dill. 2007. A decision procedure for bit-vectors and arrays. In Proceedings of the 19th International Conference on Computer Aided Verification (Berlin, Germany) ( CAV'07). Springer-Verlag, Berlin, Heidelberg, 519-531. http://dl.acm.org/citation.cfm?id= 1770351. 1770421

[31]

Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: directed automated random testing. In Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation (Chicago, IL, USA) ( PLDI '05). ACM, New York, NY, USA, 213-223. https://doi.org/10.1145/1065010.1065036

Digital Library

[32]

Patrice Godefroid, Michael Y. Levin, and David Molnar. 2012. SAGE: Whitebox Fuzzing for Security Testing. Queue 10, 1, Article 20 ( Jan. 2012 ), 8 pages. https: //doi.org/10.1145/2090147.2094081

Digital Library

[33]

Alex Groce, Chaoqiang Zhang, Eric Eide, Yang Chen, and John Regehr. 2012. Swarm testing. In International Symposium on Software Testing and Analysis, ISSTA 2012, Minneapolis, MN, USA, July 15-20, 2012, Mats Per Erik Heimdahl and Zhendong Su (Eds.). ACM, 78-88. https://doi.org/10.1145/2338965.2336763

Digital Library

[34]

Hai-Feng Guo and Zongyan Qiu. 2013. Automatic Grammar-Based Test Generation. In Testing Software and Systems-25th IFIP WG 6.1 International Conference, ICTSS 2013, Istanbul, Turkey, November 13-15, 2013, Proceedings (Lecture Notes in Computer Science, Vol. 8254 ), Hüsnü Yenigün, Cemal Yilmaz, and Andreas Ulrich (Eds.). Springer, 17-32. https://doi.org/10.1007/978-3-642-41707-8_2

[35]

Liana Hadarean, Kshitij Bansal, Dejan Jovanović, Clark Barrett, and Cesare Tinelli. 2014. A tale of two solvers: Eager and lazy approaches to bit-vectors. In Proceedings of the 16th International Conference on Computer Aided Verification-Volume 8559. Springer-Verlag, Berlin, Heidelberg, 680-695. https://doi.org/10.1007/978-3-319-08867-9_45

Digital Library

[36]

Axel Halin, Alexandre Nuttinck, Mathieu Acher, Xavier Devroey, Gilles Perrouin, and Benoit Baudry. 2019. Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack. Empir. Softw. Eng. 24, 2 ( 2019 ), 674-717. https://doi.org/10.1007/s10664-018-9635-4

Digital Library

[37]

HyungSeok Han, DongHyeon Oh, and Sang Kil Cha. 2019. CodeAlchemist: Semantics-Aware Code Generation to Find Vulnerabilities in JavaScript Engines. In 26th Annual Network and Distributed System Security Symposium, NDSS 2019, San Diego, California, USA, February 24-27, 2019. The Internet Society. https://www.ndss-symposium.org/ndss-paper/codealchemist-semanticsaware-code-generation-to-find-vulnerabilities-in-javascript-engines/

[38]

Mark Hennessy and James F. Power. 2005. An analysis of rule coverage as a criterion in generating minimal test suites for grammar-based software. In 20th IEEE/ACM International Conference on Automated Software Engineering (ASE 2005 ), November 7-11, 2005, Long Beach, CA, USA, David F. Redmiles, Thomas Ellman, and Andrea Zisman (Eds.). ACM, 104-113. https://doi.org/10.1145/1101908.1101926

Digital Library

[39]

Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. 2011. Sequential ModelBased Optimization for General Algorithm Configuration. In Learning and Intelligent Optimization-5th International Conference, LION 5, Rome, Italy, January 17-21, 2011. Selected Papers (Lecture Notes in Computer Science, Vol. 6683 ), Carlos A. Coello Coello (Ed.). Springer, 507-523. https://doi.org/10.1007/978-3-642-25566-3_40

Digital Library

[40]

Frank Hutter, Holger H. Hoos, Kevin Leyton-Brown, and Thomas Stützle. 2009. ParamILS: An Automatic Algorithm Configuration Framework. J. Artif. Intell. Res. 36 ( 2009 ), 267-306. https://doi.org/10.1613/jair.2861

[41]

Christian Kaltenecker, Alexander Grebhahn, Norbert Siegmund, Jianmei Guo, and Sven Apel. 2019. Distance-based sampling of software configuration spaces. In Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019, Joanne M. Atlee, Tevfik Bultan, and Jon Whittle (Eds.). IEEE / ACM, 1084-1094. https://doi.org/10.1109/ICSE. 2019.00112

Digital Library

[42]

George Klees, Andrew Ruef, Benji Cooper, Shiyi Wei, and Michael Hicks. 2018. Evaluating Fuzz Testing. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, CCS 2018, Toronto, ON, Canada, October 15-19, 2018, David Lie, Mohammad Mannan, Michael Backes, and XiaoFeng Wang (Eds.). ACM, 2123-2138. https://doi.org/10.1145/3243734.3243804

Digital Library

[43]

Ralf Lämmel and Wolfram Schulte. 2006. Controllable Combinatorial Coverage in Grammar-Based Testing. In Testing of Communicating Systems, 18th IFIP TC6/WG6.1 International Conference, TestCom 2006, New York, NY, USA, May 16-18, 2006, Proceedings (Lecture Notes in Computer Science, Vol. 3964 ), M. Ümit Uyar, Ali Y. Duale, and Mariusz A. Fecko (Eds.). Springer, 19-38. https://doi.org/10.1007/11754008_2

Digital Library

[44]

Roberto E. Lopez-Herrejon, Francisco Chicano, Javier Ferrer, Alexander Egyed, and Enrique Alba. 2013. Multi-objective Optimal Test Suite Computation for Software Product Line Pairwise Testing. In 2013 IEEE International Conference on Software Maintenance, Eindhoven, The Netherlands, September 22-28, 2013. IEEE Computer Society, 404-407. https://doi.org/10.1109/ICSM. 2013.58

Digital Library

[45]

Feng Lu, Li-C. Wang, Kwang-Ting Cheng, and Ric C.-Y. Huang. 2003. A Circuit SAT Solver With Signal Correlation Guided Learning. In 2003 Design, Automation and Test in Europe Conference and Exposition (DATE 2003 ), 3-7 March 2003, Munich, Germany. IEEE Computer Society, 10892-10897. https://doi.org/10.1109/DATE. 2003.10018

[46]

Muhammad Numair Mansur, Maria Christakis, Valentin Wüstholz, and Fuyuan Zhang. 2020. Detecting critical bugs in SMT solvers using blackbox mutational fuzzing. In ESEC/FSE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020, Prem Devanbu, Myra B. Cohen, and Thomas Zimmermann (Eds.). ACM, 701-712. https://doi.org/10.1145/3368089.3409763

Digital Library

[47]

Peter M. Maurer. 1990. Generating Test Data with Enhanced Context-Free Grammars. IEEE Softw. 7, 4 ( 1990 ), 50-55. https://doi.org/10.1109/52.56422

Digital Library

[48]

Changhai Nie and Hareton Leung. 2011. A survey of combinatorial testing. ACM Comput. Surv. 43, 2 ( 2011 ), 11 : 1-11 : 29. https://doi.org/10.1145/1883612.1883618

Digital Library

[49]

Aina Niemetz and Armin Biere. 2013. ddSMT: a delta debugger for the SMTLIB v2 format. In Proceedings of the 11th International Workshop on Satisfiability Modulo Theories (SMT'13), afiliated to SAT, Vol. 13. 36-45. https://doi.org/doi= 10.1.1.380.134

[50]

Aina Niemetz, Mathias Preiner, and Armin Biere. 2014. Boolector 2.0. J. Satisf. Boolean Model. Comput. 9, 1 ( 2014 ), 53-58. https://doi.org/10.3233/sat190101

[51]

Aina Niemetz, Mathias Preiner, and Armin Biere. 2017. Model-based API testing for SMT solvers. In Proceedings of the 15th International Workshop on Satisfiability Modulo Theories, SMT.

[52]

Andres Nötzli, Andrew Reynolds, Haniel Barbosa, Aina Niemetz, Mathias Preiner, Clark Barrett, and Cesare Tinelli. 2019. Syntax-Guided Rewrite Rule Enumeration for SMT Solvers. In Theory and Applications of Satisfiability Testing-SAT 2019, Mikoláš Janota and Inês Lynce (Eds.). Springer International Publishing, Cham, 279-297. https://doi.org/10.1007/978-3-030-24258-9_20

[53]

Jeho Oh, Don S. Batory, Margaret Myers, and Norbert Siegmund. 2017. Finding near-optimal configurations in product lines by random sampling. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn, Germany, September 4-8, 2017, Eric Bodden, Wilhelm Schäfer, Arie van Deursen, and Andrea Zisman (Eds.). ACM, 61-71. https://doi.org/10. 1145/3106237.3106273

Digital Library

[54]

Rohan Padhye, Caroline Lemieux, Koushik Sen, Mike Papadakis, and Yves Le Traon. 2019. Semantic fuzzing with zest. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2019, Beijing, China, July 15-19, 2019, Dongmei Zhang and Anders Møller (Eds.). ACM, 329-340. https://doi.org/10.1145/3293882.3330576

Digital Library

[55]

Stuart Pernsteiner, Calvin Loncaric, Emina Torlak, Zachary Tatlock, Xi Wang, Michael D. Ernst, and Jonathan Jacky. 2016. Investigating Safety of a Radiotherapy Machine Using System Models with Pluggable Checkers. In Computer Aided Verification-28th International Conference, CAV 2016, Toronto, ON, Canada, July 17-23, 2016, Proceedings, Part II (Lecture Notes in Computer Science, Vol. 9780 ), Swarat Chaudhuri and Azadeh Farzan (Eds.). Springer, 23-41. https://doi.org/10. 1007/978-3-319-41540-6_2

[56]

Quentin Plazar, Mathieu Acher, Gilles Perrouin, Xavier Devroey, and Maxime Cordy. 2019. Uniform Sampling of SAT Solutions for Configurable Systems: Are We There Yet?. In 12th IEEE Conference on Software Testing, Validation and Verification, ICST 2019, Xi'an, China, April 22-27, 2019. IEEE, 240-251. https: //doi.org/10.1109/ICST. 2019.00032

[57]

Mathias Preiner, Aina Niemetz, and Armin Biere. 2017. Counterexample-Guided Model Synthesis. In Tools and Algorithms for the Construction and Analysis of Systems, Axel Legay and Tiziana Margaria (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 264-280. https://doi.org/10.1007/978-3-662-54577-5_15

[58]

Paul Purdom. 1972. A sentence generator for testing parsers. BIT Numerical Mathematics 12, 3 ( 1972 ), 366-375. https://doi.org/10.1007/BF01932308

Digital Library

[59]

Joseph Scott, Federico Mora, and Vijay Ganesh. 2020. BanditFuzz: A Reinforcement-Learning Based Performance Fuzzer for SMT Solvers. In Software Verification-12th International Conference, VSTTE 2020, and 13th International Workshop, NSV 2020, Los Angeles, CA, USA, July 20-21, 2020, Revised Selected Papers (Lecture Notes in Computer Science, Vol. 12549 ), Maria Christakis, Nadia Polikarpova, Parasara Sridhar Duggirala, and Peter Schrammel (Eds.). Springer, 68-86. https://doi.org/10.1007/978-3-030-63618-0_5

Digital Library

[60]

Konstantin Serebryany, Derek Bruening, Alexander Potapenko, and Dmitriy Vyukov. 2012. AddressSanitizer: A Fast Address Sanity Checker. In 2012 USENIX Annual Technical Conference, Boston, MA, USA, June 13-15, 2012, Gernot Heiser and Wilson C. Hsieh (Eds.). USENIX Association, 309-318. https://www.usenix. org/conference/atc12/technical-sessions/presentation/serebryany

[61]

João P. Marques Silva and Karem A. Sakallah. 1996. GRASP-a new search algorithm for satisfiability. In Proceedings of the 1996 IEEE/ACM International Conference on Computer-Aided Design, ICCAD 1996, San Jose, CA, USA, November 10-14, 1996, Rob A. Rutenbar and Ralph H. J. M. Otten (Eds.). IEEE Computer Society / ACM, 220-227. https://doi.org/10.1109/ICCAD. 1996.569607

[62]

Emin Gün Sirer and Brian N. Bershad. 1999. Using production grammars in software testing. In Proceedings of the Second Conference on Domain-Specific Languages (DSL '99), Austin, Texas, USA, October 3-5, 1999, Thomas Ball (Ed.). ACM, 1-13. https://doi.org/10.1145/331960.331965

Digital Library

[63]

Ezekiel Soremekun, Esteban Pavese, Nikolas Havrikov, Lars Grunske, and Andreas Zeller. 2020. Inputs from Hell Learning Input Distributions for GrammarBased Test Generation. IEEE Transactions on Software Engineering ( 2020 ), 1-1. https://doi.org/10.1109/TSE. 2020.3013716

[64]

Thomas Thüm, Sven Apel, Christian Kästner, Ina Schaefer, and Gunter Saake. 2014. A Classification and Survey of Analysis Strategies for Software Product Lines. ACM Comput. Surv. 47, 1 ( 2014 ), 6 : 1-6 : 45. https://doi.org/10.1145/2580950

Digital Library

[65]

Dominik Winterer, Chengyu Zhang, and Zhendong Su. 2020. On the unusual efectiveness of type-aware operator mutations for testing SMT solvers. Proc. ACM Program. Lang. 4, OOPSLA ( 2020 ), 193 : 1-193 : 25. https://doi.org/10.1145/ 3428261

Digital Library

[66]

Dominik Winterer, Chengyu Zhang, and Zhendong Su. 2020. Validating SMT solvers via semantic fusion. In Proceedings of the 41st ACM SIGPLAN International Conference on Programming Language Design and Implementation, PLDI 2020, London, UK, June 15-20, 2020, Alastair F. Donaldson and Emina Torlak (Eds.). ACM, 718-730. https://doi.org/10.1145/3385412.3385985

Digital Library

[67]

Akihisa Yamada, Armin Biere, Cyrille Artho, Takashi Kitamura, and Eun-Hye Choi. 2016. Greedy combinatorial test case generation using unsatisfiable cores. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, ASE 2016, Singapore, September 3-7, 2016, David Lo, Sven Apel, and Sarfraz Khurshid (Eds.). ACM, 614-624. https://doi.org/10.1145/2970276.2970335

Digital Library

Cited By

Winterer DSu Z(2024)Validating SMT Solvers for Correctness and Performance via Grammar-Based EnumerationProceedings of the ACM on Programming Languages10.1145/36897958:OOPSLA2(2378-2401)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689795
Zhang CSu Z(2024)SMT2Test: From SMT Formulas to Effective Test CasesProceedings of the ACM on Programming Languages10.1145/36897198:OOPSLA2(222-245)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689719
Chu LHuang MLi XNie Y(2024)A Review of Fuzz Testing for Configuration-Sensitive Software2024 9th International Conference on Signal and Image Processing (ICSIP)10.1109/ICSIP61881.2024.10671554(388-398)Online publication date: 12-Jul-2024
https://doi.org/10.1109/ICSIP61881.2024.10671554
Show More Cited By

Index Terms

Fuzzing SMT solvers via two-dimensional input space exploration
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation

Recommendations

On the unusual effectiveness of type-aware operator mutations for testing SMT solvers

We propose type-aware operator mutation, a simple, but unusually effective approach for testing SMT solvers. The key idea is to mutate operators of conforming types within the seed formulas to generate well-typed mutant formulas. These mutant formulas ...
Detecting critical bugs in SMT solvers using blackbox mutational fuzzing
ESEC/FSE 2020: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Formal methods use SMT solvers extensively for deciding formula satisfiability, for instance, in software verification, systematic test generation, and program synthesis. However, due to their complex implementations, solvers may contain critical bugs ...
Validating SMT solvers via semantic fusion
PLDI 2020: Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation

We introduce Semantic Fusion, a general, effective methodology for validating Satisfiability Modulo Theory (SMT) solvers. Our key idea is to fuse two existing equisatisfiable (i.e., both satisfiable or unsatisfiable) formulas into a new formula that ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA 2021: Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis

July 2021

685 pages

ISBN:9781450384599

DOI:10.1145/3460319

General Chair:
Cristian Cadar
Imperial College London, UK
,
Program Chair:
Xiangyu Zhang
Purdue University, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 July 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ISSTA '21

Sponsor:

SIGSOFT

ISSTA '21: 30th ACM SIGSOFT International Symposium on Software Testing and Analysis

July 11 - 17, 2021

Virtual, Denmark

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
568
Total Downloads

Downloads (Last 12 months)84
Downloads (Last 6 weeks)7

Reflects downloads up to 26 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Winterer DSu Z(2024)Validating SMT Solvers for Correctness and Performance via Grammar-Based EnumerationProceedings of the ACM on Programming Languages10.1145/36897958:OOPSLA2(2378-2401)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689795
Zhang CSu Z(2024)SMT2Test: From SMT Formulas to Effective Test CasesProceedings of the ACM on Programming Languages10.1145/36897198:OOPSLA2(222-245)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689719
Chu LHuang MLi XNie Y(2024)A Review of Fuzz Testing for Configuration-Sensitive Software2024 9th International Conference on Signal and Image Processing (ICSIP)10.1109/ICSIP61881.2024.10671554(388-398)Online publication date: 12-Jul-2024
https://doi.org/10.1109/ICSIP61881.2024.10671554
Mansur MWüstholz VChristakis MJust RFraser G(2023)Dependency-Aware Metamorphic Testing of Datalog EnginesProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598052(236-247)Online publication date: 13-Jul-2023
https://doi.org/10.1145/3597926.3598052
Zhang YXie XLi YLin YChen SLiu YLi X(2023)Demystifying Performance Regressions in String SolversIEEE Transactions on Software Engineering10.1109/TSE.2022.316837349:3(947-961)Online publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1109/TSE.2022.3168373
Kim JSo SOh HGrundy JPollock LPenta M(2023)DIVER: Oracle-Guided SMT Solver Testing with Unrestricted Random MutationsProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00187(2224-2236)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00187
Sun MYang YWen MWang YZhou YJin HGrundy JPollock LPenta M(2023)Validating SMT Solvers via Skeleton Enumeration Empowered by Historical Bug-Triggering InputsProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00018(69-81)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00018
Sun MYang YWang YWen MJia HZhou YBissyandé TKlein JBird CSarro F(2023)SMT Solver Validation Empowered by Large Pre-Trained Language ModelsProceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE56229.2023.00180(1288-1300)Online publication date: 11-Nov-2023
https://dl.acm.org/doi/10.1109/ASE56229.2023.00180
Tang WHu YFan GYao PWu RBai GWang PZhang CGrundy J(2021)TranscodeProceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE51524.2021.9678823(829-841)Online publication date: 15-Nov-2021
https://dl.acm.org/doi/10.1109/ASE51524.2021.9678823

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten