research-article

Public Access

Efficient sampling of SAT solutions for testing

Authors:

Jonathan Bachrach,

Koushik SenAuthors Info & Claims

ICSE '18: Proceedings of the 40th International Conference on Software Engineering

Pages 549 - 559

https://doi.org/10.1145/3180155.3180248

Published: 27 May 2018 Publication History

Abstract

In software and hardware testing, generating multiple inputs which satisfy a given set of constraints is an important problem with applications in fuzz testing and stimulus generation. However, it is a challenge to perform the sampling efficiently, while generating a diverse set of inputs which satisfy the constraints. We developed a new algorithm QuickSampler which requires a small number of solver calls to produce millions of samples which satisfy the constraints with high probability. We evaluate QuickSampler on large real-world benchmarks and show that it can produce unique valid solutions orders of magnitude faster than other state-of-the-art sampling tools, with a distribution which is reasonably close to uniform in practice.

References

[1]

Saswat Anand and Mary Jean Harrold. 2011. Heap cloning: Enabling dynamic symbolic execution of java programs. In ASE. 33--42.

Digital Library

[2]

Saswat Anand, Corina S. Păsăreanu, and Willem Visser. 2007. JPF-SE: a symbolic execution extension to Java PathFinder. In TACAS'07.

Digital Library

[3]

Shay Artzi, Adam Kiezun, Julian Dolby, Frank Tip, Danny Dig, Amit Paradkar, and Michael D. Ernst. 2008. Finding bugs in dynamic web applications. In ISSTA'08.

Digital Library

[4]

Thanassis Avgerinos, Alexandre Rebert, Sang Kil Cha, and David Brumley. 2014. Enhancing Symbolic Execution with Veritesting. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014). ACM, New York, NY, USA, 1083--1094.

Digital Library

[5]

Nikolaj Bjørner, Anh-Dung Phan, and Lars Fleckenstein. 2015. vZ-An Optimizing SMT Solver. In TACAS, Vol. 15. 194--199.

Digital Library

[6]

Marcel Böhme, Van-Thuan Pham, and Abhik Roychoudhury. 2016. Coverage-based greybox fuzzing as markov chain. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. ACM, 1032--1043.

Digital Library

[7]

Jacob Burnim and Koushik Sen. 2008. Heuristics for Scalable Dynamic Test Generation. In ASE'08.

Digital Library

[8]

Cristian Cadar, Daniel Dunbar, and Dawson Engler. 2008. KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs. In OSDI'08.

Digital Library

[9]

Supratik Chakraborty, Daniel J Fremont, Kuldeep S Meel, Sanjit A Seshia, and Moshe Y Vardi. 2015. On Parallel Scalable Uniform SAT Witness Generation. In TACAS. 304--319.

Digital Library

[10]

Supratik Chakraborty, Kuldeep S Meel, and Moshe Y Vardi. 2013. A scalable approximate model counter. In International Conference on Principles and Practice of Constraint Programming. Springer, 200--216.

[11]

Supratik Chakraborty, Kuldeep S Meel, and Moshe Y Vardi. 2014. Balancing scalability and uniformity in SAT witness generator. In Design Automation Conference (DAC), 2014 51st ACM/EDAC/IEEE. IEEE, 1--6.

Digital Library

[12]

Vitaly Chipounov, Volodymyr Kuznetsov, and George Candea. 2012. The S2E Platform: Design, Implementation, and Applications. ACM Trans. Comput. Syst. 30, 1 (2012), 2.

Digital Library

[13]

Lori A. Clarke. 1976. A program testing system. In Proc. of the 1976 annual conference. 488--491.

Digital Library

[14]

Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: An efficient SMT solver. Tools and Algorithms for the Construction and Analysis of Systems (2008), 337--340.

Digital Library

[15]

Stefano Ermon, Carla P Gomes, Ashish Sabharwal, and Bart Selman. 2013. Embed and project: Discrete sampling with universal hashing. In Advances in Neural Information Processing Systems. 2085--2093.

Digital Library

[16]

Stefano Ermon, Carla P Gomes, and Bart Selman. 2012. Uniform solution sampling using a constraint solver as an oracle. Conference on Uncertainty in Artificial Intelligence (2012).

Digital Library

[17]

Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE '11). ACM, New York, NY, USA, 416--419.

Digital Library

[18]

P. Godefroid, N. Klarlund, and K. Sen. 2005. DART: Directed Automated Random Testing. In PLDI'05.

Digital Library

[19]

P. Godefroid, M.Y. Levin, and D. Molnar. 2008. Automated Whitebox Fuzz Testing. In NDSS'08.

[20]

Carla P Gomes, Ashish Sabharwal, and Bart Selman. 2007. Near-uniform sampling of combinatorial spaces using XOR constraints. In Advances In Neural Information Processing Systems. 481--488.

Digital Library

[21]

Christian Holler, Kim Herzig, and Andreas Zeller. 2012. Fuzzing with Code Fragments. In Proceedings of the 21st USENIX Conference on Security Symposium (Security'12). USENIX Association, Berkeley, CA, USA, 38--38.

Digital Library

[22]

Allen D. Householder and Jonathan M. Foote. 2012. Probability-Based Parameter Selection for Black-Box Fuzz Testing. Technical Report. Carnegie Mellon University Software Engineering Institute.

[23]

Alexander Ivrii, Sharad Malik, Kuldeep S Meel, and Moshe Y Vardi. 2016. On computing minimal independent support and its applications to sampling and counting. Constraints 21, 1 (2016), 41--58.

Digital Library

[24]

Karthick Jayaraman, David Harvison, Vijay Ganesh, and Adam Kiezun. 2009. jFuzz: A Concolic Whitebox Fuzzer for Java. In In NFM'09.

[25]

James C.King. 1976. Symbolic execution and program testing. Commun. ACM 19 (July 1976), 385--394. Issue 7.

Digital Library

[26]

Nathan Kitchen and Andreas Kuehlmann. 2007. Stimulus generation for constrained random simulation. In Computer-Aided Design, 2007. ICCAD 2007. IEEE/ACM International Conference on. IEEE, 258--265.

Digital Library

[27]

Nathan Boyd Kitchen. 2010. Markov Chain Monte Carlo Stimulus Generation for Constrained Random Simulation. University of California, Berkeley.

[28]

Guodong Li, Indradeep Ghosh, and Sreeranga P. Rajan. 2011. KLOVER: A Symbolic Execution and Automatic Test Generation Tool for C++ Programs. In CAV. 609--615.

Digital Library

[29]

Kuldeep S Meel. 2014. Sampling techniques for boolean satisfiability. Master's thesis (2014).

[30]

Kuldeep S Meel, Moshe Y Vardi, Supratik Chakraborty, Daniel J Fremont, Sanjit A Seshia, Dror Fried, Alexander Ivrii, and Sharad Malik. 2016. Constrained Sampling and Counting: Universal Hashing Meets SAT Solving. In AAAI Workshop: Beyond NP.

[31]

Alexander Nadel. 2011. Generating Diverse Solutions in SAT. In SAT. Springer, 287--301.

Digital Library

[32]

Reuven Naveh and Amit Metodi. 2013. Beyond feasibility: CP usage in constrained-random functional hardware verification. In International Conference on Principles and Practice of Constraint Programming. Springer, 823--831.

[33]

Yehuda Naveh, Michal Rimon, Itai Jaeger, Yoav Katz, Michael Vinov, Eitan s Marcu, and Gil Shurek. 2007. Constraint-based random stimuli generation for hardware verification. AI magazine 28, 3 (2007), 13.

[34]

Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. 2007. Feedback-directed random test generation. In ICSE'07, Proceedings of the 29th International Conference on Software Engineering. Minneapolis, MN, USA, 75--84.

Digital Library

[35]

C. Pasareanu, P. Mehlitz, D. Bushnell, K. Gundy-Burlet, M. Lowry, S. Person, and M. Pape. 2008. Combining Unit-level Symbolic Execution and System-level Concrete Execution for Testing NASA Software. In ISSTA'08.

Digital Library

[36]

Prateek Saxena, Devdatta Akhawe, Steve Hanna, Feng Mao, Stephen McCamant, and Dawn Song. 2010. A Symbolic Execution Framework for JavaScript. In Proceedings of the 2010 IEEE Symposium on Security and Privacy (SP '10). IEEE, 513--528.

Digital Library

[37]

Koushik Sen and Gul Agha. 2006. CUTE and jCUTE : Concolic Unit Testing and Explicit Path Model-Checking Tools. In CAV'06.

Digital Library

[38]

Koushik Sen, Swaroop Kalasapur, Tasneem Brutch, and Simon Gibbs. 2013. Jalangi: A Selective Record-Replay and Dynamic Analysis Framework for JavaScript. In ESEC/FSE'13. To appear.

Digital Library

[39]

Koushik Sen, Darko Marinov, and Gul Agha. 2005. CUTE: A Concolic Unit Testing Engine for C. In ESEC/FSE'05.

Digital Library

[40]

Dawn Song, David Brumley, Heng Yin, Juan Caballero, Ivan Jager, Min Gyung Kang, Zhenkai Liang, James Newsome, Pongsin Poosankam, and Prateek Saxena. 2008. BitBlaze: A New Approach to Computer Security via Binary Analysis. In ICISS'08.

Digital Library

[41]

Marc Thurley. 2006. sharpSAT-counting models with advanced component caching and implicit BCP. SAT 4121 (2006), 424--429.

Digital Library

[42]

Nikolai Tillmann and Jonathan de Halleux. 2008. Pex - White Box Test Generation for .NET. In TAP'08.

Digital Library

[43]

Wei Wei, Jordan Erenrich, and Bart Selman. 2004. Towards efficient sampling: Exploiting random walk strategies. In AAAI, Vol. 4. 670--676.

Digital Library

[44]

Wei Wei and Bart Selman. 2005. A new approach to model counting. In SAT. Springer, 324--339.

Digital Library

[45]

Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and Understanding Bugs in C Compilers. In Proceedings of the 32Nd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI '11). ACM, New York, NY, USA, 283--294.

Digital Library

[46]

MichaÅĆ Zalewski. {n. d.}. American Fuzzy Lop. http://lcamtuf.coredump.cx/afl. ({n. d.}). Accessed October 1, 2016.

[47]

Yanni Zhao, Jinian Bian, Shujun Deng, and Zhiqiu Kong. 2009. Random stimulus generation with self-tuning. In Computer Supported Cooperative Work in Design, 2009. CSCWD 2009. 13th International Conference on. IEEE, 62--65.

Digital Library

Cited By

Sundermann CLoth JThüm TFilkov VRay BZhou M(2024)Efficient Slicing of Feature Models via Projected d-DNNF CompilationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695594(1332-1344)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695594
Fernandez-Amoros DHeradio RHorcas Aguilera JGalindo JBenavides DFuentes L(2024)Pragmatic Random Sampling of the Linux Kernel: Enhancing the Randomness and Correctness of the conf ToolProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672586(24-35)Online publication date: 2-Sep-2024
https://dl.acm.org/doi/10.1145/3646548.3672586
Zeyen OCordy MPerrouin GAcher MFuria CLopes APlat NGnesi S(2024)Preprocessing is What You Need: Understanding and Predicting the Complexity of SAT-based Uniform Random SamplingProceedings of the 2024 IEEE/ACM 12th International Conference on Formal Methods in Software Engineering (FormaliSE)10.1145/3644033.3644371(23-32)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3644033.3644371
Show More Cited By

Index Terms

Efficient sampling of SAT solutions for testing
1. Hardware
  1. Hardware validation
    1. Functional verification
      1. Theorem proving and SAT solving
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Formal software verification
      2. Software defect analysis
        Software testing and debugging

Recommendations

Search-based testing using constraint-based mutation

Many modern automated test generators are based on either metaheuristic search techniques or use constraint solvers. Both approaches have their advantages, but they also have specific drawbacks: Search-based methods may get stuck in local optima and ...
Search-based testing and system testing: a marriage in heaven
SBST '17: Proceedings of the 10th International Workshop on Search-Based Software Testing

Software test generation can take place at the function level or the system level; and be driven by random, constraint-based, and/or search-based techniques. In this paper, we argue that the best way to generate tests is at the system level, as it ...
Lightweight Data-Flow Analysis for Execution-Driven Constraint Solving
ICST '12: Proceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation

Constraint-based testing is a methodology for finding bugs in code, which has been successfully used for testing real systems. A key element of the methodology is generation of test inputs from input constraints, i.e., properties of desired inputs, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '18: Proceedings of the 40th International Conference on Software Engineering

May 2018

1307 pages

ISBN:9781450356381

DOI:10.1145/3180155

Conference Chair:
Michel Chaudron
Chalmers University of Technology, University of Gothenburg, Sweden
,
General Chair:
Ivica Crnkovic
Chalmers University of Technology, University of Gothenburg, Sweden
,
Program Chairs:
Marsha Chechik
University of Toronto, Canada
,
Mark Harman
Facebook and University College London, United Kingdom

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 May 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

CAPES
DARPA CRAFT
NSF

Conference

ICSE '18

Sponsor:

SIGSOFT
IEEE-CS

ICSE '18: 40th International Conference on Software Engineering

May 27 - June 3, 2018

Gothenburg, Sweden

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

48
Total Citations
View Citations
744
Total Downloads

Downloads (Last 12 months)184
Downloads (Last 6 weeks)28

Reflects downloads up to 12 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sundermann CLoth JThüm TFilkov VRay BZhou M(2024)Efficient Slicing of Feature Models via Projected d-DNNF CompilationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695594(1332-1344)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695594
Fernandez-Amoros DHeradio RHorcas Aguilera JGalindo JBenavides DFuentes L(2024)Pragmatic Random Sampling of the Linux Kernel: Enhancing the Randomness and Correctness of the conf ToolProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672586(24-35)Online publication date: 2-Sep-2024
https://dl.acm.org/doi/10.1145/3646548.3672586
Zeyen OCordy MPerrouin GAcher MFuria CLopes APlat NGnesi S(2024)Preprocessing is What You Need: Understanding and Predicting the Complexity of SAT-based Uniform Random SamplingProceedings of the 2024 IEEE/ACM 12th International Conference on Formal Methods in Software Engineering (FormaliSE)10.1145/3644033.3644371(23-32)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3644033.3644371
Heß TSchmidt TOstheimer LKrieter SThüm T(2024)UnWise: High T-Wise Coverage from Uniform SamplingProceedings of the 18th International Working Conference on Variability Modelling of Software-Intensive Systems10.1145/3634713.3634716(37-45)Online publication date: 7-Feb-2024
https://dl.acm.org/doi/10.1145/3634713.3634716
Baranov EChakraborty SLegay AMeel KVinodchandran N(2024)A Scalable $t$t-Wise Coverage Estimator: Algorithms and ApplicationsIEEE Transactions on Software Engineering10.1109/TSE.2024.341991950:8(2021-2039)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3419919
Baranov ELegay A(2024)Baital: Sampling Configurable Systems with high t-wise coverageScience of Computer Programming10.1016/j.scico.2024.103209(103209)Online publication date: Sep-2024
https://doi.org/10.1016/j.scico.2024.103209
Entekhabi SMostowski WMousavi M(2023)Automated and Efficient Test-Generation for Grid-Based Multiagent Systems: Comparing Random Input Filtering versus Constraint SolvingACM Transactions on Software Engineering and Methodology10.1145/362473633:1(1-32)Online publication date: 23-Nov-2023
https://dl.acm.org/doi/10.1145/3624736
Wang ZNie PMiao XChen YWan CBu LZhao JJust RFraser G(2023)GenCoG: A DSL-Based Approach to Generating Computation Graphs for TVM TestingProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598105(904-916)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598105
Nikfarjam ARothenberger RNeumann FFriedrich TSilva SPaquete L(2023)Evolutionary Diversity Optimisation in Constructing Satisfying AssignmentsProceedings of the Genetic and Evolutionary Computation Conference10.1145/3583131.3590517(938-945)Online publication date: 15-Jul-2023
https://dl.acm.org/doi/10.1145/3583131.3590517
Ammermann JBittner TEichhorn DSchaefer ISeidl C(2023)Can Quantum Computing Improve Uniform Random Sampling of Large Configuration Spaces?2023 IEEE/ACM 4th International Workshop on Quantum Software Engineering (Q-SE)10.1109/Q-SE59154.2023.00012(34-41)Online publication date: May-2023
https://doi.org/10.1109/Q-SE59154.2023.00012
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten