research-article

Enhancing symbolic execution with veritesting

Authors:

Thanassis Avgerinos,

Alexandre Rebert,

David BrumleyAuthors Info & Claims

ICSE 2014: Proceedings of the 36th International Conference on Software Engineering

Pages 1083 - 1094

https://doi.org/10.1145/2568225.2568293

Published: 31 May 2014 Publication History

Abstract

We present MergePoint, a new binary-only symbolic execution system for large-scale and fully unassisted testing of commodity off-the-shelf (COTS) software. MergePoint introduces veritesting, a new technique that employs static symbolic execution to amplify the effect of dynamic symbolic execution. Veritesting allows MergePoint to find twice as many bugs, explore orders of magnitude more paths, and achieve higher code coverage than previous dynamic symbolic execution systems. MergePoint is currently running daily on a 100 node cluster analyzing 33,248 Linux binaries; has generated more than 15 billion SMT queries, 200 million test cases, 2,347,420 crashes, and found 11,687 bugs in 4,379 distinct applications.

References

[1]

Online Bibliography for Symbolic Execution. http:// sites.google.com/site/symexbib.

[2]

A. V. Aho, R. Sethi, and J. D. Ullman. Compilers: Principles, Techniques, and Tools. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 1986.

Digital Library

[3]

J. R. Allen, K. Kennedy, C. Porterfield, and J. Warren. Conversion of Control Dependence to Data Dependence. In Proceedings of the 10th ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages, pages 177–189, New York, NY, USA, 1983. ACM Press.

Digital Library

[4]

S. Anand, P. Godefroid, and N. Tillmann. Demand-Driven Compositional Symbolic Execution. In Proceedings of the Theory and Practice of Software, 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, pages 367–381, Berlin, Heidelberg, 2008. Springer-Verlag.

Digital Library

[5]

D. Babic. Exploiting structure for scalable software verification. PhD thesis, University of British Columbia, Vancouver, Canada, 2008.

[6]

D. Babic and A. J. Hu. Calysto: Scalable and Precise Extended Static Checking. In Proceedings of the 30th International Conference on Software Engineering, pages 211–220, New York, NY, USA, 2008. ACM.

Digital Library

[7]

S. Bardin, P. Herrmann, J. Leroux, O. Ly, R. Tabary, and A. Vincent. The BINCOA Framework for Binary Code Analysis. In Proceedings of the 23rd International Conference on Computer Aided Verification, pages 165– 170, Berlin, Heidelberg, 2011. Springer-Verlag.

Digital Library

[8]

D. Beyer, T. A. Henzinger, and G. Theoduloz. Configurable Software Verification: Concretizing the Convergence of Model Checking and Program Analysis. In Proceedings of the 19th International Conference on Computer Aided Verification, pages 504–518, Berlin, Heidelberg, 2007. Springer-Verlag.

Digital Library

[9]

P. Boonstoppel, C. Cadar, and D. Engler. RWset: Attacking Path Explosion in Constraint-Based Test Generation. In Proceedings of the Theory and Practice of Software, 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, pages 351–366, Berlin, Heidelberg, 2008. Springer-Verlag.

Digital Library

[10]

E. Bounimova, P. Godefroid, and D. Molnar. Billions and Billions of Constraints: Whitebox Fuzz Testing in Production. In Proceedings of the 35th IEEE International Conference on Software Engineering, pages 122–131, Piscataway, NJ, USA, 2013. IEEE Press.

Digital Library

[11]

R. S. Boyer, B. Elspas, and K. N. Levitt. SELECT—a formal system for testing and debugging programs by symbolic execution. ACM SIGPLAN Notices, 10(6): 234–245, 1975.

Digital Library

[12]

D. Brumley, I. Jager, T. Avgerinos, and E. J. Schwartz. BAP: A Binary Analysis Platform. In Proceedings of the 23rd International Conference on Computer Aided Verification, pages 463–469. Springer-Verlag, 2011.

Digital Library

[13]

S. Bucur, V. Ureche, C. Zamfir, and G. Candea. Parallel symbolic execution for automated real-world software testing. In Proceedings of the 6th ACM SIGOPS European Conference on Computer Systems, pages 183–198. ACM Press, 2011.

Digital Library

[14]

C. Cadar and K. Sen. Symbolic execution for software testing: three decades later. Communications of the ACM, 56(2):82–90, 2013.

Digital Library

[15]

C. Cadar, V. Ganesh, P. M. Pawlowski, D. L. Dill, and D. R. Engler. EXE : Automatically Generating Inputs of Death. In Proceedings of the 13th ACM Conference on Computer and Communications Security, New York, NY, USA, 2006. ACM.

Digital Library

[16]

C. Cadar, D. Dunbar, and D. Engler. KLEE: Unassisted and Automatic Generation of High-coverage Tests for Complex Systems Programs. In Proceedings of the 8th USENIX Symposium on Operating System Design and Implementation, pages 209–224, Berkeley, CA, USA, 2008. USENIX Association.

Digital Library

[17]

C. Cadar, D. Dunbar, and D. R. Engler. KLEE Coreutils Experiment. http://klee.github.io/klee/ CoreutilsExperiments.html, 2008.

[18]

S. K. Cha, T. Avgerinos, A. Rebert, and D. Brumley. Unleashing Mayhem on Binary Code. In Proceedings of the 33rd IEEE Symposium on Security and Privacy, pages 380–394, Washington, DC, USA, 2012. IEEE Computer Society.

Digital Library

[19]

V. Chipounov, V. Kuznetsov, and G. Candea. S2E: A platform for in-vivo multi-path analysis of software systems. In Proceedings of the 16th International Conference on Architectural Support for Programming Languages and Operating Systems, pages 265–278, New York, NY, USA, 2011. ACM.

Digital Library

[20]

P. Collingbourne, C. Cadar, and P. H. Kelly. Symbolic crosschecking of floating-point and SIMD code. Proceedings of the 6th ACM SIGOPS European conference on Computer Systems, pages 315–328, 2011.

Digital Library

[21]

L. De Moura and N. Bjørner. Z3: An Eﬃcient SMT Solver. In Proceedings of 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, pages 337–340, Berlin, Heidelberg, 2008. Springer-Verlag.

[22]

I. Dillig, T. Dillig, and A. Aiken. Sound, Complete and Scalable Path-Sensitive Analysis. In Proceedings of the 29th ACM Conference on Programming Language Design and Implementation, pages 270–280, New York, NY, USA, 2008. ACM.

Digital Library

[23]

J. Filliˆ atre and S. Conchon. Type-safe modular hashconsing. In Proceedings of the Workshop on ML, pages 12–19, New York, NY, USA, 2006. ACM.

Digital Library

[24]

C. Flanagan and J. Saxe. Avoiding exponential explosion: Generating compact verification conditions. In Proceedings of the 28th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, pages 193–205, New York, NY, USA, 2001. ACM.

Digital Library

[25]

P. Godefroid. Compositional Dynamic Test Generation. In Proceedings of the 34th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, pages 47–54, New York, NY, USA, 2007. ACM.

Digital Library

[26]

P. Godefroid, N. Klarlund, and K. Sen. DART : Directed Automated Random Testing. In Proceedings of the 26th ACM Conference on Programming Language Design and Implementation, New York, NY, USA, 2005. ACM.

Digital Library

[27]

P. Godefroid, M. Y. Levin, and D. Molnar. Automated Whitebox Fuzz Testing. In Proceedings of the 15th Network and Distributed System Security Symposium. The Internet Society, 2008.

[28]

P. Godefroid, M. Y. Levin, and D. Molnar. SAGE: Whitebox Fuzzing for Security Testing. Communications of the ACM, 55(3):40–44, 2012.

Digital Library

[29]

E. Goto. Monocopy and Associative Algorithms in Extended Lisp. Technical Report TR-74-03, University of Tokyo, 1974.

[30]

T. Hansen, P. Schachte, and H. Søndergaard. State Joining and Splitting for the Symbolic Execution of Binaries. Runtime Verification, pages 76–92, 2009.

Digital Library

[31]

W. Howden. Methodology for the Generation of Program Test Data. IEEE Transactions on Computers, C-24(5):554–560, 1975.

Digital Library

[32]

J. Kinder and H. Veith. Jakstab: A Static Analysis Platform for Binaries. In Proceedings of the 20th International Conference on Computer Aided Verification, pages 423–427, Berlin, Heidelberg, 2008. Springer-Verlag.

Digital Library

[33]

J. C. King. Symbolic execution and program testing. Communications of the ACM, 19(7):385–394, 1976.

Digital Library

[34]

A. Koelbl and C. Pixley. Constructing Eﬃcient Formal Models from High-Level Descriptions Using Symbolic Simulation. International Journal of Parallel Programming, 33(6):645–666, Dec. 2005.

Digital Library

[35]

V. Kuznetsov, J. Kinder, S. Bucur, and G. Candea. Eﬃcient state merging in symbolic execution. In Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 193–204, New York, NY, USA, 2012. ACM.

Digital Library

[36]

C. Lattner and V. Adve. LLVM: A compilation framework for lifelong program analysis & transformation. In Proceedings of the International Symposium on Code Generation and Optimization: Feedback-directed and Runtime Optimization, pages 75–86, Washington, DC, USA, 2004. IEEE Computer Society.

Digital Library

[37]

K. R. M. Leino. Eﬃcient weakest preconditions. Information Processing Letters, 93(6):281–288, 2005.

Digital Library

[38]

C.-K. Luk, R. Cohn, R. Muth, H. Patil, A. Klauser, G. Lowney, S. Wallace, V. J. Reddi, and K. Hazelwood. Pin: Building Customized Program Analysis Tols with Dynamic Instrumentation. In Proceedings of the 26th ACM Conference on Programming Language Design and Implementation, pages 190–200, New York, NY, USA, 2005. ACM.

Digital Library

[39]

P. D. Marinescu and C. Cadar. Make test-zesti: A symbolic execution solution for improving regression testing. In Proceedings of the 34th International Conference on Software Engineering, pages 716–726, Piscataway, NJ, USA, 2012. IEEE Press.

Digital Library

[40]

Mayhem. 1.2K Crashes in Debian, 2013. URL http://lists.debian.org/debian-devel/2013/06/ msg00720.html.

[41]

Mayhem. Open Source Statistics & Analysis, 2013. URL http://www.forallsecure.com/summaries.

[42]

D. Molnar, X. Li, and D. Wagner. Dynamic test generation to find integer bugs in x86 binary linux programs. In Proceedings of the USENIX Security Symposium, pages 67–82, 2009.

Digital Library

[43]

C. S. Păsăreanu and W. Visser. A survey of new trends in symbolic execution for software testing and analysis. International Journal on Software Tools for Technology Transfer, 11(4):339–353, Aug. 2009.

Digital Library

[44]

A. J. Romano. Linux Bug Release, July 2013. URL http://www.bugsdujour.com/release/.

[45]

E. J. Schwartz, T. Avgerinos, and D. Brumley. All You Ever Wanted to Know about Dynamic Taint Analysis and Forward Symbolic Execution (but Might Have Been Afraid to Ask). In Proceedings of the 31st IEEE Symposium on Security and Privacy, pages 317–331, Washington, DC, USA, 2010. IEEE Computer Society.

Digital Library

[46]

K. Sen, D. Marinov, and G. Agha. CUTE: A Concolic Unit Testing Engine for C. In Proceedings of the 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 263–272, New York, NY, USA, 2005. ACM.

Digital Library

[47]

P. Tu and D. Padua. Eﬃcient building and placing of gating functions. In Proceedings of the 16th ACM Conference on Programming Language Design and Implementation, pages 47–55, New York, NY, USA, 1995. ACM.

Digital Library

[48]

Y. Xie and A. Aiken. Scalable error detection using boolean satisfiability. In Proceedings of the 32nd ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, pages 351–363, New York, NY, USA, 2005. ACM.

Digital Library

[49]

H. Zhu, P. A. V. Hall, and J. H. R. May. Software unit test coverage and adequacy. ACM Computing Surveys, 29(4):366–427, 1997.

Digital Library

Cited By

Alshmrany KAldughaim MBhayat ACordeiro L(2024)FuSeBMC v4: Improving Code Coverage with Smart Seeds via BMC, Fuzzing and Static AnalysisFormal Aspects of Computing10.1145/366533736:2(1-25)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3665337
Jain RTihanyi NNdhlovu MFerrag MCordeiro Ld'Amorim M(2024)Rapid Taint Assisted Concolic Execution (TACE)Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663812(627-631)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663812
Yadav AWilson JHuyen PTan SMechtaev SKhurshid S(2024)BOSS: A dataset to train ML-based systems to repair programs with out-of-bounds write flawsProceedings of the 5th ACM/IEEE International Workshop on Automated Program Repair10.1145/3643788.3648013(26-33)Online publication date: 20-Apr-2024
https://dl.acm.org/doi/10.1145/3643788.3648013
Show More Cited By

Index Terms

Enhancing symbolic execution with veritesting
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
2. Theory of computation
  1. Semantics and reasoning
    1. Program semantics

Recommendations

Veritesting Challenges in Symbolic Execution of Java

Scaling symbolic execution to industrial-sized programs is an important open research problem. Veritesting is a promising technique that improves scalability by combining the advantages of static symbolic execution with those of dynamic symbolic ...
Verifying systems rules using rule-directed symbolic execution
ASPLOS '13

Systems code must obey many rules, such as "opened files must be closed." One approach to verifying rules is static analysis, but this technique cannot infer precise runtime effects of code, often emitting many false positives. An alternative is ...
Verifying systems rules using rule-directed symbolic execution
ASPLOS '13

Systems code must obey many rules, such as "opened files must be closed." One approach to verifying rules is static analysis, but this technique cannot infer precise runtime effects of code, often emitting many false positives. An alternative is ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE 2014: Proceedings of the 36th International Conference on Software Engineering

May 2014

1139 pages

ISBN:9781450327565

DOI:10.1145/2568225

General Chair:
Pankaj Jalote
IIIT-Delhi, India
,
Program Chairs:
Lionel Briand
University of Luxembourg, Luxembourg
,
André van der Hoek
University of California, Irvine, USA

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

TCSE: IEEE Computer Society's Tech. Council on Software Engin.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 May 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '14

Sponsor:

SIGSOFT

ICSE '14: 36th International Conference on Software Engineering

May 31 - June 7, 2014

Hyderabad, India

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

156
Total Citations
View Citations
1,776
Total Downloads

Downloads (Last 12 months)165
Downloads (Last 6 weeks)13

Reflects downloads up to 22 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Alshmrany KAldughaim MBhayat ACordeiro L(2024)FuSeBMC v4: Improving Code Coverage with Smart Seeds via BMC, Fuzzing and Static AnalysisFormal Aspects of Computing10.1145/366533736:2(1-25)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3665337
Jain RTihanyi NNdhlovu MFerrag MCordeiro Ld'Amorim M(2024)Rapid Taint Assisted Concolic Execution (TACE)Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663812(627-631)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663812
Yadav AWilson JHuyen PTan SMechtaev SKhurshid S(2024)BOSS: A dataset to train ML-based systems to repair programs with out-of-bounds write flawsProceedings of the 5th ACM/IEEE International Workshop on Automated Program Repair10.1145/3643788.3648013(26-33)Online publication date: 20-Apr-2024
https://dl.acm.org/doi/10.1145/3643788.3648013
Kim JGustaman SCha S(2024)PoE: A Domain-Specific Language for Exploitation2024 Silicon Valley Cybersecurity Conference (SVCC)10.1109/SVCC61185.2024.10637307(1-6)Online publication date: 17-Jun-2024
https://doi.org/10.1109/SVCC61185.2024.10637307
Haltermann JJakobs MRichter CWehrheim H(2024)Parallel Program Analysis on Path RangesScience of Computer Programming10.1016/j.scico.2024.103154(103154)Online publication date: May-2024
https://doi.org/10.1016/j.scico.2024.103154
Haltermann JWehrheim H(2024)Exchanging information in cooperative software validationSoftware and Systems Modeling (SoSyM)10.1007/s10270-024-01155-323:3(695-719)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s10270-024-01155-3
Zhao XQu HXu JLi XLv WWang G(2024)A systematic review of fuzzingSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-023-09306-228:6(5493-5522)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1007/s00500-023-09306-2
Mues MRüschoff JHermann B(2024)Exploring Loose Coupling of Slicing with Dynamic Symbolic Execution on the JVMTests and Proofs10.1007/978-3-031-72044-4_9(168-175)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-72044-4_9
Verma AChoudhary ATiwari S(2023)Software Test Case Generation Tools and Techniques: A ReviewInternational Journal of Mathematical, Engineering and Management Sciences10.33889/IJMEMS.2023.8.2.0188:2(293-315)Online publication date: 1-Apr-2023
https://doi.org/10.33889/IJMEMS.2023.8.2.018
Geretto EHohnerlein JGiuffrida CBos Hvan der Kouwe Ev. Gleissenthall K(2023)Triereme: Speeding up hybrid fuzzing through efficient query schedulingProceedings of the 39th Annual Computer Security Applications Conference10.1145/3627106.3627173(56-70)Online publication date: 4-Dec-2023
https://dl.acm.org/doi/10.1145/3627106.3627173
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents