Article

QSYM: a practical concolic execution engine tailored for hybrid fuzzing

Authors:

Taesoo KimAuthors Info & Claims

SEC'18: Proceedings of the 27th USENIX Conference on Security Symposium

Pages 745 - 761

Published: 15 August 2018 Publication History

Abstract

Recently, hybrid fuzzing has been proposed to address the limitations of fuzzing and concolic execution by combining both approaches. The hybrid approach has shown its effectiveness in various synthetic benchmarks such as DARPA Cyber Grand Challenge (CGC) binaries, but it still suffers from scaling to find bugs in complex, realworld software. We observed that the performance bottleneck of the existing concolic executor is the main limiting factor for its adoption beyond a small-scale study.

To overcome this problem, we design a fast concolic execution engine, called QSYM, to support hybrid fuzzing. The key idea is to tightly integrate the symbolic emulation with the native execution using dynamic binary translation, making it possible to implement more fine-grained, so faster, instruction-level symbolic emulation. Additionally, QSYM loosens the strict soundness requirements of conventional concolic executors for better performance, yet takes advantage of a faster fuzzer for validation, providing unprecedented opportunities for performance optimizations, e.g., optimistically solving constraints and pruning uninteresting basic blocks.

Our evaluation shows that QSYM does not just outperform state-of-the-art fuzzers (i.e., found 14× more bugs than VUzzer in the LAVA-M dataset, and outperformed Driller in 104 binaries out of 126), but also found 13 previously unknown security bugs in eight real-world programs like Dropbox Lepton, ffmpeg, and OpenJPEG, which have already been intensively tested by the state-of-the-art fuzzers, AFL and OSS-Fuzz.

References

[1]

M. Zalewski, "american fuzzy lop," http://lcamtuf.coredump.cx/afl/, 2015.

[2]

Google, "honggfuzz," https://github.com/google/honggfuzz, 2010.

[3]

Google, "OSS-Fuzz - continuous fuzzing of open source software," https://github.com/google/ossfuzz, 2016.

[4]

P. Godefroid, M. Y. Levin, and D. A. Molnar, "Automated whitebox fuzz testing," in Proceedings of the 15th Annual Network and Distributed System Security Symposium (NDSS), San Diego, CA, Feb. 2008.

[5]

V. Chipounov, V. Kuznetsov, and G. Candea, "S2E: A platform for in-vivo multi-path analysis of software systems," in Proceedings of the 16th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), Newport Beach, CA, Mar. 2011.

[6]

R. Majumdar and K. Sen, "Hybrid Concolic Testing," in Proceedings of the 29th International Conference on Software Engineering (ICSE), Minneapolis, MN, May 2007.

[7]

B. S. Pak, "Hybrid fuzz testing: Discovering software bugs via fuzzing and symbolic execution," Master's thesis, Carnegie Mellon University Pittsburgh, PA, 2012.

[8]

N. Stephens, J. Grosen, C. Salls, A. Dutcher, R. Wang, J. Corbetta, Y. Shoshitaishvili, C. Kruegel, and G. Vigna, "Driller: Augmenting fuzzing through selective symbolic execution," in Proceedings of the 2016 Annual Network and Distributed System Security Symposium (NDSS), San Diego, CA, Feb. 2016.

[9]

S. Rawat, V. Jain, A. Kumar, L. Cojocar, C. Giuffrida, and H. Bos, "VUzzer: Application-aware evolutionary fuzzing," in Proceedings of the 2017 Annual Network and Distributed System Security Symposium (NDSS), San Diego, CA, Feb.-Mar. 2017.

[10]

B. Dolan-Gavitt, P. Hulin, E. Kirda, T. Leek, A. Mambretti, W. Robertson, F. Ulrich, and R. Whelan, "LAVA: Large-scale automated vulnerability addition," in Proceedings of the 37th IEEE Symposium on Security and Privacy (Oakland), San Jose, CA, May 2016.

[11]

Google, "Fuzzing for Security," https://blog.chromium.org/2012/04/fuzzing-for-security.html, 2012.

[12]

X. Leroy and D. Doligez, "mosml/md5sum.c at master," https://github.com/kfl/mosml/blob/master/src/runtime/md5sum.c, 2014.

[13]

S. Heule, E. Schkufza, R. Sharma, and A. Aiken, "Stratified synthesis: automatically learning the x86- 64 instruction set," in Proceedings of the 2016 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), Santa Barbara, CA, Jun. 2016.

[14]

Intel, "Intel?R 64 and ia-32 architectures software developer's manual," Volume 2: Instruction Set Reference, A-Z, 2016.

[15]

L. Project, "LLVM language reference manual," https://llvm.org/docs/LangRef.html#llvm-language-reference-manual, 2003.

[16]

N. Nethercote and J. Seward, "Valgrind: A Framework for Heavyweight Dynamic Binary Instrumentation," in Proceedings of the 2007 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), San Diego, CA, Jun. 2007.

[17]

R. David, S. Bardin, J. Feist, L. Mounier, M.-L. Potet, T. D. Ta, and J.-Y. Marion, "Specification of concretization and symbolization policies in symbolic execution." in Proceedings of the International Symposium on Software Testing and Analysis (ISSTA), Saarbrucken, Germany, Jul. 2016.

[18]

T. Liu, M. Araujo, M. d'Amorim, and M. Taghdiri, "A comparative study of incremental constraint solving approaches in symbolic execution," in Proceedings of the Haifa Verification Conference(HVC'14), Haifa, Israel, Nov. 2014.

[19]

P. Godefroid, N. Klarlund, and K. Sen, "DART: Directed Automated Random Testing," in Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), Chicago, IL, Jun. 2005.

[20]

Y. Shoshitaishvili, R. Wang, C. Salls, N. Stephens, M. Polino, A. Dutcher, J. Grosen, S. Feng, C. Hauser, C. Kruegel, and G. Vigna, "SoK: (State of) The Art of War: Offensive Techniques in Binary Analysis," in Proceedings of the 37th IEEE Symposium on Security and Privacy (Oakland), San Jose, CA, May 2016.

[21]

S. K. Cha, T. Avgerinos, A. Rebert, and D. Brumley, "Unleashing mayhem on binary code," in Proceedings of the 33rd IEEE Symposium on Security and Privacy (Oakland), San Francisco, CA, May 2012.

[22]

T. Bao, R. Wang, Y. Shoshitaishvili, and D. Brumley, "Your exploit is mine: Automatic shellcode transplant for remote exploits," in Proceedings of the 38th IEEE Symposium on Security and Privacy (Oakland), San Jose, CA, May 2017.

[23]

J. Hendrix and B. F. Jones, "Bounded integer linear constraint solving via lattice search," in Proceedings of the International Workshop on Satisfiability Modulo Theories, 2015.

[24]

C.-K. Luk, R. Cohn, R. Muth, H. Patil, A. Klauser, G. Lowney, S. Wallace, V. J. Reddi, and K. Hazelwood, "Pin: building customized program analysis tools with dynamic instrumentation," in Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), Chicago, IL, Jun. 2005.

[25]

K. Jee, G. Portokalidis, V. P. Kemerlis, S. Ghosh, D. I. August, and A. D. Keromytis, "A general approach for efficiently accelerating software-based dynamic data flow tracking on commodity hardware." in Proceedings of the 19th Annual Network and Distributed System Security Symposium (NDSS), San Diego, CA, Feb. 2012.

[26]

"CVE-2017-11543," https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-11543.

[27]

"CVE-2017-1000249," https://cve.mitre.org/cgibin/cvename.cgi?name=CVE-2017-1000249.

[28]

O. Chang, A. Arya, K. Serebryany, and J. Armour, "OSS-Fuzz: Five months later, and rewarding projects," https://opensource.googleblog.com/2017/05/oss-fuzz-five-months-later-and.html, 2017.

[29]

"PNG specification: Chunk specifications," https://www.w3.org/TR/PNG-Chunks.html, 1996.

[30]

DARPA, "Cyber Grand Challenge," https://www.cybergrandchallenge.com/, 2016.

[31]

Shellphish, "Shellphish AFL package," https://github.com/shellphish/shellphish-afl, 2016.

[32]

"Cppcheck: A tool for static C/C++ code analysis," http://cppcheck.sourceforge.net/.

[33]

M. Rajpal, W. Blum, and R. Singh, "Not all bytes are equal: Neural byte sieve for fuzzing," arXiv preprint arXiv:1711.04596, 2017.

[34]

M. Bohme, V.-T. Pham, and A. Roychoudhury, "Coverage-based Greybox Fuzzing as Markov Chain," in Proceedings of the 23rd ACM Conference on Computer and Communications Security (CCS), Vienna, Austria, Oct. 2016.

[35]

A. Reid, R. Chen, A. Deligiannis, D. Gilday, D. Hoyes, W. Keen, A. Pathirane, O. Shepherd, P. Vrabel, and A. Zaidi, "End-to-end verification of processors with isa-formal," in Proceedings of the 28th International Conference on Computer Aided Verification (CAV), Toronto, Canada, Jul. 2016.

[36]

S. Gan, C. Zhang, X. Qin, X. Tu, K. Li, Z. Pei, and Z. Chen, "CollAFL: Path sensitive fuzzing," in Proceedings of the 39th IEEE Symposium on Security and Privacy (Oakland), San Francisco, CA, May 2018.

[37]

Y. Li, B. Chen, M. Chandramohan, S.-W. Lin, Y. Liu, and A. Tiu, "Steelix: Program-State Based Binary Fuzzing," in Proceedings of the 11th ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE), Paderborn, Germany, Sep. 2017.

[38]

C. Lemieux and K. Sen, "FairFuzz: Targeting Rare Branches to Rapidly Increase Greybox Fuzz Testing Coverage," ArXiv e-prints, Sep. 2017.

[39]

P. Chen and H. Chen, "Angora: Efficient fuzzing by principled search," in Proceedings of the 39th IEEE Symposium on Security and Privacy (Oakland), San Francisco, CA, May 2018.

[40]

H. Peng, Y. Shoshitaishvili, and M. Payer, "T-Fuzz: fuzzing by program transformation," in Proceedings of the 39th IEEE Symposium on Security and Privacy (Oakland), San Francisco, CA, May 2018.

[41]

C. Cadar, V. Ganesh, P. M. Pawlowski, D. L. Dill, and D. R. Engler, "EXE: Automatically Generating Inputs of Death," in Proceedings of the 13th ACM Conference on Computer and Communications Security (CCS), Alexandria, VA, Oct.-Nov. 2006.

[42]

C. Cadar, D. Dunbar, and D. Engler, "KLEE: Unassisted and Automatic Generation of High-coverage Tests for Complex Systems Programs," in Proceedings of the 8th USENIX Symposium on Operating Systems Design and Implementation (OSDI), San Diego, CA, Dec. 2008.

[43]

L. Martignoni, S. McCamant, P. Poosankam, D. Song, and P. Maniatis, "Path-exploration lifting: Hi-fi tests for lo-fi emulators," in Proceedings of the 18th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), Houston, TX, Mar. 2013.

[44]

E. Bounimova, P. Godefroid, and D. Molnar, "Billions and billions of constraints: Whitebox fuzz testing in production," in Proceedings of the 35th International Conference on Software Engineering (ICSE), San Francisco, CA, May 2013.

[45]

K. Sen, D. Marinov, and G. Agha, "CUTE: a concolic unit testing engine for C," in Proceedings of the 10th European Software Engineering Conference (ESEC) / 13th ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE), Lisbon, Portugal, Sep. 2005.

[46]

I. Haller, A. Slowinska, M. Neugschwandtner, and H. Bos, "Dowsing for Overflows: A Guided Fuzzer to Find Buffer Boundary Violations," in Proceedings of the 22th USENIX Security Symposium (Security), Washington, DC, Aug. 2013.

Cited By

Güler ESchumilo SSchloegel MBars NGörz PXu XKaygusuz CHolz TBalzarotti DXu W(2024)AtroposProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699167(4765-4782)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3699167
Xiang YZhang XLiu PJi SXiao XLiang HXu JWang WBalzarotti DXu W(2024)Critical code guided directed greybox fuzzing for commitsProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699038(2459-2474)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3699038
Qi ZHu JXiao ZYin HBalzarotti DXu W(2024)SYMFITProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3698924(415-431)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3698924
Show More Cited By

QSYM: a practical concolic execution engine tailored for hybrid fuzzing
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis

Recommendations

A Static Approach to Prioritizing JUnit Test Cases

Test case prioritization is used in regression testing to schedule the execution order of test cases so as to expose faults earlier in testing. Over the past few years, many test case prioritization techniques have been proposed in the literature. Most ...
Spartan: a spectral and entropy-based partial-scan and test point insertion algorithm
Faster mutation testing inspired by test prioritization and reduction
ISSTA 2013: Proceedings of the 2013 International Symposium on Software Testing and Analysis

Mutation testing is a well-known but costly approach for determining test adequacy. The central idea behind the approach is to generate mutants, which are small syntactic transformations of the program under test, and then to measure for a given test ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

SEC'18: Proceedings of the 27th USENIX Conference on Security Symposium

August 2018

1740 pages

ISBN:9781931971461

Program Chairs:
William Enck
North Carolina State University
,
Adrienne Porter Felt
Google

Sponsors

Google Inc.
Baidu Research: Baidu Research
NSF
Facebook: Facebook

Publisher

USENIX Association

United States

Publication History

Published: 15 August 2018

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

31
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Güler ESchumilo SSchloegel MBars NGörz PXu XKaygusuz CHolz TBalzarotti DXu W(2024)AtroposProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699167(4765-4782)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3699167
Xiang YZhang XLiu PJi SXiao XLiang HXu JWang WBalzarotti DXu W(2024)Critical code guided directed greybox fuzzing for commitsProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699038(2459-2474)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3699038
Qi ZHu JXiao ZYin HBalzarotti DXu W(2024)SYMFITProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3698924(415-431)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3698924
Bars NSchloegel MSchiller NBernhard LHolz TLuo BLiao XXu JKirda ELie D(2024)No Peer, no Cry: Network Application Fuzzing via Fault InjectionProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security10.1145/3658644.3690274(750-764)Online publication date: 2-Dec-2024
https://dl.acm.org/doi/10.1145/3658644.3690274
Fang HZhang KYu DZhang YChristakis MPradel M(2024)DDGF: Dynamic Directed Greybox Fuzzing with Path ProfilingProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680324(832-843)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680324
Wu WNongpoh BNour MMarcozzi MBardin SHauser C(2024)Fine-grained Coverage-based Fuzzing - RCR ReportACM Transactions on Software Engineering and Methodology10.1145/364959233:5(1-4)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1145/3649592
Wu WNongpoh BNour MMarcozzi MBardin SHauser C(2024)Fine-grained Coverage-based FuzzingACM Transactions on Software Engineering and Methodology10.1145/358715833:5(1-41)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1145/3587158
Kuliamin V(2024)A Survey of Software Dynamic Analysis MethodsProgramming and Computing Software10.1134/S036176882401007950:1(90-114)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1134/S0361768824010079
Kim JUjcich BTian DCalandrino JTroncoso C(2023)INTENDERProceedings of the 32nd USENIX Conference on Security Symposium10.5555/3620237.3620487(4463-4480)Online publication date: 9-Aug-2023
https://dl.acm.org/doi/10.5555/3620237.3620487
Scharnowski TWörner SBuchmann FBars NSchloegel MHolz TCalandrino JTroncoso C(2023)HOEDURProceedings of the 32nd USENIX Conference on Security Symposium10.5555/3620237.3620399(2885-2902)Online publication date: 9-Aug-2023
https://dl.acm.org/doi/10.5555/3620237.3620399
Show More Cited By

View Options

View options

Figures

Tables

Media

View Table of Conten