short-paper

A Feature-Oriented Corpus for Understanding, Evaluating and Improving Fuzz Testing

Authors:

Jingling XueAuthors Info & Claims

Asia CCS '19: Proceedings of the 2019 ACM Asia Conference on Computer and Communications Security

Pages 658 - 663

https://doi.org/10.1145/3321705.3329845

Published: 02 July 2019 Publication History

Abstract

Fuzzing is a promising technique for detecting security vulnerabilities. Newly developed fuzzers are typically evaluated in terms of the number of bugs found on vulnerable programs/binaries. However, existing corpora usually do not capture the features that prevent fuzzers from finding bugs, leading to ambiguous conclusions on the pros and cons of the fuzzers evaluated. In this paper, we propose to address the above problem by generating corpora based on search-hampering features. As a proof-of-concept, we designed FEData, a prototype corpus that currently focuses on three search-hampering features to generate vulnerable programs for fuzz testing. Unlike existing corpora that can only answer "how", FEData can also further answer "why" by exposing (or understanding) the reasons for the identified weaknesses in a fuzzer. The "why" information serves as the key to the improvement of fuzzers. Based on the "why" information, our FEData programs enabled us to identify the weakness of AFLFast, called cycle explosion, behind. We further developed an improved version of AFLFast, called AFLFast+, which has overcome the cycle explosion problem. AFLFast+ retains the efficiency of AFLFast in path search while maintaining or even surpassing the bug-finding capability of AFL for the corpus evaluated.

References

[1]

. 2017. Cyber Grand Challenge Corpus. http://www.lungetech.com/cgc-corpus/.

[2]

. 2018a. American Fuzzy Lop. http://lcamtuf.coredump.cx/afl/.

[3]

. 2018b. Fuzze Test Suite. https://github.com/google/fuzzer-test-suite.

[4]

. 2018c. National Institute of Standards and Technology. https://www.nist.gov.

[5]

. 2018 d. Standard Performance Evaluation Corporation. https://www.spec.org /benchmarks.html.

[6]

Stephen M Blackburn, Robin Garner, Chris Hoffmann, Asjad M Khang, Kathryn S McKinley, Rotem Bentzur, Amer Diwan, Daniel Feinberg, Daniel Frampton, Samuel Z Guyer, et almbox. 2006. The DaCapo benchmarks: Java benchmarking development and analysis. In ACM Sigplan Notices, Vol. 41(10). ACM, ACM, New York, NY, USA, 169--190.

Digital Library

[7]

Marcel Böhme, Van-Thuan Pham, and Abhik Roychoudhury. 2016. Coverage-based greybox fuzzing as markov chain. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. ACM, IEEE, Vienna, Austria, 1032--1043.

Digital Library

[8]

Peng Chen and Hao Chen. 2018. Angora: Efficient Fuzzing by Principled Search. In 2018 IEEE Symposium on Security and Privacy. IEEE, San Francisco, CA, USA, 711--725.

[9]

Jake Corina, Aravind Machiry, Christopher Salls, Yan Shoshitaishvili, Shuang Hao, Christopher Kruegel, and Giovanni Vigna. 2017. Difuze: interface aware fuzzing for kernel drivers. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. ACM, ACM, Dallas, Texas, USA, 2123--2138.

Digital Library

[10]

Brendan Dolan-Gavitt, Patrick Hulin, Engin Kirda, Tim Leek, Andrea Mambretti, Wil Robertson, Frederick Ulrich, and Ryan Whelan. 2016. LAVA: Large-scale automated vulnerability addition. In Security and Privacy, 2016 IEEE Symposium on. IEEE, IEEE, SAN JOSE, CA, USA, 110--121.

[11]

Jeffrey Foster. 2005. A call for a public bug and tool registry. In Workshop on the Evaluation of Software Defect Detection Tools .

[12]

Shuitao Gan, Chao Zhang, Xiaojun Qin, Xuwen Tu, Kang Li, Zhongyu Pei, and Zuoning Chen. 2018. CollAFL: Path sensitive fuzzing. In 2018 IEEE Symposium on Security and Privacy. IEEE, IEEE, San Francisco, CA, USA, 679--696.

[13]

Istvan Haller, Asia Slowinska, Matthias Neugschwandtner, and Herbert Bos. 2013. Dowsing for Overflows: A Guided Fuzzer to Find Buffer Boundary Violations. In Presented as part of the 22nd USENIX Security Symposium (USENIX Security 13). USENIX, Washington, D.C., 49--64.

Digital Library

[14]

George Klees, Andrew Ruef, Benji Cooper, Shiyi Wei, and Michael Hicks. 2018. Evaluating Fuzz Testing. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security. ACM, ACM, Toronto, Canada, 2123--2138.

Digital Library

[15]

Yuekang Li, Bihuan Chen, Mahinthan Chandramohan, Shang-Wei Lin, Yang Liu, and Alwen Tiu. 2017. Steelix: program-state based binary fuzzing. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. ACM, ACM, Paderborn, Germany, 627--637.

Digital Library

[16]

Shan Lu, Zhenmin Li, Feng Qin, Lin Tan, Pin Zhou, and Yuanyuan Zhou. 2005. Bugbench: Benchmarks for evaluating bug detection tools. In Workshop on the evaluation of software defect detection tools, Vol. 5.

[17]

Barton P Miller, Louis Fredriksen, and Bryan So. 1990. An empirical study of the reliability of UNIX utilities. Commun. ACM, Vol. 33, 12 (1990), 32--44.

Digital Library

[18]

Marius Muench, Jan Stijohann, Frank Kargl, Aurélien Francillon, and Davide Balzarotti. 2018. What you corrupt is not what you crash: Challenges in fuzzing embedded devices. In 25th Annual Network and Distributed System Security Symposium, NDSS 2018, San Diego, California, USA, February 18--21, 2018. The Internet Society, San Diego, CA, USA.

[19]

Hui Peng, Yan Shoshitaishvili, and Mathias Payer. 2018. T-Fuzz: fuzzing by program transformation. In 2018 IEEE Symposium on Security and Privacy. IEEE, IEEE, San Francisco, CA, USA, 697--710.

[20]

Van-Thuan Pham, Marcel Böhme, and Abhik Roychoudhury. 2016. Model-based whitebox fuzzing for program binaries. In Automated Software Engineering (ASE), 2016 31st IEEE/ACM International Conference on. IEEE, IEEE, Singapore, Singapore, 543--553.

Digital Library

[21]

Sanjay Rawat, Vivek Jain, Ashish Kumar, Lucian Cojocar, Cristiano Giuffrida, and Herbert Bos. 2017. Vuzzer: Application-aware evolutionary fuzzing. In 24th Annual Network and Distributed System Security Symposium, February 26 - March 1, 2017. The Internet Society, San Diego, California, USA.

[22]

Sergej Schumilo, Cornelius Aschermann, Robert Gawlik, Sebastian Schinzel, and Thorsten Holz. 2017. kAFL: Hardware-assisted feedback fuzzing for OS kernels. In 26th USENIX Security Symposium (USENIX Security 17). USENIX Association, Vancouver, BC, 167--182.

Digital Library

[23]

Koushik Sen. 2007. Concolic testing. In Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering. ACM, 571--572.

Digital Library

[24]

Jaime Spacco, David Hovemeyer, and William Pugh. 2005. Bug specimens are important. In Workshop on the Evaluation of Software Defect Detection Tools .

[25]

Nick Stephens, John Grosen, Christopher Salls, Andrew Dutcher, Ruoyu Wang, Jacopo Corbetta, Yan Shoshitaishvili, Christopher Kruegel, and Giovanni Vigna. 2016. Driller: Augmenting Fuzzing Through Selective Symbolic Execution. In 3rd Annual Network and Distributed System Security Symposium, NDSS 2016, San Diego, California, USA, February 21--24, 2016, Vol. 16. NDSS, The Internet Society, San Diego, California, USA, 1--16.

[26]

Junjie Wang, Bihuan Chen, Lei Wei, and Yang Liu. 2017. Skyfire: Data-driven seed generation for fuzzing. In 2017 IEEE Symposium on Security and Privacy. IEEE, IEEE, San Jose, CA, USA, 579--594.

[27]

John Wilander and Mariam Kamkar. 2003. A Comparison of Publicly Available Tools for Dynamic Buffer Overflow Prevention. In Proceedings of the Network and Distributed System Security Symposium, NDSS 2003, Vol. 3. The Internet Society, San Diego, California, USA, 149--162.

[28]

Misha Zitser, Richard Lippmann, and Tim Leek. 2004. Testing static analysis tools using exploitable buffer overflows from open source code. In Proceedings of the 12th ACM SIGSOFT International Symposium on Foundations of Software Engineering, Vol. 29(6). ACM, ACM, Newport Beach, CA, USA, 97--106.

Digital Library

Cited By

Zhu XLiu SJolfaei A(2024)A Fuzzing Method for Security Testing of SensorsIEEE Sensors Journal10.1109/JSEN.2023.330151724:5(5522-5529)Online publication date: 1-Mar-2024
https://doi.org/10.1109/JSEN.2023.3301517
Feng XZhu XHan QZhou WWen SXiang Y(2023)Detecting Vulnerability on IoT Device Firmware: A SurveyIEEE/CAA Journal of Automatica Sinica10.1109/JAS.2022.10586010:1(25-41)Online publication date: Jan-2023
https://doi.org/10.1109/JAS.2022.105860
Zhu XWen SCamtepe SXiang Y(2022)Fuzzing: A Survey for RoadmapACM Computing Surveys10.1145/351234554:11s(1-36)Online publication date: 9-Sep-2022
https://dl.acm.org/doi/10.1145/3512345
Show More Cited By

Recommendations

Evaluating Fuzz Testing
CCS '18: Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security

Fuzz testing has enjoyed great success at discovering security critical bugs in real software. Recently, researchers have devoted significant effort to devising new fuzzing techniques, strategies, and algorithms. Such new ideas are primarily evaluated ...
Improving Security Testing with Usage-Based Fuzz Testing
Revised Selected Papers of the Third International Workshop on Risk Assessment and Risk-Driven Testing - Volume 9488

Along with the increasing importance of software systems for our daily life, attacks on these systems may have a critical impact. Since the number of attacks and their effects increases the more systems are connected, the secure operation of IT systems ...
Turning programs against each other: high coverage fuzz-testing using binary-code mutation and dynamic slicing
ESEC/FSE 2015: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering

Mutation-based fuzzing is a popular and widely employed black-box testing technique for finding security and robustness bugs in software. It owes much of its success to its simplicity; a well-formed seed input is mutated, e.g. through random bit-...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

Asia CCS '19: Proceedings of the 2019 ACM Asia Conference on Computer and Communications Security

July 2019

708 pages

ISBN:9781450367523

DOI:10.1145/3321705

General Chairs:
Steven Galbraith
University of Auckland, New Zealand
,
Giovanni Russello
University of Auckland, New Zealand
,
Willy Susilo
University of Wollongong, Australia
,
Program Chairs:
Dieter Gollmann
Hamburg University of Technology, Germany & Nanyang Technological University, Singapore
,
Engin Kirda
Northeastern University, USA
,
Zhenkai Liang
National University of Singapore, Singapore

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSAC: ACM Special Interest Group on Security, Audit, and Control

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 July 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

Asia CCS '19

Sponsor:

SIGSAC

Asia CCS '19: ACM Asia Conference on Computer and Communications Security

July 9 - 12, 2019

Auckland, New Zealand

Acceptance Rates

Overall Acceptance Rate 418 of 2,322 submissions, 18%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
245
Total Downloads

Downloads (Last 12 months)39
Downloads (Last 6 weeks)4

Reflects downloads up to 12 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhu XLiu SJolfaei A(2024)A Fuzzing Method for Security Testing of SensorsIEEE Sensors Journal10.1109/JSEN.2023.330151724:5(5522-5529)Online publication date: 1-Mar-2024
https://doi.org/10.1109/JSEN.2023.3301517
Feng XZhu XHan QZhou WWen SXiang Y(2023)Detecting Vulnerability on IoT Device Firmware: A SurveyIEEE/CAA Journal of Automatica Sinica10.1109/JAS.2022.10586010:1(25-41)Online publication date: Jan-2023
https://doi.org/10.1109/JAS.2022.105860
Zhu XWen SCamtepe SXiang Y(2022)Fuzzing: A Survey for RoadmapACM Computing Surveys10.1145/351234554:11s(1-36)Online publication date: 9-Sep-2022
https://dl.acm.org/doi/10.1145/3512345
Zhu XWen SJolfaei AHaghighi MCamtepe SXiang Y(2022)Vulnerability Detection in SIoT Applications: A Fuzzing Method on their BinariesIEEE Transactions on Network Science and Engineering10.1109/TNSE.2020.30381429:3(970-979)Online publication date: 1-May-2022
https://doi.org/10.1109/TNSE.2020.3038142
Ren MYin ZMa FXu ZJiang YSun CLi HCai YCadar CZhang X(2021)Empirical evaluation of smart contract testing: what is the best choice?Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3460319.3464837(566-579)Online publication date: 11-Jul-2021
https://dl.acm.org/doi/10.1145/3460319.3464837
Zhu XWen SJolfaei ACamtepe SXiang Y(2021)Synthesized Corpora to Evaluate Fuzzing for Green Internet of Things ProgramsIEEE Transactions on Green Communications and Networking10.1109/TGCN.2021.30684665:3(1041-1050)Online publication date: Sep-2021
https://doi.org/10.1109/TGCN.2021.3068466
Riom TSawadogo AAllix KBissyandé TMoha NKlein J(2021)Revisiting the VCCFinder approach for the identification of vulnerability-contributing commitsEmpirical Software Engineering10.1007/s10664-021-09944-w26:3Online publication date: 29-Mar-2021
https://doi.org/10.1007/s10664-021-09944-w
Zhu XFeng XMeng XWen SCamtepe SXiang YRen K(2020)CSI-Fuzz: Full-speed Edge Tracing Using Coverage Sensitive InstrumentationIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2020.3008826(1-1)Online publication date: 2020
https://doi.org/10.1109/TDSC.2020.3008826

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents