research-article

Scheduling black-box mutational fuzzing

Authors:

Samantha Gottlieb,

David BrumleyAuthors Info & Claims

CCS '13: Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security

Pages 511 - 522

https://doi.org/10.1145/2508859.2516736

Published: 04 November 2013 Publication History

Abstract

Black-box mutational fuzzing is a simple yet effective technique to find bugs in software. Given a set of program-seed pairs, we ask how to schedule the fuzzings of these pairs in order to maximize the number of unique bugs found at any point in time. We develop an analytic framework using a mathematical model of black-box mutational fuzzing and use it to evaluate 26 existing and new randomized online scheduling algorithms. Our experiments show that one of our new scheduling algorithms outperforms the multi-armed bandit algorithm in the current version of the CERT Basic Fuzzing Framework (BFF) by finding 1.5x more unique bugs in the same amount of time.

References

[1]

A. Arcuri, M. Z. Iqbal, and L. Briand. Formal Analysis of the Effectiveness and Predictability of Random Testing. In International Symposium on Software Testing and Analysis, pages 219--229, 2010.

Digital Library

[2]

P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. The Nonstochastic Multiarmed Bandit Problem. Journal on Computing, 32(1):48--77, 2002.

Digital Library

[3]

P. Auer, N. Cesa-Bianchi, and F. Paul. Finite-time Analysis of the Multiarmed Bandit Problem. Machine Learning, 47(2--3):235--256, 2002.

Digital Library

[4]

T. Avgerinos, S. K. Cha, B. T. H. Lim, and D. Brumley. AEG: Automatic Exploit Generation. In Proceedings of the Network and Distributed Systems Security Symposium, 2011.

[5]

D. A. Berry and B. Fristedt. Bandit Problems: Sequential Allocation of Experiments. Chapman and Hall, 1985.

[6]

A. Bertolino. Software testing research: Achievements, challenges, dreams. In Future of Software Engineering, pages 85--103, 2007.

Digital Library

[7]

E. Bounimova, P. Godefroid, and D. Molnar. Billions and Billions of Constraints: Whitebox Fuzz Testing in Production. In Proceedings of the International Conference on Software Engineering, pages 122--131, 2013.

Digital Library

[8]

C. Cadar, D. Dunbar, and D. Engler. KLEE: Unassisted and Automatic Generation of High-coverage Tests for Complex Systems Programs. In Proceedings of the USENIX Symposium on Operating System Design and Implementation, pages 209--224, 2008.

Digital Library

[9]

S. K. Cha, T. Avgerinos, A. Rebert, and D. Brumley. Unleashing Mayhem on Binary Code. In Proceedings of the IEEE Symposium on Security and Privacy, pages 380--394, 2012.

Digital Library

[10]

D. Engler, D. Chen, S. Hallem, A. Chou, and B. Chelf. Bugs as Deviant Behavior: A General Approach to Inferring Errors in Systems Code. In Proceedings of the ACM Symposium on Operating System Principles, pages 57--72, 2001.

Digital Library

[11]

P. Godefroid, M. Y. Levin, and D. Molnar. SAGE: Whitebox Fuzzing for Security. Communications of the ACM, 55(3):40--44, 2012.

Digital Library

[12]

A. L. Goel. Software Reliability Models: Assumptions, Limitations, and Applicability. IEEE Transactions on Software Engineering, 11(12):1411--1423, 1985.

Digital Library

[13]

N. Gupta, A. P. Mathur, and M. L. Soffa. Automated Test Data Generation Using An Iterative Relaxation Method. In Proceedings of the ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 231--244, 1998.

Digital Library

[14]

A. D. Householder and J. M. Foote. Probability-Based Parameter Selection for Black-Box Fuzz Testing. Technical Report August, CERT, 2012.

[15]

B. D. Jovanovic and P. S. Levy. A Look at the Rule of Three. The American Statistician, 51(2):137--139, 1997.

[16]

C. Labs. zzuf: multi-purpose fuzzer. http://caca.zoy.org/wiki/zzuf.

[17]

R. McNally, K. Yiu, D. Grove, and D. Gerhardy. Fuzzing: The State of the Art. Technical Report DSTO--TN--1043, Defence Science and Technology Organisation, 2012.

[18]

B. P. Miller, L. Fredriksen, and B. So. An Empirical Study of the Reliability of UNIX Utilities. Communications of the ACM, 33(12):32--44, 1990.

Digital Library

[19]

D. Molnar, X. Li, and D. Wagner. Dynamic Test Generation To Find Integer Bugs in x86 Binary Linux Programs. In Proceedings of the USENIX Security Symposium, pages 67--82, 2009.

Digital Library

[20]

C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball. Feedback-Directed Random Test Generation. In Proceedings of the International Conference on Software Engineering, pages 75--84, 2007.

Digital Library

[21]

D. Wagner, J. S. Foster, E. A. Brewer, and A. Aiken. A First Step towards Automated Detection of Buffer Overrun Vulnerabilities. In Proceedings of the Network and Distributed Systems Security Symposium, pages 3--17, 2000.

[22]

D. Wolpert and W. Macready. No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1(1):67--82, 1997.

Digital Library

Cited By

Yang JLiu CFang B(2024)Adaptive scheduling-based fine-grained greybox fuzzing for cloud-native applicationsJournal of Cloud Computing10.1186/s13677-024-00681-113:1Online publication date: 26-Jun-2024
https://doi.org/10.1186/s13677-024-00681-1
Xu HChen LGan SZhang CLi ZJi JChen BHu F(2024)Graphuzz: Data-driven Seed Scheduling for Coverage-guided Greybox FuzzingACM Transactions on Software Engineering and Methodology10.1145/366460333:7(1-36)Online publication date: 26-Aug-2024
https://dl.acm.org/doi/10.1145/3664603
Luo SHerrera AQuirk PChase MRanasinghe DKanhere SQuek TGao DZhou JCardenas A(2024)Make out like a (Multi-Armed) Bandit: Improving the Odds of Fuzzer Seed Scheduling with T-SchedulerProceedings of the 19th ACM Asia Conference on Computer and Communications Security10.1145/3634737.3637639(1463-1479)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3634737.3637639
Show More Cited By

Index Terms

Scheduling black-box mutational fuzzing
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Introducing elitist black-box models: When does elitist behavior weaken the performance of evolutionary algorithms?

Black-box complexity theory provides lower bounds for the runtime of black-box optimizers like evolutionary algorithms and other search heuristics and serves as an inspiration for the design of new genetic algorithms. Several black-box models covering ...
Program-Adaptive Mutational Fuzzing
SP '15: Proceedings of the 2015 IEEE Symposium on Security and Privacy

We present the design of an algorithm to maximize the number of bugs found for black-box mutational fuzzing given a program and a seed input. The major intuition is to leverage white-box symbolic analysis on an execution trace for a given program-seed ...
OneMax in Black-Box Models with Several Restrictions
GECCO '15: Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation

As in classical runtime analysis the OneMax problem is the most prominent test problem also in black-box complexity theory. It is known that the unrestricted, the memory-restricted, and the ranking-based black-box complexities of this problem are all of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CCS '13: Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security

November 2013

1530 pages

ISBN:9781450324779

DOI:10.1145/2508859

General Chair:
Ahmad-Reza Sadeghi
TU Darmstadt, CASED, Intel ICRI-SC, Germany
,
Program Chairs:
Virgil Gligor
Carnegie Mellon University, USA
,
Moti Yung
Columbia University, USA

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSAC: ACM Special Interest Group on Security, Audit, and Control

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 November 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CCS'13

Sponsor:

SIGSAC

CCS'13: 2013 ACM SIGSAC Conference on Computer and Communications Security

November 4 - 8, 2013

Berlin, Germany

Acceptance Rates

CCS '13 Paper Acceptance Rate 105 of 530 submissions, 20%;

Overall Acceptance Rate 1,261 of 6,999 submissions, 18%

Upcoming Conference

CCS '24

Sponsor:
sigsac

ACM SIGSAC Conference on Computer and Communications Security

October 14 - 18, 2024

Salt Lake City , UT , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

114
Total Citations
View Citations
1,439
Total Downloads

Downloads (Last 12 months)126
Downloads (Last 6 weeks)13

Reflects downloads up to 01 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yang JLiu CFang B(2024)Adaptive scheduling-based fine-grained greybox fuzzing for cloud-native applicationsJournal of Cloud Computing10.1186/s13677-024-00681-113:1Online publication date: 26-Jun-2024
https://doi.org/10.1186/s13677-024-00681-1
Xu HChen LGan SZhang CLi ZJi JChen BHu F(2024)Graphuzz: Data-driven Seed Scheduling for Coverage-guided Greybox FuzzingACM Transactions on Software Engineering and Methodology10.1145/366460333:7(1-36)Online publication date: 26-Aug-2024
https://dl.acm.org/doi/10.1145/3664603
Luo SHerrera AQuirk PChase MRanasinghe DKanhere SQuek TGao DZhou JCardenas A(2024)Make out like a (Multi-Armed) Bandit: Improving the Odds of Fuzzer Seed Scheduling with T-SchedulerProceedings of the 19th ACM Asia Conference on Computer and Communications Security10.1145/3634737.3637639(1463-1479)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3634737.3637639
Kuliamin V(2024)A Survey of Software Dynamic Analysis MethodsProgramming and Computing Software10.1134/S036176882401007950:1(90-114)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1134/S0361768824010079
Gao ZXiong HDong WChang RYang RZhou YJiang L(2024)FA-Fuzz: A Novel Scheduling Scheme Using Firefly Algorithm for Mutation-Based FuzzingIEEE Transactions on Software Engineering10.1109/TSE.2023.332614450:1(1-15)Online publication date: Jan-2024
https://doi.org/10.1109/TSE.2023.3326144
Cheng HLi DZhao MLi HWong W(2024)A Comprehensive Review of Learning-based Fuzz Testing Techniques2024 10th International Symposium on System Security, Safety, and Reliability (ISSSR)10.1109/ISSSR61934.2024.00024(150-161)Online publication date: 16-Mar-2024
https://doi.org/10.1109/ISSSR61934.2024.00024
Brandi CPerrone GRomano S(2024)Sniping at web applications to discover input-handling vulnerabilitiesJournal of Computer Virology and Hacking Techniques10.1007/s11416-024-00518-0Online publication date: 12-Apr-2024
https://doi.org/10.1007/s11416-024-00518-0
Zhao XQu HXu JLi XLv WWang G(2024)A systematic review of fuzzingSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-023-09306-228:6(5493-5522)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1007/s00500-023-09306-2
Park YKo YKim J(2024)Mutation Methods for Structured Input to Enhance Path Coverage of FuzzersInformation Security Applications10.1007/978-981-99-8024-6_20(257-268)Online publication date: 11-Jan-2024
https://doi.org/10.1007/978-981-99-8024-6_20
Paliath VTrickel EBao TWang RDoupé AShoshitaishvili Y(2024)SandPuppy: Deep-State Fuzzing Guided by Automatic Detection of State-Representative VariablesDetection of Intrusions and Malware, and Vulnerability Assessment10.1007/978-3-031-64171-8_12(227-250)Online publication date: 9-Jul-2024
https://doi.org/10.1007/978-3-031-64171-8_12
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents