research-article

Taming compiler fuzzers

Authors:

Chaoqiang Zhang,

Weng-Keen Wong,

John RegehrAuthors Info & Claims

PLDI '13: Proceedings of the 34th ACM SIGPLAN Conference on Programming Language Design and Implementation

Pages 197 - 208

https://doi.org/10.1145/2491956.2462173

Published: 16 June 2013 Publication History

Abstract

Aggressive random testing tools ("fuzzers") are impressively effective at finding compiler bugs. For example, a single test-case generator has resulted in more than 1,700 bugs reported for a single JavaScript engine. However, fuzzers can be frustrating to use: they indiscriminately and repeatedly find bugs that may not be severe enough to fix right away. Currently, users filter out undesirable test cases using ad hoc methods such as disallowing problematic features in tests and grepping test results. This paper formulates and addresses the fuzzer taming problem: given a potentially large number of random test cases that trigger failures, order them such that diverse, interesting test cases are highly ranked. Our evaluation shows our ability to solve the fuzzer taming problem for 3,799 test cases triggering 46 bugs in a C compiler and 2,603 test cases triggering 28 bugs in a JavaScript engine.

References

[1]

James H. Andrews, Alex Groce, Melissa Weston, and Ru-Gang Xu. Random test run length and effectiveness. In Proc. ASE, pages 19--28, September 2008.

Digital Library

[2]

Abhishek Arya and Cris Neckar. Fuzzing for security, April 2012. http://blog.chromium.org/2012/04/fuzzing-for-security.html.

[3]

Mariano Ceccato, Alessandro Marchetto, Leonardo Mariani, Cu D. Nguyen, and Paolo Tonella. An empirical study about the effectiveness of debugging when random test cases are used. In Proc. ICSE, pages 452--462, June 2012.

Digital Library

[4]

Silvio Cesare and Yang Xiang. Malware variant detection using similarity search over sets of control flow graphs. In Proc. TRUSTCOM, pages 181--189, November 2011.

Digital Library

[5]

Sagar Chaki, Alex Groce, and Ofer Strichman. Explaining abstract counterexamples. In Proc. FSE, pages 73--82, 2004.

Digital Library

[6]

Koen Claessen and John Hughes. QuickCheck: a lightweight tool for random testing of Haskell programs. In Proc. ICFP, pages 268--279, 2000.

Digital Library

[7]

Holger Cleve and Andreas Zeller. Locating causes of program failures. In Proc.\ ICSE, pages 342--351, May 2005.

Digital Library

[8]

Shai Fine and Yishay Mansour. Active sampling for multiple output identification. Machine Learning, 69(2--3):213--228, 2007.

Digital Library

[9]

Patrick Francis, David Leon, Melinda Minch, and Andy Podgurski. Tree-based methods for classifying software failures. In Proc. ISSRE, pages 451--462, November 2004.

Digital Library

[10]

Teofilo F. Gonzalez. Clustering to minimize the maximum intercluster distance. Theoretical Computer Science, 38:293--306, 1985.

[11]

Alex Groce. Error explanation with distance metrics. In Proc. TACAS, pages 108--122, March 2004.

[12]

Alex Groce, Gerard Holzmann, and Rajeev Joshi. Randomized differential testing as a prelude to formal verification. In Proc. ICSE, pages 621--631, May 2007.

Digital Library

[13]

Alex Groce, Chaoqiang Zhang, Eric Eide, Yang Chen, and John Regehr. Swarm testing. In Proc. ISSTA, pages 78--88, July 2012.balance\phantom.

Digital Library

[14]

Christian Holler, Kim Herzig, and Andreas Zeller. Fuzzing with code fragments. In Proc. USENIX Security, pages 445--458, August 2012.

Digital Library

[15]

James A. Jones, James F. Bowring, and Mary Jean Harrold. Debugging in parallel. In Proc. ISSTA, pages 16--26, July 2007.

Digital Library

[16]

James A. Jones and Mary Jean Harrold. Empirical evaluation of the Tarantula automatic fault-localization technique. In Proc. ASE, pages 273--282, November 2005.

Digital Library

[17]

James A. Jones, Mary Jean Harrold, and John Stasko. Visualization of test information to assist fault localization. In Proc. ICSE, pages 467--477, May 2002.

Digital Library

[18]

Yungbum Jung, Jaehwang Kim, Jaeho Shin, and Kwangkeun Yi. Taming false alarms from a domain-unaware C analyzer by a Bayesian statistical post analysis. In Proc. SAS, pages 203--217, September 2005.

Digital Library

[19]

Ted Kremenek and Dawson Engler. Z-ranking: using statistical analysis to counter the impact of static analysis approximations. In Proc. SAS, pages 295--315, June 2003.

Digital Library

[20]

Vladimir I. Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10:707--710, 1966.

[21]

Ben Liblit, Alex Aiken, Alice X. Zheng, and Michael I. Jordan. Bug isolation via remote program sampling. In Proc. PLDI, pages 141--154, June 2003.

Digital Library

[22]

Ben Liblit, Mayur Naik, Alice X. Zheng, Alex Aiken, and Michael I. Jordan. Scalable statistical bug isolation. In Proc. PLDI, pages 15--26, June 2005.

Digital Library

[23]

Chao Liu and Jiawei Han. Failure proximity: a fault localization-based approach. In Proc. FSE, pages 46--56, November 2006.

Digital Library

[24]

William M. McKeeman. Differential testing for software. Digital Technical Journal, 10(1):100--107, December 1998.

[25]

Nicholas Nethercote and Julian Seward. Valgrind: a framework for heavyweight dynamic binary instrumentation. In Proc. PLDI, pages 89--100, June 2007.

Digital Library

[26]

Dan Pelleg and Andrew Moore. Active learning for anomaly and rare-category detection. In Advances in Neural Information Processing Systems 18, December 2004.

[27]

Dan Pelleg and Andrew W. Moore. X-means: Extending K-means with efficient estimation of the number of clusters. In Proc. ICML, pages 727--734, June/July 2000.

Digital Library

[28]

Andy Podgurski, David Leon, Patrick Francis, Wes Masri, Melinda Minch, Jiayang Sun, and Bin Wang. Automated support for classifying software failure reports. In Proc. ICSE, pages 465--475, May 2003.

Digital Library

[29]

John Regehr, Yang Chen, Pascal Cuoq, Eric Eide, Chucky Ellison, and Xuejun Yang. Test-case reduction for C compiler bugs. In Proc. PLDI, pages 335--346, June 2012.

Digital Library

[30]

Manos Renieris and Steven Reiss. Fault localization with nearest neighbor queries. In Proc. ASE, pages 30--39, October 2003.

[31]

Jesse Ruderman. Introducing jsfunfuzz. http://www.squarefree.com/2007/08/02/introducing-jsfunfuzz/.

[32]

Jesse Ruderman. Mozilla bug 349611. https://bugzilla.mozilla.org/show_bug.cgi?id=349611 (A meta-bug containing all bugs found using jsfunfuzz.).

[33]

Jesse Ruderman. How my DOM fuzzer ignores known bugs, 2010. http://www.squarefree.com/2010/11/21/how-my-dom-fuzzer-ignores-known-bugs/.

[34]

G. Salton, A. Wong, and C. S. Yang. A vector space model for automatic indexing. CACM, 18(11):613--620, November 1975.

Digital Library

[35]

Saul Schleimer, Daniel S. Wilkerson, and Alex Aiken. Winnowing: local algorithms for document fingerprinting. In Proc. SIGMOD, pages 76--85, June 2003.

Digital Library

[36]

Alexander Strehl and Joydeep Ghosh. Cluster ensembles--a knowledge reuse framework for combining multiple partitions. The Journal of Machine Learning Research, 3:583--617, 2003.

Digital Library

[37]

Chengnian Sun, David Lo, Siau-Cheng Khoo, and Jing Jiang. Towards more accurate retrieval of duplicate bug reports. In Proc. ASE, pages 253--262, November 2011.

Digital Library

[38]

Chengnian Sun, David Lo, Xiaoyin Wang, Jing Jiang, and Siau-Cheng Khoo. A discriminative model approach for accurate duplicate bug report retrieval. In Proc. ICSE, pages 45--54, May 2010.

Digital Library

[39]

Vipindeep Vangala, Jacek Czerwonka, and Phani Talluri. Test case comparison and clustering using program profiles and static execution. In Proc. ESEC/FSE, pages 293--294, August 2009.

Digital Library

[40]

Pavan Vatturi and Weng-Keen Wong. Category detection using hierarchical mean shift. In Proc. KDD, pages 847--856, June/July 2009.

Digital Library

[41]

Andrew Walenstein, Mohammad El-Ramly, James R. Cordy, William S. Evans, Kiarash Mahdavi, Markus Pizka, Ganesan Ramalingam, and Jürgen Wolff von Gudenberg. Similarity in programs. In Duplication, Redundancy, and Similarity in Software, Dagstuhl Seminar Proceedings, July 2006.

[42]

Xiaoyin Wang, Lu Zhang, Tao Xie, John Anvik, and Jiasu Sun. An approach to detecting duplicate bug reports using natural language and execution information. In Proc. ICSE, pages 461--470, May 2008.

Digital Library

[43]

David B. Whalley. Automatic isolation of compiler errors. TOPLAS, 16(5):1648--1659, September 1994.

Digital Library

[44]

Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. Finding and understanding bugs in C compilers. In Proc. PLDI, pages 283--294, June 2011.

Digital Library

[45]

Andreas Zeller and Ralf Hildebrandt. Simplifying and isolating failure-inducing input. IEEE TSE, 28(2):183--200, February 2002.

Digital Library

Cited By

Chaliasos SErnstberger JTheodore DWong DJahanara MLivshits BBalzarotti DXu W(2024)SoKProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699116(3855-3872)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3699116
Ba JRigger M(2024)Keep It Simple: Testing Databases via Differential Query PlansProceedings of the ACM on Management of Data10.1145/36549912:3(1-26)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3654991
Xia XFeng YShi QJones JZhang XXu B(2024)Enumerating Valid Non-Alpha-Equivalent Programs for Interpreter TestingACM Transactions on Software Engineering and Methodology10.1145/364799433:5(1-31)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1145/3647994
Show More Cited By

Index Terms

Taming compiler fuzzers
1. Information systems
  1. Information retrieval
    1. Retrieval models and ranking
    2. Retrieval tasks and goals
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software notations and tools
    1. Compilers

Recommendations

Finding and understanding bugs in C compilers
PLDI '11: Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation

Compilers should be correct. To improve the quality of C compilers, we created Csmith, a randomized test-case generation tool, and spent three years using it to find compiler bugs. During this period we reported more than 325 previously unknown bugs to ...
Taming compiler fuzzers
PLDI '13

Aggressive random testing tools ("fuzzers") are impressively effective at finding compiler bugs. For example, a single test-case generator has resulted in more than 1,700 bugs reported for a single JavaScript engine. However, fuzzers can be frustrating ...
Test-case reduction for C compiler bugs
PLDI '12

To report a compiler bug, one must often find a small test case that triggers the bug. The existing approach to automated test-case reduction, delta debugging, works by removing substrings of the original input; the result is a concatenation of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

PLDI '13: Proceedings of the 34th ACM SIGPLAN Conference on Programming Language Design and Implementation

June 2013

546 pages

ISBN:9781450320146

DOI:10.1145/2491956

General Chair:
Hans-J. Boehm
HP Labs
,
Program Chair:
Cormac Flanagan
University of California, Santa Cruz

ACM SIGPLAN Notices Volume 48, Issue 6
PLDI '13
June 2013
515 pages
ISSN:0362-1340
EISSN:1558-1160
DOI:10.1145/2499370
Issue’s Table of Contents

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 June 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

PLDI '13

Sponsor:

SIGPLAN

PLDI '13: ACM SIGPLAN Conference on Programming Language Design and Implementation

June 16 - 19, 2013

Washington, Seattle, USA

Acceptance Rates

PLDI '13 Paper Acceptance Rate 46 of 267 submissions, 17%;

Overall Acceptance Rate 406 of 2,067 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

138
Total Citations
View Citations
1,017
Total Downloads

Downloads (Last 12 months)67
Downloads (Last 6 weeks)4

Reflects downloads up to 27 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chaliasos SErnstberger JTheodore DWong DJahanara MLivshits BBalzarotti DXu W(2024)SoKProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699116(3855-3872)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3699116
Ba JRigger M(2024)Keep It Simple: Testing Databases via Differential Query PlansProceedings of the ACM on Management of Data10.1145/36549912:3(1-26)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3654991
Xia XFeng YShi QJones JZhang XXu B(2024)Enumerating Valid Non-Alpha-Equivalent Programs for Interpreter TestingACM Transactions on Software Engineering and Methodology10.1145/364799433:5(1-31)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1145/3647994
Deb SJain Kvan Tonder RLe Goues CGroce A(2024)Syntax Is All You Need: A Universal-Language Approach to Mutant GenerationProceedings of the ACM on Software Engineering10.1145/36437561:FSE(654-674)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643756
Xu JLu KDu ZDing ZLi LWu QPayer MMao BCalandrino JTroncoso C(2023)Silent bugs matterProceedings of the 32nd USENIX Conference on Security Symposium10.5555/3620237.3620442(3655-3672)Online publication date: 9-Aug-2023
https://dl.acm.org/doi/10.5555/3620237.3620442
Tian YZhang XDong YXu ZZhang MJiang YCheung SSun C(2023)On the Caching Schemes to Speed Up Program ReductionACM Transactions on Software Engineering and Methodology10.1145/361717233:1(1-30)Online publication date: 5-Sep-2023
https://dl.acm.org/doi/10.1145/3617172
Shrestha SJust RFraser G(2023)Harnessing Large Language Models for Simulink Toolchain Testing and Developing Diverse Open-Source Corpora of Simulink Models for Metric and Evolution AnalysisProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3605233(1541-1545)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3605233
Yang CChen JFan XJiang JSun JJust RFraser G(2023)Silent Compiler Bug De-duplication via Three-Dimensional AnalysisProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598087(677-689)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598087
Zhao YChen JFu RYe HWang ZJust RFraser G(2023)Testing the Compiler for a New-Born Programming Language: An Industrial Case Study (Experience Paper)Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598077(551-563)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598077
Lecoeur BMohsin HDonaldson A(2023)Program Reconditioning: Avoiding Undefined Behaviour When Finding and Reducing Compiler BugsProceedings of the ACM on Programming Languages10.1145/35912947:PLDI(1801-1825)Online publication date: 6-Jun-2023
https://dl.acm.org/doi/10.1145/3591294
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten