research-article

Public Access

Making no-fuss compiler fuzzing effective

Authors:

Rijnard van Tonder,

Goutamkumar Tulajappa Kalburgi,

Claire Le GouesAuthors Info & Claims

CC 2022: Proceedings of the 31st ACM SIGPLAN International Conference on Compiler Construction

Pages 194 - 204

https://doi.org/10.1145/3497776.3517765

Published: 18 March 2022 Publication History

Abstract

Developing a bug-free compiler is difficult; modern optimizing compilers are among the most complex software systems humans build. Fuzzing is one way to identify subtle compiler bugs that are hard to find with human-constructed tests. Grammar-based fuzzing, however, requires a grammar for a compiler’s input language, and can miss bugs induced by code that does not actually satisfy the grammar the compiler should accept. Grammar-based fuzzing also seldom uses advanced modern fuzzing techniques based on coverage feedback. However, modern mutation-based fuzzers are often ineffective for testing compilers because most inputs they generate do not even come close to getting past the parsing stage of compilation. This paper introduces a technique for taking a modern mutation-based fuzzer (AFL in our case, but the method is general) and augmenting it with operators taken from mutation testing, and program splicing. We conduct a controlled study to show that our hybrid approaches significantly improve fuzzing effectiveness qualitatively (consistently finding unique bugs that baseline approaches do not) and quantitatively (typically finding more unique bugs in the same time span, despite fewer program executions). Our easy-to-apply approach has allowed us to report more than 100 confirmed and fixed bugs in production compilers, and found a bug in the Solidity compiler that earned a security bounty.

References

[1]

Iftekhar Ahmed, Carlos Jensen, Alex Groce, and Paul E. McKenney. 2017. Applying Mutation Analysis on Kernel Test Suites: an Experience Report. In International Workshop on Mutation Analysis. 110–115.

[2]

Scott Bauer, Pascal Cuoq, and John Regehr. 2017. POC||GTFO. No Starch Press.

[3]

Moritz Beller, Chu-Pan Wong, Johannes Bader, Andrew Scott, Mateusz Machalica, Satish Chandra, and Erik Meijer. 2021. What It Would Take to Use Mutation Testing in Industry—A Study at Facebook. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). 268–277. https://doi.org/10.1109/ICSE-SEIP52600.2021.00036

Digital Library

[4]

Marcel Böhme, Cristian Cadar, and Abhik Roychoudhury. 2021. Fuzzing: Challenges and Reflections. IEEE Softw., 38, 3 (2021), 79–86.

[5]

Marcel Böhme and Brandon Falk. 2020. Fuzzing: On the Exponential Cost of Vulnerability Discovery. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2020). Association for Computing Machinery, New York, NY, USA. 713–724. isbn:9781450370431 https://doi.org/10.1145/3368089.3409729

Digital Library

[6]

Timothy Budd, Richard J. Lipton, Richard A DeMillo, and Frederick G Sayward. 1979. Mutation analysis. Yale University, Department of Computer Science.

[7]

Junjie Chen, Jibesh Patra, Michael Pradel, Yingfei Xiong, Hongyu Zhang, Dan Hao, and Lu Zhang. 2020. A survey of compiler testing. ACM Computing Surveys (CSUR), 53, 1 (2020), 1–36.

Digital Library

[8]

Peng Chen and Hao Chen. 2018. Angora: Efficient Fuzzing by Principled Search. In 2018 IEEE Symposium on Security and Privacy (SP). 711–725. https://doi.org/10.1109/SP.2018.00046

[9]

Yang Chen, Alex Groce, Chaoqiang Zhang, Weng-Keen Wong, Xiaoli Fern, Eric Eide, and John Regehr. 2013. Taming Compiler Fuzzers. In ACM SIGPLAN Symposium on Programming Language Design and Implementation. 197–208. https://doi.org/10.1145/2499370.2462173

Digital Library

[10]

Jaeseung Choi, Joonun Jang, Choongwoo Han, and Sang Kil Cha. 2019. Grey-box Concolic Testing on Binary Code. In Proceedings of the International Conference on Software Engineering. 736–747.

Digital Library

[11]

Rahul Gopinath, Iftekhar Ahmed, Mohammad Amin Alipour, Carlos Jensen, and Alex Groce. 2017. Mutation Reduction Strategies Considered Harmful. IEEE Trans. Reliab., 66, 3 (2017), 854–874. https://doi.org/10.1109/TR.2017.2705662

[12]

Alex Groce, Josie Holmes, Darko Marinov, August Shi, and Lingming Zhang. 2018. An Extensible, Regular-expression-based Tool for Multi-language Mutant Generation. In Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings (ICSE ’18). ACM, New York, NY, USA. 25–28. isbn:978-1-4503-5663-3 https://doi.org/10.1145/3183440.3183485

Digital Library

[13]

Christian Holler, Kim Herzig, and Andreas Zeller. 2012. Fuzzing with Code Fragments. In Proceedings of the 21st USENIX Conference on Security Symposium (Security’12). USENIX Association, USA. 38.

Digital Library

[14]

George Klees, Andrew Ruef, Benji Cooper, Shiyi Wei, and Michael Hicks. 2018. Evaluating Fuzz Testing. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security (CCS ’18). Association for Computing Machinery, New York, NY, USA. 2123–2138. isbn:9781450356930 https://doi.org/10.1145/3243734.3243804

Digital Library

[15]

Richard J. Lipton, Richard A DeMillo, and Frederick G Sayward. 1978. Hints on test data selection: Help for the practicing programmer. Computer, 11, 4 (1978), 34–41.

Digital Library

[16]

V. Manes, H. Han, C. Han, s. cha, M. Egele, E. J. Schwartz, and M. Woo. 5555. The Art, Science, and Engineering of Fuzzing: A Survey. IEEE Transactions on Software Engineering, oct, 1–1. issn:1939-3520 https://doi.org/10.1109/TSE.2019.2946563

[17]

William McKeeman. 1998. Differential testing for software. Digital Technical Journal of Digital Equipment Corporation, 10(1) (1998), 100–107.

[18]

Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Mutation testing advances: an analysis and survey. In Advances in Computers. 112, Elsevier, 275–378.

[19]

Goran Petrović and Marko Ivanković. 2018. State of Mutation Testing at Google. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP ’18). Association for Computing Machinery, New York, NY, USA. 163–171. isbn:9781450356596 https://doi.org/10.1145/3183519.3183521

Digital Library

[20]

Christopher Salls, Chani Jindal, Jake Corina, Christopher Kruegel, and Giovanni Vigna. 2021. Token-Level Fuzzing. In 30th USENIX Security Symposium (USENIX Security 21).

[21]

Rijnard van Tonder, John Kotheimer, and Claire Le Goues. 2018. Semantic Crash Bucketing. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE 2018). ACM, New York, NY, USA. 612–622. isbn:978-1-4503-5937-5 https://doi.org/10.1145/3238147.3238200

Digital Library

[22]

Rijnard van Tonder and Claire Le Goues. 2019. Lightweight Multi-Language Syntax Transformation with Parser Parser Combinators. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 2019). Association for Computing Machinery, New York, NY, USA. 363–378. isbn:9781450367127 https://doi.org/10.1145/3314221.3314589

Digital Library

[23]

Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and understanding bugs in C compilers. In Proceedings of the 32nd ACM SIGPLAN conference on Programming language design and implementation. 283–294.

Digital Library

Cited By

Ma HZhang WShen QTian YChen JCheung SChristakis MPradel M(2024)Towards Understanding the Bugs in Solidity CompilerProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680362(1312-1324)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680362
Deb SJain Kvan Tonder RLe Goues CGroce A(2024)Syntax Is All You Need: A Universal-Language Approach to Mutant GenerationProceedings of the ACM on Software Engineering10.1145/36437561:FSE(654-674)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643756
Hu FJi JShu HLi ZLiu TZhang C(2024)Formatted Stateful Greybox Fuzzing of TLS Server2024 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST60714.2024.00022(151-160)Online publication date: 27-May-2024
https://doi.org/10.1109/ICST60714.2024.00022
Show More Cited By

Index Terms

Making no-fuss compiler fuzzing effective
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software organization and properties
    1. Software functional properties
      1. Formal methods
        Dynamic analysis

Recommendations

Compiler fuzzing: how much does it matter?

Despite much recent interest in randomised testing (fuzzing) of compilers, the practical impact of fuzzer-found compiler bugs on real-world applications has barely been assessed. We present the first quantitative and qualitative study of the tangible ...
Fuzzing for CPS Mutation Testing
ASE '23: Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering

Mutation testing can help reduce the risks of releasing faulty software. For such reason, it is a desired practice for the development of embedded software running in safety-critical cyber-physical systems (CPS). Unfortunately, state-of-the-art test data ...
Fine-grained Coverage-based Fuzzing
Fuzzing is a popular software testing method that discovers bugs by massively feeding target applications with automatically generated inputs. Many state-of-the-art fuzzers use branch coverage as a feedback metric to guide the fuzzing process. The fuzzer ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CC 2022: Proceedings of the 31st ACM SIGPLAN International Conference on Compiler Construction

March 2022

253 pages

ISBN:9781450391832

DOI:10.1145/3497776

General Chairs:
Bernhard Egger
Seoul National University, South Korea
,
Aaron Smith
Microsoft, USA

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 March 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

CC '22

Sponsor:

SIGPLAN

CC '22: 31st ACM SIGPLAN International Conference on Compiler Construction

April 2 - 3, 2022

Seoul, South Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
575
Total Downloads

Downloads (Last 12 months)257
Downloads (Last 6 weeks)25

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ma HZhang WShen QTian YChen JCheung SChristakis MPradel M(2024)Towards Understanding the Bugs in Solidity CompilerProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680362(1312-1324)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680362
Deb SJain Kvan Tonder RLe Goues CGroce A(2024)Syntax Is All You Need: A Universal-Language Approach to Mutant GenerationProceedings of the ACM on Software Engineering10.1145/36437561:FSE(654-674)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643756
Hu FJi JShu HLi ZLiu TZhang C(2024)Formatted Stateful Greybox Fuzzing of TLS Server2024 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST60714.2024.00022(151-160)Online publication date: 27-May-2024
https://doi.org/10.1109/ICST60714.2024.00022
Donaldson AClayton BHarrison RMohsin HNeto DTeliman VWatson H(2023)Industrial Deployment of Compiler Fuzzing Techniques for Two GPU Shading Languages2023 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST57152.2023.00042(374-385)Online publication date: Apr-2023
https://doi.org/10.1109/ICST57152.2023.00042
Mitropoulos CSotiropoulos TIoannidis SMitropoulos D(2023)Syntax-Aware Mutation for Testing the Solidity CompilerComputer Security – ESORICS 202310.1007/978-3-031-51479-1_17(327-347)Online publication date: 25-Sep-2023
https://dl.acm.org/doi/10.1007/978-3-031-51479-1_17

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents