research-article

Finding and understanding bugs in C compilers

Authors:

John RegehrAuthors Info & Claims

PLDI '11: Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation

Pages 283 - 294

https://doi.org/10.1145/1993498.1993532

Published: 04 June 2011 Publication History

Abstract

Compilers should be correct. To improve the quality of C compilers, we created Csmith, a randomized test-case generation tool, and spent three years using it to find compiler bugs. During this period we reported more than 325 previously unknown bugs to compiler developers. Every compiler we tested was found to crash and also to silently generate wrong code when presented with valid input. In this paper we present our compiler-testing tool and the results of our bug-hunting study. Our first contribution is to advance the state of the art in compiler testing. Unlike previous tools, Csmith generates programs that cover a large subset of C while avoiding the undefined and unspecified behaviors that would destroy its ability to automatically find wrong-code bugs. Our second contribution is a collection of qualitative and quantitative results about the bugs we have found in open-source C compilers.

References

[1]

ACE Associated Computer Experts. SuperTest C/C+ compiler test and validation suite. http://www.ace.nl/compiler/supertest.html

[2]

F. Bellard. TCC: Tiny C compiler, ver. 0.9.25, May 2009. http://bellard.org/tcc/.

[3]

C. L. Biffle. Undefined behavior in Google NaCl, Jan. 2010. http://code.google.com/p/nativeclient/issues/detail?id=245.

[4]

A. S. Boujarwah and K. Saleh. Compiler test case generation methods: a survey and assessment. Information and Software Technology, 39(9):617--625, 1997.

[5]

C. J. Burgess and M. Saidi. The automatic generation of test cases for optimizing Fortran compilers. Information and Software Technology, 38(2):111--119, 1996.

[6]

E. Eide and J. Regehr. Volatiles are miscompiled, and what to do about it. In Proc. EMSOFT, pages 255--264, Oct. 2008.

Digital Library

[7]

X. Feng and A. J. Hu. Cutpoints for formal equivalence verification of embedded software. In Proc. EMSOFT, pages 307--316, Sept. 2005.

Digital Library

[8]

P. Godefroid, A. Kiezun, and M. Y. Levin. Grammar-based whitebox fuzzing. In Proc. PLDI, pages 206--215, June 2008.

Digital Library

[9]

R. Hamlet. Random testing. In J. Marciniak, editor, Encyclopedia of Software Engineering. Wiley, second edition, 2001.

[10]

K. V. Hanford. Automatic generation of test cases. IBM Systems Journal, 9(4):242--257, Dec. 1970.

Digital Library

[11]

International Organization for Standardization. ISO/IEC 9899:TC2: Programming Languages-C, May 2005. http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1124.pdf.

[12]

G. Klein et al. seL4: Formal verification of an OS kernel. In Proc. SOSP, pages 207--220, Oct. 2009.

Digital Library

[13]

J. C. Knight and N. G. Leveson. An experimental evaluation of the assumption of independence in multiversion programming. IEEE Trans. Software Eng., 12(1):96--109, Jan. 1986.

Digital Library

[14]

X. Leroy. Formal verification of a realistic compiler. Commun. ACM, 52(7):107--115, July 2009.

Digital Library

[15]

C. Lindig. Random testing of C calling conventions. In Proc. AADEBUG, pages 3--12, Sept. 2005.

Digital Library

[16]

W. M. McKeeman. Differential testing for software. Digital Technical Journal, 10(1):100--107, Dec. 1998.

[17]

B. P. Miller, L. Fredriksen, and B. So. An empirical study of the reliability of UNIX utilities. Commun. ACM, 33(12):32--44, Dec. 1990.

Digital Library

[18]

G. Misherghi and Z. Su. HDD: Hierarchical delta debugging. In Proc. ICSE, pages 142--151, May 2006.

Digital Library

[19]

Perennial, Inc. ACVS ANSI/ISO/FIPS-160 C validation suite, ver. 4.5, Jan. 1998. http://www.peren.com/pages/acvs_set.htm.

[20]

Plum Hall, Inc. The Plum Hall validation suite for C. http://www.plumhall.com/stec.html.

[21]

P. Purdom. A sentence generator for testing parsers. BIT Numerical Mathematics, 12(3):366--375, 1972.

Digital Library

[22]

R. L. Sauder. A general test data generator for COBOL. In AFIPS Joint Computer Conferences, pages 317--323, May 1962.

Digital Library

[23]

F. Sheridan. Practical testing of a C99 compiler using output comparison. Software--Practice and Experience, 37(14):1475--1488, Nov. 2007.

Digital Library

[24]

J. Souyris, V. Wiels, D. Delmas, and H. Delseny. Formal verification of avionics software products. In Proc. FM, pages 532--546, Nov. 2009.

Digital Library

[25]

S. Summit. comp.lang.c frequently asked questions. http://c-faq.com/.

Digital Library

[26]

Z. Tatlock and S. Lerner. Bringing extensibility to verified compilers. In Proc. PLDI, pages 111--121, June 2010.

Digital Library

[27]

B. Turner. Random Program Generator, Jan. 2007. http://sites.google.com/site/brturn2/randomcprogramgenerator.

[28]

B. White et al. An integrated experimental environment for distributed systems and networks. In Proc. OSDI, pages 255--270, Dec. 2002.

Digital Library

[29]

D. S. Wilkerson. Delta ver. 2006.08.03, Aug. 2006. http://delta.tigris.org/.

[30]

M. Wolfe. How compilers and tools differ for embedded systems. In Proc. CASES, Sept. 2005. Keynote address. http://www.pgroup.com/lit/articles/pgi_article_cases.pdf.

Digital Library

[31]

A. Zeller and R. Hildebrandt. Simplifying and isolating failure-inducing input. IEEE Trans. Software Eng., 28(2):183--200, Feb. 2002.

Digital Library

[32]

C. Zhao et al. Automated test program generation for an industrial optimizing compiler. In Proc. ICSE Workshop on Automation of Software Test, pages 36--43, May 2009.

Cited By

Zhang CKappé TNarváez DNaus N(2025)CF-GKAT: Efficient Validation of Control-Flow TransformationsProceedings of the ACM on Programming Languages10.1145/37048579:POPL(600-626)Online publication date: 9-Jan-2025
https://dl.acm.org/doi/10.1145/3704857
Bowman WAllais GLiu Y(2025)The Ethical Compiler: Addressing the Is-Ought Gap in Compilation (Invited Talk)Proceedings of the 2025 ACM SIGPLAN International Workshop on Partial Evaluation and Program Manipulation10.1145/3704253.3706135(1-9)Online publication date: 10-Jan-2025
https://dl.acm.org/doi/10.1145/3704253.3706135
Chaliasos SErnstberger JTheodore DWong DJahanara MLivshits BBalzarotti DXu W(2024)SoKProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699116(3855-3872)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3699116
Show More Cited By

Index Terms

Finding and understanding bugs in C compilers
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software notations and tools
    1. Compilers
    2. General programming languages
      1. Language types

Recommendations

Random testing for C and C++ compilers with YARPGen

Compilers should not crash and they should not miscompile applications. Random testing is an effective method for finding compiler bugs that have escaped other kinds of testing. This paper presents Yet Another Random Program Generator (YARPGen), a random ...
Finding and understanding bugs in C compilers
PLDI '11

Compilers should be correct. To improve the quality of C compilers, we created Csmith, a randomized test-case generation tool, and spent three years using it to find compiler bugs. During this period we reported more than 325 previously unknown bugs to ...
Volatiles are miscompiled, and what to do about it
EMSOFT '08: Proceedings of the 8th ACM international conference on Embedded software

C's volatile qualifier is intended to provide a reliable link between operations at the source-code level and operations at the memory-system level. We tested thirteen production-quality C compilers and, for each, found situations in which the compiler ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

PLDI '11: Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation

June 2011

668 pages

ISBN:9781450306638

DOI:10.1145/1993498

General Chair:
Mary Hall
University of Utah
,
Program Chair:
David Padua
University of Illinois at Urbana-Champaign

ACM SIGPLAN Notices Volume 46, Issue 6
PLDI '11
June 2011
652 pages
ISSN:0362-1340
EISSN:1558-1160
DOI:10.1145/1993316
Issue’s Table of Contents

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 June 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

PLDI '11

Sponsor:

SIGPLAN

PLDI '11: ACM SIGPLAN Conference on Programming Language Design and Implementation

June 4 - 8, 2011

California, San Jose, USA

Acceptance Rates

Overall Acceptance Rate 406 of 2,067 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

806
Total Citations
View Citations
3,752
Total Downloads

Downloads (Last 12 months)534
Downloads (Last 6 weeks)39

Reflects downloads up to 27 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang CKappé TNarváez DNaus N(2025)CF-GKAT: Efficient Validation of Control-Flow TransformationsProceedings of the ACM on Programming Languages10.1145/37048579:POPL(600-626)Online publication date: 9-Jan-2025
https://dl.acm.org/doi/10.1145/3704857
Bowman WAllais GLiu Y(2025)The Ethical Compiler: Addressing the Is-Ought Gap in Compilation (Invited Talk)Proceedings of the 2025 ACM SIGPLAN International Workshop on Partial Evaluation and Program Manipulation10.1145/3704253.3706135(1-9)Online publication date: 10-Jan-2025
https://dl.acm.org/doi/10.1145/3704253.3706135
Chaliasos SErnstberger JTheodore DWong DJahanara MLivshits BBalzarotti DXu W(2024)SoKProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699116(3855-3872)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3699116
Zou MKhan AWu RGao HBianchi ATian DBalzarotti DXu W(2024)D-HelixProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3698923(397-414)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3698923
Goldweber EYu WGhahani SKapritsos MGavrilovska ATerry D(2024)IronSpecProceedings of the 18th USENIX Conference on Operating Systems Design and Implementation10.5555/3691938.3691985(875-891)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.5555/3691938.3691985
Sun HSu ZGavrilovska ATerry D(2024)Validating the eBPF verifier via state embeddingProceedings of the 18th USENIX Conference on Operating Systems Design and Implementation10.5555/3691938.3691971(615-628)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.5555/3691938.3691971
Venkitaraman A(2024)Early Bug Detection through Shift Left TestingInternational Journal of Innovative Science and Research Technology (IJISRT)10.38124/ijisrt/IJISRT24NOV177(185-190)Online publication date: 18-Nov-2024
https://doi.org/10.38124/ijisrt/IJISRT24NOV177
Han JZhang ZDu YWang WChen X(2024)ESFuzzer: An Efficient Way to Fuzz WebAssembly InterpreterElectronics10.3390/electronics1308149813:8(1498)Online publication date: 15-Apr-2024
https://doi.org/10.3390/electronics13081498
Bendrissou BCadar CDonaldson A(2024)Grammar Mutation for Testing Input ParsersACM Transactions on Software Engineering and Methodology10.1145/3708517Online publication date: 20-Dec-2024
https://dl.acm.org/doi/10.1145/3708517
Noble JWeir C(2024)The Faultless Way of ProgrammingProceedings of the 29th European Conference on Pattern Languages of Programs, People, and Practices10.1145/3698322.3698340(1-7)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3698322.3698340
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten