research-article

Finding and analyzing compiler warning defects

Authors:

Zhendong SuAuthors Info & Claims

ICSE '16: Proceedings of the 38th International Conference on Software Engineering

Pages 203 - 213

https://doi.org/10.1145/2884781.2884879

Published: 14 May 2016 Publication History

Abstract

Good compiler diagnostic warnings facilitate software development as they indicate likely programming mistakes or code smells. However, due to compiler bugs, the warnings may be erroneous, superfluous or missing, even for mature production compilers like GCC and Clang. In this paper, we (1) propose the first randomized differential testing technique to detect compiler warning defects and (2) describe our extensive evaluation in finding warning defects in widely-used C compilers.

At the high level, our technique starts with generating random programs to trigger compilers to emit a variety of compiler warnings, aligns the warnings from different compilers, and identifies inconsistencies as potential bugs. We develop effective techniques to overcome three specific challenges: (1) How to generate random programs, (2) how to align textual warnings, and (3) how to reduce test programs for bug reporting?

Our technique is very effective --- we have found and reported 60 bugs for GCC (38 confirmed, assigned or fixed) and 39 for Clang (14 confirmed or fixed). This case study not only demonstrates our technique's effectiveness, but also highlights the need to continue improving compilers' warning support, an essential, but rather neglected aspect of compilers.

References

[1]

Clang Compiler User's Manual 3.5. http://clang.llvm.org/docs/UsersManual.html#diagnostics-enable-everything, accessed: 2014-03-04.

[2]

Clang Static Analyzer. http://clang-analyzer.llvm.org/, accessed: 2015-08-10.

[3]

GCC Bug #28901. https://gcc.gnu.org/bugzilla/show_bug.cgi?id=28901, accessed: 2014-03-04.

[4]

GDB Bug. https://sourceware.org/ml/gdb-patches/2014-02/msg00342.html, accessed: 2014-03-04.

[5]

Options to Request or Suppress Warnings -- GCC. http://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html, accessed: 2014-03-04.

[6]

PMD. http://pmd.github.io/, accessed: 2015-08-10.

[7]

ACE. SuperTest compiler test and validation suite. http://www.ace.nl/compiler/supertest.html.

[8]

A. Allain. Why Bother with Compiler Warnings. http://www.cprogramming.com/tutorial/compiler_warnings.html, accessed: 2014-12-10.

[9]

P. Ammann and J. Offutt. Introduction to Software Testing. Cambridge University Press, New York, NY, USA, 1 edition, 2008.

Digital Library

[10]

N. Ayewah, W. Pugh, J. D. Morgenthaler, J. Penix, and Y. Zhou. Evaluating static analysis defect warnings on production software. In Proceedings of the 7th ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering, pages 1--8. ACM, 2007.

Digital Library

[11]

A. Balestrat. CCG: A random C code generator. https://github.com/Merkil/ccg/.

[12]

P. J. Brown. Error messages: The neglected area of the man/machine interface. Commun. ACM, 26(4):246--249, Apr. 1983.

Digital Library

[13]

Y. Chen, A. Groce, C. Zhang, W.-K. Wong, X. Fern, E. Eide, and J. Regehr. Taming Compiler Fuzzers. In PLDI, pages 197--208, 2013.

Digital Library

[14]

N. J. Coull. SNOOPIE: development of a learning support tool for novice programmers within a conceptual framework. Ph.d. thesis, University of St Andrews, St Andrews, Scotland, UK, Oct. 2008.

[15]

P. Cuoq, B. Monate, A. Pacalet, V. Prevosto, J. Regehr, B. Yakobowski, and X. Yang. Testing Static Analyzers with Randomly Generated Programs. In A. Goodloe and S. Person, editors, NASA Formal Methods, volume 7226 of Lecture Notes in Computer Science, pages 120--125. Springer Berlin Heidelberg, 2012.

Digital Library

[16]

T. Flowers, C. Carver, and J. Jackson. Empowering students and building confidence in novice programmers through gauntlet. In Frontiers in Education, 2004. FIE 2004. 34th Annual, pages T3H/10--T3H/13 Vol. 1, Oct 2004.

[17]

B. Hartmann, D. MacDougall, J. Brandt, and S. R. Klemmer. What Would Other Programmers Do: Suggesting Solutions to Error Messages. In CHI, pages 1019--1028, 2010.

Digital Library

[18]

M. Howard. A Process for Performing Security Code Reviews. IEEE Security & Privacy, 4(4):0074--79, 2006.

Digital Library

[19]

International Organization for Standarization. ISO/IEC9899:201x:Programming Languages--C. http://www.open-std.org/JTC1/SC22/WG14/www/docs/n1570.pdf, May 2011.

[20]

Y. Jia and M. Harman. An Analysis and Survey of the Development of Mutation Testing. IEEE Trans. Softw. Eng., 37(5):649--678, Sept. 2011.

Digital Library

[21]

V. Le, M. Afshari, and Z. Su. Compiler Validation via Equivalence Modulo Inputs. In PLDI, 2014.

Digital Library

[22]

V. Le, C. Sun, and Z. Su. Finding Deep Compiler Bugs via Guided Stochastic Program Mutation. In Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2015, pages 386--399, New York, NY, USA, 2015. ACM.

Digital Library

[23]

V. Le, C. Sun, and Z. Su. Randomized Stress-testing of Link-time Optimizers. In Proceedings of the 2015 International Symposium on Software Testing and Analysis, pages 327--337, 2015.

Digital Library

[24]

B. S. Lerner, M. Flower, D. Grossman, and C. Chambers. Searching for Type-error Messages. In PLDI, pages 425--434, 2007.

Digital Library

[25]

L. Martignoni, R. Paleari, G. Fresi Roglia, and D. Bruschi. Testing System Virtual Machines. In ISSTA, pages 171--182, 2010.

Digital Library

[26]

L. Martignoni, R. Paleari, A. Reina, G. F. Roglia, and D. Bruschi. A methodology for testing cpu emulators. ACM Trans. Softw. Eng. Methodol., 22(4):29:1--29:26, Oct. 2013.

Digital Library

[27]

W. M. McKeeman. Differential testing for software. Digital Technical Journal, 10(1):100--107, 1998.

[28]

S. McPeak, D. S. Wilkerson, and S. Goldsmith. Berkeley Delta. http://delta.tigris.org/.

[29]

E. Nagai, A. Hashimoto, and N. Ishiura. Scaling up size and number of expressions in random testing of arithmetic optimization of C compilers. In SASIMI), pages 88--93, 2013.

[30]

P. Norvig. Learning Programming (by Humans, by Machine). http://vimeo.com/69631070, accessed: 2014-03-04.

[31]

T. Pearse and P. Oman. Maintainability measurements on industrial source code maintenance activities. In Software Maintenance, 1995. Proceedings., International Conference on, pages 295--303. IEEE, 1995.

Digital Library

[32]

Plum Hall, Inc. The Plum Hall Validation Suite for C. http://www.plumhall.com/stec.html.

[33]

J. Regehr, Y. Chen, P. Cuoq, E. Eide, C. Ellison, and X. Yang. Test-Case Reduction for C Compiler Bugs. In PLDI, pages 335--346, 2012.

Digital Library

[34]

B. Shneiderman. Designing computer system messages. Commun. ACM, 25(9):610--611, Sept. 1982.

Digital Library

[35]

V. J. Traver. On compiler error messages: What they say and what they mean. Adv. in Hum.-Comp. Int., 2010:3:1--3:26, Jan. 2010.

Digital Library

[36]

M. Wand. Finding the Source of Type Errors. In POPL, pages 38--43, 1986.

Digital Library

[37]

X. Yang, Y. Chen, E. Eide, and J. Regehr. Finding and Understanding Bugs in C Compilers. In PLDI, pages 283--294, 2011.

Digital Library

[38]

A. Zeller and R. Hildebrandt. Simplifying and isolating failure-inducing input. Software Engineering, IEEE Transactions on, 28(2):183--200, 2002.

Digital Library

[39]

C. Zhao, Y. Xue, Q. Tao, L. Guo, and Z. Wang. Automated Test Program Generation for an Industrial Optimizing Compiler. In AST, pages 36--43, 2009.

Cited By

Sanusi BOgunshile EAydin MOlabiyisi S(2024)Assuring Correctness, Testing, and Verification of X-Compiler by Integrating Communicating Stream X-MachineProceedings of the 2024 6th International Conference on Software Engineering and Development10.1145/3686614.3686622(64-72)Online publication date: 29-May-2024
https://dl.acm.org/doi/10.1145/3686614.3686622
Zeng JTang YZhang JLi WHuang Z(2024)Detecting Optimizing Compiler Bugs via History-Driven Test Program MutationProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3671387(145-154)Online publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1145/3671016.3671387
Tan JRigger MChristakis MPradel M(2024)Inconsistencies in TeX-Produced DocumentsProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680370(1415-1427)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680370
Show More Cited By

Index Terms

Finding and analyzing compiler warning defects
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. Usability testing
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software notations and tools
    1. Compilers

Recommendations

Finding compiler bugs via live code mutation
OOPSLA 2016: Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications

Validating optimizing compilers is challenging because it is hard to generate valid test programs (i.e., those that do not expose any undefined behavior). Equivalence Modulo Inputs (EMI) is an effective, promising methodology to tackle this problem. ...
Finding deep compiler bugs via guided stochastic program mutation
OOPSLA '15

Compiler testing is important and challenging. Equivalence Modulo Inputs (EMI) is a recent promising approach for compiler validation. It is based on mutating the unexecuted statements of an existing program under some inputs to produce new equivalent ...
Toward understanding compiler bugs in GCC and LLVM
ISSTA 2016: Proceedings of the 25th International Symposium on Software Testing and Analysis

Compilers are critical, widely-used complex software. Bugs in them have significant impact, and can cause serious damage when they silently miscompile a safety-critical application. An in-depth understanding of compiler bugs can help detect and fix ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '16: Proceedings of the 38th International Conference on Software Engineering

May 2016

1235 pages

ISBN:9781450339001

DOI:10.1145/2884781

General Chair:
Laura Dillon
Michigan State University
,
Program Chairs:
Willem Visser
Stellenbosch University, South Africa
,
Laurie Williams
North Carolina State University

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

ACM: Association for Computing Machinery
SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS\TCSE: TC on Software Engineering
IEEE-CS\DATC: IEEE Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 May 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

ICSE '16

Sponsor:

ACM
SIGSOFT
IEEE-CS\TCSE
IEEE-CS\DATC

ICSE '16: 38th International Conference on Software Engineering

May 14 - 22, 2016

Texas, Austin

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

57
Total Citations
View Citations
448
Total Downloads

Downloads (Last 12 months)26
Downloads (Last 6 weeks)2

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sanusi BOgunshile EAydin MOlabiyisi S(2024)Assuring Correctness, Testing, and Verification of X-Compiler by Integrating Communicating Stream X-MachineProceedings of the 2024 6th International Conference on Software Engineering and Development10.1145/3686614.3686622(64-72)Online publication date: 29-May-2024
https://dl.acm.org/doi/10.1145/3686614.3686622
Zeng JTang YZhang JLi WHuang Z(2024)Detecting Optimizing Compiler Bugs via History-Driven Test Program MutationProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3671387(145-154)Online publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1145/3671016.3671387
Tan JRigger MChristakis MPradel M(2024)Inconsistencies in TeX-Produced DocumentsProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680370(1415-1427)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680370
Kim HKim SLee JCha SChristakis MPradel M(2024)AsFuzzer: Differential Testing of Assemblers with Error-Driven Grammar InferenceProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680345(1099-1111)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680345
Xia XFeng YShi QJones JZhang XXu B(2024)Enumerating Valid Non-Alpha-Equivalent Programs for Interpreter TestingACM Transactions on Software Engineering and Methodology10.1145/364799433:5(1-31)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1145/3647994
Johansson AHolmberg CGomes De Oliveira Neto FLeitner PBaysal OLinares-Vasquez MMoran KSteinmacher I(2024)The Impact of Compiler Warnings on Code Quality in C++ ProjectsProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension10.1145/3643916.3644410(270-279)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643916.3644410
Li XGuo SCheng HJiang H(2024)Simulink Compiler Testing via Configuration Diversification With Reinforcement LearningIEEE Transactions on Reliability10.1109/TR.2023.331764373:2(1060-1074)Online publication date: Jun-2024
https://doi.org/10.1109/TR.2023.3317643
Ullah SRashid A(2024)Porting to Morello: An In-depth Study on Compiler Behaviors, CERT Guideline Violations, and Security Implications2024 IEEE 9th European Symposium on Security and Privacy (EuroS&P)10.1109/EuroSP60621.2024.00028(381-397)Online publication date: 8-Jul-2024
https://doi.org/10.1109/EuroSP60621.2024.00028
Mitropoulos CSotiropoulos TIoannidis SMitropoulos D(2024)Syntax-Aware Mutation for Testing the Solidity CompilerComputer Security – ESORICS 202310.1007/978-3-031-51479-1_17(327-347)Online publication date: 12-Jan-2024
https://doi.org/10.1007/978-3-031-51479-1_17
Xu JLu KDu ZDing ZLi LWu QPayer MMao BCalandrino JTroncoso C(2023)Silent bugs matterProceedings of the 32nd USENIX Conference on Security Symposium10.5555/3620237.3620442(3655-3672)Online publication date: 9-Aug-2023
https://dl.acm.org/doi/10.5555/3620237.3620442
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents