Article

Have things changed now?: an empirical study of bug characteristics in modern open source software

Authors:

Chengxiang ZhaiAuthors Info & Claims

ASID '06: Proceedings of the 1st workshop on Architectural and system support for improving software dependability

Pages 25 - 33

https://doi.org/10.1145/1181309.1181314

Published: 21 October 2006 Publication History

Abstract

Software errors are a major cause for system failures. To effectively design tools and support for detecting and recovering from software failures requires a deep understanding of bug characteristics. Recently, software and its development process have significantly changed in many ways, including more help from bug detection tools, shift towards multi-threading architecture, the open-source development paradigm and increasing concerns about security and user-friendly interface. Therefore, results from previous studies may not be applicable to present software. Furthermore, many new aspects such as security, concurrency and open-source-related characteristics have not well studied. Additionally, previous studies were based on a small number of bugs, which may lead to non-representative results.To investigate the impacts of the new factors on software errors, we analyze bug characteristics by first sampling hundreds of real world bugs in two large, representative open-source projects. To validate the representativeness of our results, we use natural language text classification techniques and automatically analyze around 29, 000 bugs from the Bugzilla databases of the software.Our study has discovered several new interesting characteristics: (1) memory-related bugs have decreased because quite a few effective detection tools became available recently; (2) surprisingly, some simple memory-related bugs such as NULL pointer dereferences that should have been detected by existing tools in development are still a major component, which indicates that the tools have not been used with their full capacity; (3) semantic bugs are the dominant root causes, as they are application specific and difficult to fix, which suggests that more efforts should be put into detecting and fixing them; (4) security bugs are increasing, and the majority of them cause severe impacts.

References

[1]

ASF bugzilla. http://issues.apache.org/bugzilla, 2005.

[2]

Coverity: Automated error prevention and source code analysis. http://www.coverity.com, 2005.

[3]

Def. of NVD severity metric. http://nvd.nist.gov/faq.cfm#8, 2005.

[4]

Mozilla.org Bugzilla. https://bugzilla.mozilla.org, 2005.

[5]

National vulnerability database. http://nvd.nist.gov, 2005.

[6]

Anonymous. Once upon a free(). http://www.phrack.org/phrack/57/p57-0x09.

[7]

B. Beizer. Software testing techniques (2nd ed.). Van Nostrand Reinhold Co., New York, NY, USA, 1990.

Digital Library

[8]

S. Chandra and P. Chen. Whither generic recovery from application faults? a fault study using open-source software. In DSN '00: Proceedings of the 2000 International Conference on Dependable Systems and Networks (formerly FTCS-30 and DCCA-8), 2000.

Digital Library

[9]

R. Chillarege, W.-L. Kao, and R. G. Condit. Defect type and its impact on the growth curve. In ICSE '91: Proceedings of the 13th international conference on Software engineering, 1991.

Digital Library

[10]

A. Chou, J. Yang, B. Chelf, S. Hallem, and D. R. Engler. An empirical study of operating system errors. In SOSP, 2001.

Digital Library

[11]

C. Cowan. Software security for open-source systems. IEEE Security and Privacy, 2003.

Digital Library

[12]

C. Cowan, C. Pu, D. Maier, J. Walpole, P. Bakke, S. Beattie, A. Grier, P. Wagle, Q. Zhang, and H. Hinton. StackGuard: Automatic adaptive detection and prevention of buffer-overflow attacks. In Proc. 7th USENIX Security Conference, pages 63--78, Jan 1998.

Digital Library

[13]

C. Cowan, P. Wagle, C. Pu, S. Beattie, and J. Walpole. Buffer overflows: Attacks and defenses for the vulnerability of the decade. In Proceedings of the DARPA Information Survivability Conference and Exposition (DISCEX), Jan. 2000.

[14]

A. Endres. An analysis of errors and their causes in system programs. In Proc. of the Intl. Conf. on Reliable software, 1975.

Digital Library

[15]

M. D. Ernst, J. Cockrell, W. G. Griswold, and D. Notkin. Dynamically discovering likely program invariants to support program evolution. IEEE TSE, 27(2):99--123, 2001.

Digital Library

[16]

N. E. Fenton and N. Ohlsson. Quantitative analysis of faults and failures in a complex software system. IEEE TSE, 2000.

Digital Library

[17]

R. Glass. Persistent software errors. IEEE TSE, 7(2), 1981.

[18]

J. Gray. Why do computer stop and what can be about it? In the 5th Symposium on Reliablity in Dist. Softw. and Database Sys., 1985.

[19]

W. Gu, Z. Kalbarczyk, R. K. Iyer, and Z.-Y. Yang. Characterization of linux kernel behavior under errors. In Proceedings of the 2003 International Conference on Dependable Systems and Networks, 2003.

[20]

R. Hastings and B. Joyce. Purify: Fast detection of memory leaks and access errors. In Proc. of the USENIX Winter Technical Conf., 1992.

[21]

T. Joachims. Learning to classify text using support vector machines. Kluwer Academic Publishers, 2002.

Digital Library

[22]

K. S. Jones, S. Walker, and S. E. Robertson. A probabilistic model of information retrieval: development and comparative experiments part 2. Inf. Process. Manage., 36(6):809--840, 2000.

Digital Library

[23]

Z. Li, S. Lu, S. Myagmar, and Y. Zhou. CP-Miner: A tool for finding copy-paste and related bugs in operating system code. In Sixth Symposium on Operating Systems Design and Implementation, 2004.

Digital Library

[24]

Z. Li and Y. Zhou. PR-Miner: Automatically extracting implicit programming rules and detecting violations in large software code. In 13th ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE'05), 2005.

Digital Library

[25]

S. Lu, Z. Li, F. Qin, L. Tan, P. Zhou, and Y. Zhou. BugBench: A benchmark for evaluating bug detection tools. In Bugs 2005 (Workshop on the Evaluation of Software Defect Detection Tools) on Programming Language Design and Implementation (PLDI) 2005, 2005.

[26]

A. M. Memon. GUI testing: Pitfalls and process. Computer, 2002.

Digital Library

[27]

N. Nethercote and J. Seward. Valgrind: A program supervision framework. In the 3rd Workshop on Runtime Verification, 2003.

[28]

T. Ostrand and E. Weyuker. Collecting and categorizing software error data in an industrial environment. Journal of Sys. and Softw., 1984.

Digital Library

[29]

T. Ostrand and E. Weyuker. The distribution of faults in a large industrial software system. In ISSTA '02: Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis, 2002.

Digital Library

[30]

T. Ostrand, E. Weyuker, and R. Bell. Predicting the location and number of faults in large software systems. IEEE TSE, 31(4), 2005.

Digital Library

[31]

C. Payne. On the security of open source software. Information Systems Journal, 2002.

[32]

A. Podgurski, D. Leon, P. Francis, W. Masri, M. Minch, J. Sun, and B. Wang. Automated support for classifying software failure reports. In ICSE '03: Proceedings of the 23th International Conference on Software Engineering, pages 465--475, 2003.

Digital Library

[33]

D. Roth and D. Zelenko. Part of speech tagging using a network of linear separators. In COLING-ACL, pages 1136--1142, 1998.

Digital Library

[34]

G. Rothermel, R. Untch, C. Chu, and M. Harrold. Test case prioritization: An empirical study. In ICSM '99: Proceedings of the IEEE International Conference on Software Maintenance, 1999.

Digital Library

[35]

M. Sullivan and R. Chillarege. Software defects and their impact on system availability - a study of field failures in operating systems. In 21st Int. Symp. on Fault-Tolerant Computing (FTCS-21), 1991.

[36]

M. Sullivan and R. Chillarege. A comparison of software defects in database management systems and operating systems. In FTCS '92: 22nd Annual International Symposium on Fault-Tolerant Computing, 1992.

[37]

V. Vapnik. The Nature of Statistical Learning Theory. Springer, 1995.

Digital Library

Cited By

Venkitaraman A(2024)Early Bug Detection through Shift Left TestingInternational Journal of Innovative Science and Research Technology (IJISRT)10.38124/ijisrt/IJISRT24NOV177(185-190)Online publication date: 18-Nov-2024
https://doi.org/10.38124/ijisrt/IJISRT24NOV177
Liu BZhang LLiu ZJiang J(2024)Developer Assignment Method for Software Defects Based on Related Issue PredictionMathematics10.3390/math1203042512:3(425)Online publication date: 28-Jan-2024
https://doi.org/10.3390/math12030425
He WDi PMing MZhang CSu TLi SSui Y(2024)Finding and Understanding Defects in Static Analyzers by Constructing Automated OraclesProceedings of the ACM on Software Engineering10.1145/36607811:FSE(1656-1678)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660781
Show More Cited By

Index Terms

Have things changed now?: an empirical study of bug characteristics in modern open source software

Recommendations

Bug characteristics in open source software

To design effective tools for detecting and recovering from software failures requires a deep understanding of software bug characteristics. We study software bug characteristics by sampling 2,060 real world bugs in three large, representative open-...
A comprehensive empirical study on bug characteristics of deep learning frameworks
Abstract Context:
Deep Learning (DL) frameworks enable developers to build DNN models without learning the underlying algorithms and models. While some of these DL-based software systems have been deployed in safety-critical areas, ...
A comprehensive study on security bug characteristics
Abstract
Security bugs can catastrophically impact our increasingly digital lives. Designing effective tools for detecting and fixing software security bugs requires a deep understanding of security bug characteristics. In this paper, we conducted a ...

From the figure above, we can find that data leakage is the main consequence of authentication restriction and data encryption security bugs. In addition, memory operation is the most common security bug, but its project imbalance means that the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ASID '06: Proceedings of the 1st workshop on Architectural and system support for improving software dependability

October 2006

76 pages

ISBN:1595935762

DOI:10.1145/1181309

Moderator:
Josep Torrellas
University of Illinois

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 October 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ASPLOS06

ASPLOS06: Architectural Support for Programming Languages and Operating Systems

October 21, 2006

California, San Jose

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

213
Total Citations
View Citations
2,902
Total Downloads

Downloads (Last 12 months)108
Downloads (Last 6 weeks)15

Reflects downloads up to 18 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Venkitaraman A(2024)Early Bug Detection through Shift Left TestingInternational Journal of Innovative Science and Research Technology (IJISRT)10.38124/ijisrt/IJISRT24NOV177(185-190)Online publication date: 18-Nov-2024
https://doi.org/10.38124/ijisrt/IJISRT24NOV177
Liu BZhang LLiu ZJiang J(2024)Developer Assignment Method for Software Defects Based on Related Issue PredictionMathematics10.3390/math1203042512:3(425)Online publication date: 28-Jan-2024
https://doi.org/10.3390/math12030425
He WDi PMing MZhang CSu TLi SSui Y(2024)Finding and Understanding Defects in Static Analyzers by Constructing Automated OraclesProceedings of the ACM on Software Engineering10.1145/36607811:FSE(1656-1678)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660781
Liu PLuo RJiang CGao TLi Y(2024)AI-Assisted Bug Detection in Open-Source Software2024 11th International Conference on Dependable Systems and Their Applications (DSA)10.1109/DSA63982.2024.00065(428-429)Online publication date: 2-Nov-2024
https://doi.org/10.1109/DSA63982.2024.00065
Semujju SLiu FHuang HXiang YYan XHao Z(2024)Knowledge transfer based many-objective approach for finding bugs in multi-path loopsComplex & Intelligent Systems10.1007/s40747-023-01323-wOnline publication date: 24-Jan-2024
https://doi.org/10.1007/s40747-023-01323-w
Lai TSimmons ABarnett SSchneider JVasa R(2024)Comparative analysis of real issues in open-source machine learning projectsEmpirical Software Engineering10.1007/s10664-024-10467-329:3Online publication date: 2-May-2024
https://doi.org/10.1007/s10664-024-10467-3
Gatla OZhang DXu WZheng M(2023)Understanding Persistent-memory-related Issues in the Linux KernelACM Transactions on Storage10.1145/360594619:4(1-28)Online publication date: 3-Oct-2023
https://dl.acm.org/doi/10.1145/3605946
He LSu PZhang CCai YMa JDruschel PKaufmann AMace JFlinn JSeltzer M(2023)One Simple API Can Cause Hundreds of Bugs An Analysis of Refcounting Bugs in All Modern Linux KernelsProceedings of the 29th Symposium on Operating Systems Principles10.1145/3600006.3613162(52-65)Online publication date: 23-Oct-2023
https://dl.acm.org/doi/10.1145/3600006.3613162
Xiong YXu MSu TSun JWang JWen HPu GHe JSu ZJust RFraser G(2023)An Empirical Study of Functional Bugs in Android AppsProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598138(1319-1331)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598138
Assi MHassan SGeorgiou SZou Y(2023)Predicting the Change Impact of Resolving Defects by Leveraging the Topics of Issue Reports in Open Source Software SystemsACM Transactions on Software Engineering and Methodology10.1145/359380232:6(1-34)Online publication date: 30-Sep-2023
https://dl.acm.org/doi/10.1145/3593802
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents