research-article

It's not a bug, it's a feature: how misclassification impacts bug prediction

Authors:

Andreas ZellerAuthors Info & Claims

ICSE '13: Proceedings of the 2013 International Conference on Software Engineering

Pages 392 - 401

Published: 18 May 2013 Publication History

Abstract

In a manual examination of more than 7,000 issue reports from the bug databases of five open-source projects, we found 33.8% of all bug reports to be misclassified---that is, rather than referring to a code fix, they resulted in a new feature, an update to documentation, or an internal refactoring. This misclassification introduces bias in bug prediction models, confusing bugs and features: On average, 39% of files marked as defective actually never had a bug. We discuss the impact of this misclassification on earlier studies and recommend manual data validation for future studies.

References

[1]

G. Antoniol, K. Ayari, M. Di Penta, F. Khomh, and Y.-G. Guéhéneuc, “Is it a bug or an enhancement? A text-based approach to classify change requests,” in Proceedings of the 2008 conference of the center for advanced studies on collaborative research: meeting of minds. ACM, 2008, pp. 23:304–23:318.

Digital Library

[2]

T. Zimmermann, R. Premraj, and A. Zeller, “Predicting defects for Eclipse,” in Proceedings of the Third International Workshop on Predictor Models in Software Engineering. IEEE Computer Society, 2007, pp. 9–.

Digital Library

[3]

M. Fischer, M. Pinzger, and H. Gall, “Populating a release history database from version control and bug tracking systems,” in Proceedings of the International Conference on Software Maintenance. IEEE Computer Society, 2003, pp. 23–32.

Digital Library

[4]

D. ˇ Cubrani´c, G. C. Murphy, J. Singer, and K. S. Booth, “Hipikat: A project memory for software development,” IEEE Trans. Softw. Eng., vol. 31, no. 6, pp. 446–465, Jun. 2005.

Digital Library

[5]

J. Śliwerski, T. Zimmermann, and A. Zeller, “When do changes induce fixes?” in Proceedings of the 2005 international workshop on Mining software repositories. ACM, 2005, pp. 1–5.

Digital Library

[6]

S. Kim, T. Zimmermann, E. J. Whitehead Jr., and A. Zeller, “Predicting faults from cached history,” in Proceedings of the 29th international conference on Software Engineering. IEEE Computer Society, 2007, pp. 489–498.

Digital Library

[7]

R. Premraj and K. Herzig, “Network versus code metrics to predict defects: A replication study,” in Proceedings of the 2011 International Symposium on Empirical Software Engineering and Measurement. IEEE Computer Society, 2011, pp. 215–224.

Digital Library

[8]

N. Nagappan, A. Zeller, T. Zimmermann, K. Herzig, and B. Murphy, “Change bursts as defect predictors,” in Proceedings of the 2010 IEEE 21st International Symposium on Software Reliability Engineering. IEEE Computer Society, 2010, pp. 309–318.

Digital Library

[9]

A. Schröter, T. Zimmermann, and A. Zeller, “Predicting component failures at design time,” in Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering. ACM, 2006, pp. 18–27.

Digital Library

[10]

T. Zimmermann and N. Nagappan, “Predicting defects using network analysis on dependency graphs,” in Proceedings of the 30th international conference on Software engineering. ACM, 2008, pp. 531–540.

Digital Library

[11]

V. Dallmeier and T. Zimmermann, “Extraction of bug localization benchmarks from history,” in Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering. ACM, 2007, pp. 433–436.

Digital Library

[12]

S. Kim, E. J. Whitehead, Jr., and Y. Zhang, “Classifying software changes: Clean or buggy?” IEEE Trans. Softw. Eng., vol. 34, no. 2, pp. 181–196, Mar. 2008.

Digital Library

[13]

P. Hooimeijer and W. Weimer, “Modeling bug report quality,” in Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering. ACM, 2007, pp. 34–43.

Digital Library

[14]

N. Bettenburg, S. Just, A. Schröter, C. Weiß, R. Premraj, and T. Zimmermann, “Quality of bug reports in eclipse,” in Proceedings of the 2007 OOPSLA workshop on eclipse technology eXchange. ACM, 2007, pp. 21–25.

Digital Library

[15]

N. Bettenburg, S. Just, A. Schröter, C. Weiss, R. Premraj, and T. Zimmermann, “What makes a good bug report?” in Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering. ACM, 2008, pp. 308–318.

Digital Library

[16]

P. J. Guo, T. Zimmermann, N. Nagappan, and B. Murphy, “Characterizing and predicting which bugs get fixed: an empirical study of microsoft windows,” in Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1. ACM, 2010, pp. 495–504.

Digital Library

[17]

P. Runeson, M. Alexandersson, and O. Nyholm, “Detection of duplicate defect reports using natural language processing,” in Proceedings of the 29th international conference on Software Engineering. IEEE Computer Society, 2007, pp. 499–510.

Digital Library

[18]

X. Wang, L. Zhang, T. Xie, J. Anvik, and J. Sun, “An approach to detecting duplicate bug reports using natural language and execution information,” in Proceedings of the 30th international conference on Software engineering, 2008, pp. 461–470.

Digital Library

[19]

C. Weiss, R. Premraj, T. Zimmermann, and A. Zeller, “How long will it take to fix this bug?” in Proceedings of the Fourth International Workshop on Mining Software Repositories. Washington, DC, USA: IEEE Computer Society, 2007, pp. 1–.

Digital Library

[20]

J. J. Amor, G. Robles, and J. M. Gonzalez-Barahona, “Effort estimation by characterizing developer activity,” in Proceedings of the 2006 international workshop on Economics driven software engineering research. ACM, 2006, pp. 3–6.

Digital Library

[21]

H. Zeng and D. Rine, “Estimation of software defects fix effort using neural networks,” in Proceedings of the 28th Annual International Computer Software and Applications Conference - Workshops and Fast Abstracts - Volume 02. IEEE Computer Society, 2004, pp. 20–21.

Digital Library

[22]

E. Giger, M. Pinzger, and H. Gall, “Predicting the fix time of bugs,” in Proceedings of the 2nd International Workshop on Recommendation Systems for Software Engineering. ACM, 2010, pp. 52–56.

Digital Library

[23]

D. ˇ Cubrani´c, “Automatic bug triage using text categorization,” in In SEKE 2004: Proceedings of the Sixteenth International Conference on Software Engineering & Knowledge Engineering. KSI Press, 2004, pp. 92––97.

[24]

J. Anvik and G. C. Murphy, “Reducing the effort of bug report triage: Recommenders for development-oriented decisions,” ACM Trans. Softw. Eng. Methodol., vol. 20, no. 3, pp. 10:1–10:35, aug 2011.

Digital Library

[25]

J. Anvik, L. Hiew, and G. C. Murphy, “Who should fix this bug?” in Proceedings of the 28th international conference on Software engineering. ACM, 2006, pp. 361–370.

Digital Library

[26]

P. J. Guo, T. Zimmermann, N. Nagappan, and B. Murphy, ““Not my bug!” and other reasons for software bug report reassignments,” in Proceedings of the ACM 2011 conference on Computer supported cooperative work. ACM, 2011, pp. 395–404.

Digital Library

[27]

D. ˇ Cubrani´c and G. C. Murphy, “Hipikat: recommending pertinent software development artifacts,” in Proceedings of the 25th International Conference on Software Engineering. IEEE Computer Society, 2003, pp. 408–418.

Digital Library

[28]

A. Mockus, “Missing data in software engineering,” Guide to Advanced Empirical Software Engineering, pp. 185–200, 2008.

[29]

G. A. Liebchen and M. Shepperd, “Data sets and data quality in software engineering,” in Proceedings of the 4th international workshop on Predictor models in software engineering. ACM, 2008, pp. 39–44.

Digital Library

[30]

T. H. D. Nguyen, B. Adams, and A. E. Hassan, “A case study of bias in bug-fix datasets,” in Proceedings of the 2010 17th Working Conference on Reverse Engineering. IEEE Computer Society, 2010, pp. 259–268.

Digital Library

[31]

C. Bird, A. Bachmann, E. Aune, J. Duffy, A. Bernstein, V. Filkov, and P. Devanbu, “Fair and balanced?: bias in bug-fix datasets,” in Proceedings of the the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering. ACM, 2009, pp. 121–130.

Digital Library

[32]

A. Bachmann, C. Bird, F. Rahman, P. Devanbu, and A. Bernstein, “The missing links: bugs and bug-fix commits,” in Proceedings of the eighteenth ACM SIGSOFT international symposium on Foundations of software engineering. ACM, 2010, pp. 97–106.

Digital Library

[33]

T. H. Nguyen, B. Adams, and A. E. Hassan, “A Case Study of Bias in Bug-Fix Datasets,” in 2010 17th Working Conference on Reverse Engineering. IEEE Computer Society, 2010, pp. 259–268.

Digital Library

[34]

C. Bird, A. Bachmann, F. Rahman, and A. Bernstein, “Linkster: enabling efficient manual inspection and annotation of mined data,” in Proceedings of the eighteenth ACM SIGSOFT international symposium on Foundations of software engineering. ACM, 2010, pp. 369–370.

Digital Library

[35]

R. Wu, H. Zhang, S. Kim, and S.-C. Cheung, “Relink: recovering links between bugs and changes,” in Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. ACM, 2011, pp. 15–25.

Digital Library

[36]

S. Kim, H. Zhang, R. Wu, and L. Gong, “Dealing with noise in defect prediction,” in Proceeding of the 33rd international conference on Software engineering - ICSE ’11. New York, New York, USA: ACM Press, 2011, p. 481.

Digital Library

Cited By

Krüger JLi YLossev KZhu CChechik MBerger TRubin J(2024)A Meta-Study of Software-Change IntentionsACM Computing Surveys10.1145/366148456:12(1-41)Online publication date: 25-Apr-2024
https://dl.acm.org/doi/10.1145/3661484
Di Penta MLanubile F(2021)How Empirical Research Supports Tool DevelopmentProceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1145/3475716.3484488(1-3)Online publication date: 11-Oct-2021
https://dl.acm.org/doi/10.1145/3475716.3484488
Falessi DAhluwalia APenta M(2021)The Impact of Dormant Defects on Defect Prediction: A Study of 19 Apache ProjectsACM Transactions on Software Engineering and Methodology10.1145/346789531:1(1-26)Online publication date: 28-Sep-2021
https://dl.acm.org/doi/10.1145/3467895
Show More Cited By

Index Terms

It's not a bug, it's a feature: how misclassification impacts bug prediction
1. Social and professional topics
  1. Professional topics
    1. Management of computing and information systems
      1. Project and people management
      2. Software management
2. Software and its engineering
  1. Software creation and management
    1. Software development process management
    2. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software notations and tools
    1. Software configuration management and version control systems

Recommendations

Towards Semi-automatic Bug Triage and Severity Prediction Based on Topic Model and Multi-feature of Bug Reports
COMPSAC '14: Proceedings of the 2014 IEEE 38th Annual Computer Software and Applications Conference

Bug fixing is an essential activity in the software maintenance, because most of the software systems have unavoidable defects. When new bugs are submitted, triagers have to find and assign appropriate developers to fix the bugs. However, if the bugs are ...
Effective Bug Triage Based on Historical Bug-Fix Information
ISSRE '14: Proceedings of the 2014 IEEE 25th International Symposium on Software Reliability Engineering

For complex and popular software, project teams could receive a large number of bug reports. It is often tedious and costly to manually assign these bug reports to developers who have the expertise to fix the bugs. Many bug triage techniques have been ...
It's not a bug, it's a feature: does misclassification affect bug localization?
MSR 2014: Proceedings of the 11th Working Conference on Mining Software Repositories

Bug localization refers to the task of automatically processing bug reports to locate source code files that are responsible for the bugs. Many bug localization techniques have been proposed in the literature. These techniques are often evaluated on ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '13: Proceedings of the 2013 International Conference on Software Engineering

May 2013

1561 pages

ISBN:9781467330763

General Chair:
David Notkin,
Program Chairs:
Betty H. C. Cheng,
Klaus Pohl

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

IEEE Press

Publication History

Published: 18 May 2013

Check for updates

Qualifiers

Research-article

Conference

ICSE '13

Sponsor:

SIGSOFT

ICSE '13: 35th International Conference on Software Engineering

May 18 - 26, 2013

CA, San Francisco, USA

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

75
Total Citations
View Citations
1,283
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)1

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Krüger JLi YLossev KZhu CChechik MBerger TRubin J(2024)A Meta-Study of Software-Change IntentionsACM Computing Surveys10.1145/366148456:12(1-41)Online publication date: 25-Apr-2024
https://dl.acm.org/doi/10.1145/3661484
Di Penta MLanubile F(2021)How Empirical Research Supports Tool DevelopmentProceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1145/3475716.3484488(1-3)Online publication date: 11-Oct-2021
https://dl.acm.org/doi/10.1145/3475716.3484488
Falessi DAhluwalia APenta M(2021)The Impact of Dormant Defects on Defect Prediction: A Study of 19 Apache ProjectsACM Transactions on Software Engineering and Methodology10.1145/346789531:1(1-26)Online publication date: 28-Sep-2021
https://dl.acm.org/doi/10.1145/3467895
Babii HPrenner JStricker LKarmakar AJanes ARobbes RJuristo N(2021)Mining software repositories with a collaborative heuristic repositoryProceedings of the 43rd International Conference on Software Engineering: New Ideas and Emerging Results10.1109/ICSE-NIER52604.2021.00030(106-110)Online publication date: 25-May-2021
https://dl.acm.org/doi/10.1109/ICSE-NIER52604.2021.00030
Reichenbach CJuristo N(2021)Software ticks need no specificationsProceedings of the 43rd International Conference on Software Engineering: New Ideas and Emerging Results10.1109/ICSE-NIER52604.2021.00021(61-65)Online publication date: 25-May-2021
https://dl.acm.org/doi/10.1109/ICSE-NIER52604.2021.00021
Wang PBrown CJennings JStolee K(2020)An Empirical Study on Regular Expression BugsProceedings of the 17th International Conference on Mining Software Repositories10.1145/3379597.3387464(103-113)Online publication date: 29-Jun-2020
https://dl.acm.org/doi/10.1145/3379597.3387464
Trautsch ATrautsch FHerbold SLedel BGrabowski JRothermel GBae D(2020)The SmartSHARK ecosystem for software repository miningProceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Companion Proceedings10.1145/3377812.3382139(25-28)Online publication date: 27-Jun-2020
https://dl.acm.org/doi/10.1145/3377812.3382139
Shi LXing MLi MWang YLi SWang QRothermel GBae D(2020)Detection of hidden feature requests from massive chat messages via deep siamese networkProceedings of the ACM/IEEE 42nd International Conference on Software Engineering10.1145/3377811.3380356(641-653)Online publication date: 27-Jun-2020
https://dl.acm.org/doi/10.1145/3377811.3380356
Li MShi LYang YWang QGrundy JLe Goues CLo D(2020)A deep multitask learning approach for requirements discovery and annotation from open forumProceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering10.1145/3324884.3416627(336-348)Online publication date: 21-Dec-2020
https://dl.acm.org/doi/10.1145/3324884.3416627
Tufano MWatson CBavota GPenta MWhite MPoshyvanyk D(2019)An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine TranslationACM Transactions on Software Engineering and Methodology10.1145/334054428:4(1-29)Online publication date: 2-Sep-2019
https://dl.acm.org/doi/10.1145/3340544
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents