research-article

File-level defect prediction: unsupervised vs. supervised models

Authors:

Xiaohong ZhangAuthors Info & Claims

ESEM '17: Proceedings of the 11th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

Pages 344 - 353

https://doi.org/10.1109/ESEM.2017.48

Published: 09 November 2017 Publication History

Abstract

Background: Software defect models can help software quality assurance teams to allocate testing or code review resources. A variety of techniques have been used to build defect prediction models, including supervised and unsupervised methods. Recently, Yang et al. [1] surprisingly find that unsupervised models can perform statistically significantly better than supervised models in effort-aware change-level defect prediction. However, little is known about relative performance of unsupervised and supervised models for effort-aware file-level defect prediction. Goal: Inspired by their work, we aim to investigate whether a similar finding holds in effort-aware file-level defect prediction. Method: We replicate Yang et al.'s study on PROMISE dataset with totally ten projects. We compare the effectiveness of unsupervised and supervised prediction models for effort-aware file-level defect prediction. Results: We find that the conclusion of Yang et al. [1] does not hold under within-project but holds under cross-project setting for file-level defect prediction. In addition, following the recommendations given by the best unsupervised model, developers needs to inspect statistically significantly more files than that of supervised models considering the same inspection effort (i.e., LOC). Conclusions: (a) Unsupervised models do not perform statistically significantly better than state-of-art supervised model under within-project setting, (b) Unsupervised models can perform statistically significantly better than state-of-art supervised model under cross-project setting, (c) We suggest that not only LOC but also number of files needed to be inspected should be considered when evaluating effort-aware file-level defect prediction models.

References

[1]

Y. Yang, Y. Zhou, J. Liu, Y. Zhao, H. Lu, L. Xu, B. Xu, and H. Leung, "Effort-aware just-in-time defect prediction: simple unsupervised models could be better than supervised models," in FSE. ACM, 2016, pp. 157--168.

Digital Library

[2]

G. Tassey, "The economic impacts of inadequate infrastructure for software testing," National Institute of Standards and Technology, RTI Project, vol. 7007, no. 011, 2002.

[3]

O. F. Arar and K. Ayan, "Software defect prediction using cost-sensitive neural network," Applied Soft Computing, vol. 33, pp. 263--277, 2015.

Digital Library

[4]

E. Ceylan, F. O. Kutlubay, and A. B. Bener, "Software defect identification using machine learning techniques," in 32nd EUROMICRO Conference on Software Engineering and Advanced Applications. IEEE, 2006, pp. 240--247.

Digital Library

[5]

M. E. Fagan, "Design and code inspections to reduce errors in program development," IBM Systems Journal, vol. 38, no. 2/3, p. 258, 1999.

Digital Library

[6]

R. Moser, W. Pedrycz, and G. Succi, "A comparative analysis of the efficiency of change metrics and static code attributes for defect prediction," in ICSE. ACM, 2008, pp. 181--190.

Digital Library

[7]

F. Shull, V. Basili, B. Boehm, A. W. Brown, P. Costa, M. Lindvall, D. Port, I. Rus, R. Tesoriero, and M. Zelkowitz, "What we have learned about fighting defects," in Proceedings of the Eighth IEEE Symposium on Software Metrics. IEEE, 2002, pp. 249--258.

Digital Library

[8]

T. Menzies, J. Greenwald, and A. Frank, "Data mining static code attributes to learn defect predictors," TSE, vol. 33, no. 1, 2007.

Digital Library

[9]

N. Ohlsson and H. Alberg, "Predicting fault-prone software modules in telephone switches," TSE, vol. 22, no. 12, pp. 886--894, 1996.

Digital Library

[10]

V. R. Basili, L. C. Briand, and W. L. Melo, "A validation of object-oriented design metrics as quality indicators," TSE, vol. 22, no. 10, pp. 751--761, 1996.

Digital Library

[11]

M. DAmbros, M. Lanza, and R. Robbes, "Evaluating defect prediction approaches: a benchmark and an extensive comparison," Empirical Software Engineering, vol. 17, no. 4--5, pp. 531--577, 2012.

Digital Library

[12]

A. E. Hassan, "Predicting faults using the complexity of code changes," in ICSE. IEEE Computer Society, 2009, pp. 78--88.

Digital Library

[13]

B. Yang, Q. Yin, S. Xu, and P. Guo, "Software quality prediction using affinity propagation algorithm," in IJCNN. IEEE, 2008, pp. 1891--1896.

[14]

T. Lee, J. Nam, D. Han, S. Kim, and H. P. In, "Micro interaction metrics for defect prediction," in ESEC/FSE. ACM, 2011, pp. 311--321.

Digital Library

[15]

M. Li, H. Zhang, R. Wu, and Z.-H. Zhou, "Sample-based software defect prediction with active and semi-supervised learning," Automated Software Engineering, vol. 19, no. 2, pp. 201--230, 2012.

Digital Library

[16]

H. Lu and B. Cukic, "An adaptive approach with active learning in software fault prediction," in Proceedings of the 8th International Conference on Predictive Models in Software Engineering. ACM, 2012, pp. 79--88.

Digital Library

[17]

T. Zimmermann, N. Nagappan, H. Gall, E. Giger, and B. Murphy, "Cross-project defect prediction: a large scale experiment on data vs. domain vs. process," in ESEC/FSE. ACM, 2009, pp. 91--100.

Digital Library

[18]

X. Yang, D. Lo, X. Xia, and J. Sun, "Tlel: A two-layer ensemble learning approach for just-in-time defect prediction," IST, vol. 87, pp. 206--220, 2017.

Digital Library

[19]

X. Yang, D. Lo, X. Xia, Y. Zhang, and J. Sun, "Deep learning for just-in-time defect prediction," in QRS. IEEE, 2015, pp. 17--26.

Digital Library

[20]

X. Xia, D. Lo, X. Wang, and X. Yang, "Collective personalized change classification with multiobjective search," IEEE Transactions on Reliability, vol. 65, no. 4, pp. 1810--1829, 2016.

[21]

Y. Zhang, D. Lo, X. Xia, and J. Sun, "An empirical study of classifier combination for cross-project defect prediction," in COMPSAC, 2015, pp. 264--269.

Digital Library

[22]

A. Tosun, A. Bener, B. Turhan, and T. Menzies, "Practical considerations in deploying statistical methods for defect prediction: A case study within the turkish telecommunications industry," IST, vol. 52, no. 11, pp. 1242--1257, 2010.

Digital Library

[23]

B. Turhan, T. Menzies, A. B. Bener, and J. Di Stefano, "On the relative value of cross-company and within-company data for defect prediction," Empirical Software Engineering, vol. 14, no. 5, pp. 540--578, 2009.

Digital Library

[24]

F. Zhang, Q. Zheng, Y. Zou, and A. E. Hassan, "Cross-project defect prediction using a connectivity-based unsupervised classifier," in ICSE. ACM, 2016, pp. 309--320.

Digital Library

[25]

X. Xia, D. Lo, S. J. Pan, N. Nagappan, and X. Wang, "Hydra: Massively compositional model for cross-project defect prediction," TSE, vol. 42, no. 10, pp. 977--998, 2016.

Digital Library

[26]

A. E. Camargo Cruz and K. Ochimizu, "Towards logistic regression models for predicting fault-prone code across software projects," in ESEM. IEEE Computer Society, 2009, pp. 460--463.

Digital Library

[27]

J. Nam, S. J. Pan, and S. Kim, "Transfer defect learning," in ICSE. IEEE Press, 2013, pp. 382--391.

Digital Library

[28]

J. Nam and S. Kim, "Heterogeneous defect prediction," in ESEC/FSE. ACM, 2015, pp. 508--519.

Digital Library

[29]

S. Zhong, T. M. Khoshgoftaar, and N. Seliya, "Unsupervised learning for expert-based software quality estimation." in HASE. Citeseer, 2004, pp. 149--155.

Digital Library

[30]

Y. Kamei, S. Matsumoto, A. Monden, K.-i. Matsumoto, B. Adams, and A. E. Hassan, "Revisiting common bug prediction findings using effort-aware models," in ICSM. IEEE, 2010, pp. 1--10.

Digital Library

[31]

T. Mende and R. Koschke, "Effort-aware defect prediction models," in CSMR. IEEE, 2010, pp. 107--116.

Digital Library

[32]

E. Shihab, Y. Kamei, B. Adams, and A. E. Hassan, "Is lines of code a good measure of effort in effort-aware models?" IST, vol. 55, no. 11, pp. 1981--1993, 2013.

Digital Library

[33]

M. E. Mezouar, F. Zhang, and Y. Zou, "Local versus global models for effort-aware defect prediction," in Proceedings of the 26th Annual International Conference on Computer Science and Software Engineering. IBM Corp., 2016, pp. 178--187.

Digital Library

[34]

W. Fu and T. Menzies, "Revisiting unsupervised learning for defect prediction," in FSE. ACM, 2017, p. to appear.

Digital Library

[35]

Q. Huang, X. Xia, and D. Lo, "Supervised vs unsupervised models: A holistic look at effort-aware just-in-time defect prediction," in ICSME. IEEE, 2017, p. to appear.

[36]

B. Ghotra, S. McIntosh, and A. E. Hassan, "Revisiting the impact of classification techniques on the performance of defect prediction models," in ICSE. IEEE Press, 2015, pp. 789--800.

Digital Library

[37]

T. Hall, S. Beecham, D. Bowes, D. Gray, and S. Counsell, "A systematic literature review on fault prediction performance in software engineering," TSE, vol. 38, no. 6, pp. 1276--1304, 2012.

Digital Library

[38]

J. Nam and S. Kim, "Clami: Defect prediction on unlabeled datasets," in ASE. IEEE, 2015, pp. 452--463.

Digital Library

[39]

P. S. Bishnu and V. Bhattacherjee, "Software fault prediction using quad tree-based k-means clustering algorithm," IEEE Transactions on knowledge and data engineering, vol. 24, no. 6, pp. 1146--1150, 2012.

Digital Library

[40]

B. Fritzke et al., "A growing neural gas network learns topologies," Advances in neural information processing systems, vol. 7, pp. 625--632, 1995.

Digital Library

[41]

E. Arisholm and L. C. Briand, "Predicting fault-prone components in a java legacy system," in Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering. ACM, 2006, pp. 8--17.

Digital Library

[42]

G. Koru, H. Liu, D. Zhang, and K. El Emam, "Testing the theory of relative defect proneness for closed-source software," Empirical Software Engineering, vol. 15, no. 6, pp. 577--598, 2010.

Digital Library

[43]

A. G. Koru, K. El Emam, D. Zhang, H. Liu, and D. Mathew, "Theory of relative defect proneness," Empirical Software Engineering, vol. 13, no. 5, p. 473, 2008.

Digital Library

[44]

T. Menzies, Z. Milton, B. Turhan, B. Cukic, Y. Jiang, and A. Bener, "Defect prediction from static code features: current results, limitations, new approaches," Automated Software Engineering, vol. 17, no. 4, pp. 375--407, 2010.

Digital Library

[45]

T. M. Khoshgoftaar and E. B. Allen, "Ordering fault-prone software modules," Software Quality Journal, vol. 11, no. 1, pp. 19--37, 2003.

Digital Library

[46]

T. Mende and R. Koschke, "Revisiting the evaluation of defect prediction models," in Proceedings of the 5th International Conference on Predictor Models in Software Engineering. ACM, 2009, p. 7.

Digital Library

[47]

S. Lessmann, B. Baesens, C. Mues, and S. Pietsch, "Benchmarking classification models for software defect prediction: A proposed framework and novel findings," TSE, vol. 34, no. 4, pp. 485--496, 2008.

Digital Library

[48]

Y. Kamei, E. Shihab, B. Adams, A. E. Hassan, A. Mockus, A. Sinha, and N. Ubayashi, "A large-scale empirical study of just-in-time quality assurance," TSE, vol. 39, no. 6, pp. 757--773, 2013.

Digital Library

[49]

M. Jureczko and L. Madeyski, "Towards identifying software project clusters with regard to defect prediction," in Proceedings of the 6th International Conference on Predictive Models in Software Engineering. ACM, 2010, p. 9.

Digital Library

[50]

E. G. Jelihovschi, J. C. Faria, and I. B. Allaman, "Scottknott: a package for performing the scott-knott clustering algorithm in r," TEMA (São Carlos), vol. 15, no. 1, pp. 3--17, 2014.

[51]

J. Ferreira, A. Zwinderman et al., "On the benjamini-hochberg method," The Annals of Statistics, vol. 34, no. 4, pp. 1827--1849, 2006.

[52]

J. Romano, J. D. Kromrey, J. Coraggio, J. Skowronek, and L. Devine, "Exploring methods for evaluating group differences on the nsse and other surveys: Are the t-test and cohensd indices the most appropriate choices," in annual meeting of the Southern Association for Institutional Research, 2006.

Cited By

Wan XZheng ZQin FLu X(2024)Data Complexity: A New Perspective for Analyzing the Difficulty of Defect Prediction TasksACM Transactions on Software Engineering and Methodology10.1145/364959633:6(1-45)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3649596
Vescan AGăceanu RŞerban C(2024)Exploring the impact of data preprocessing techniques on composite classifier algorithms in cross-project defect predictionAutomated Software Engineering10.1007/s10515-024-00454-931:2Online publication date: 6-Jun-2024
https://doi.org/10.1007/s10515-024-00454-9
Zhang YZhu JYang YWen MJin H(2023)Comparing the Performance of Different Code Representations for Learning-based Vulnerability DetectionProceedings of the 14th Asia-Pacific Symposium on Internetware10.1145/3609437.3609464(174-184)Online publication date: 4-Aug-2023
https://dl.acm.org/doi/10.1145/3609437.3609464
Show More Cited By

Recommendations

Revisiting unsupervised learning for defect prediction
ESEC/FSE 2017: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering

Collecting quality data from software projects can be time-consuming and expensive. Hence, some researchers explore "unsupervised" approaches to quality prediction that does not require labelled data. An alternate technique is to use "supervised" ...
Heterogeneous defect prediction
ESEC/FSE 2015: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering

Software defect prediction is one of the most active research areas in software engineering. We can build a prediction model with defect data collected from a software project and predict defects in the same project, i.e. within-project defect ...
Further Comparison of Cross-Company and Within-Company Effort Estimation Models for Web Applications
METRICS '04: Proceedings of the Software Metrics, 10th International Symposium

This paper extends a previous study, using data on 67 Web projects from the Tukutuku database, investigating to what extent a cross-company cost model can be successfully employed to estimate effort for projects that belong to a single company, where no ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEM '17: Proceedings of the 11th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

November 2017

481 pages

ISBN:9781509040391

General Chair:
Ayse Bener
Ryerson University, Canada
,
Program Chairs:
Burak Turhan
Brunel University, UK
,
Stefan Biffl
Technische Universität Wien, Austria

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS\DATC: IEEE Computer Society

Publisher

IEEE Press

Publication History

Published: 09 November 2017

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ESEM '17

Sponsor:

SIGSOFT
IEEE-CS\DATC

ESEM '17: ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

November 9 - 10, 2017

Ontario, Markham, Canada

Acceptance Rates

Overall Acceptance Rate 130 of 594 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
115
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)1

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wan XZheng ZQin FLu X(2024)Data Complexity: A New Perspective for Analyzing the Difficulty of Defect Prediction TasksACM Transactions on Software Engineering and Methodology10.1145/364959633:6(1-45)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3649596
Vescan AGăceanu RŞerban C(2024)Exploring the impact of data preprocessing techniques on composite classifier algorithms in cross-project defect predictionAutomated Software Engineering10.1007/s10515-024-00454-931:2Online publication date: 6-Jun-2024
https://doi.org/10.1007/s10515-024-00454-9
Zhang YZhu JYang YWen MJin H(2023)Comparing the Performance of Different Code Representations for Learning-based Vulnerability DetectionProceedings of the 14th Asia-Pacific Symposium on Internetware10.1145/3609437.3609464(174-184)Online publication date: 4-Aug-2023
https://dl.acm.org/doi/10.1145/3609437.3609464
Yang PZhu LHu WKeung JLu LXiang J(2023)The Impact of the bug number on Effort-Aware Defect Prediction: An Empirical StudyProceedings of the 14th Asia-Pacific Symposium on Internetware10.1145/3609437.3609458(67-78)Online publication date: 4-Aug-2023
https://dl.acm.org/doi/10.1145/3609437.3609458
Guo ZLiu SLiu XLai WMa MZhang XNi CYang YLi YChen LZhou GZhou Y(2023)Code-line-level Bugginess Identification: How Far have We Come, and How Far have We Yet to Go?ACM Transactions on Software Engineering and Methodology10.1145/358257232:4(1-55)Online publication date: 27-May-2023
https://dl.acm.org/doi/10.1145/3582572
Yang YXia XLo DBi TGrundy JYang X(2022)Predictive Models in Software Engineering: Challenges and OpportunitiesACM Transactions on Software Engineering and Methodology10.1145/350350931:3(1-72)Online publication date: 9-Apr-2022
https://dl.acm.org/doi/10.1145/3503509
Bluemke IMalanowska A(2021)Software Testing Effort Estimation and Related ProblemsACM Computing Surveys10.1145/344269454:3(1-38)Online publication date: 17-Apr-2021
https://dl.acm.org/doi/10.1145/3442694
Meng DGuerriero MMachiry AAghakhani HBose PContinella AKruegel CVigna GCao JHo Au MLin ZYung M(2021)Bran: Reduce Vulnerability Search Space in Large Open Source Repositories by Learning Bug SymptomsProceedings of the 2021 ACM Asia Conference on Computer and Communications Security10.1145/3433210.3453115(731-743)Online publication date: 24-May-2021
https://dl.acm.org/doi/10.1145/3433210.3453115
Yan MXia XFan YLo DHassan AZhang XDevanbu PCohen MZimmermann T(2020)Effort-aware just-in-time defect identification in practice: a case study at AlibabaProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3368089.3417048(1308-1319)Online publication date: 8-Nov-2020
https://dl.acm.org/doi/10.1145/3368089.3417048
Qiao LLi XUmer QGuo P(2020)Deep learning based software defect predictionNeurocomputing10.1016/j.neucom.2019.11.067385:C(100-110)Online publication date: 14-Apr-2020
https://dl.acm.org/doi/10.1016/j.neucom.2019.11.067
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents