research-article

Automatically learning semantic features for defect prediction

Authors:

Song Wang,

Taiyue Liu,

Lin TanAuthors Info & Claims

ICSE '16: Proceedings of the 38th International Conference on Software Engineering

Pages 297 - 308

https://doi.org/10.1145/2884781.2884804

Published: 14 May 2016 Publication History

Get Access

Abstract

Software defect prediction, which predicts defective code regions, can help developers find bugs and prioritize their testing efforts. To build accurate prediction models, previous studies focus on manually designing features that encode the characteristics of programs and exploring different machine learning algorithms. Existing traditional features often fail to capture the semantic differences of programs, and such a capability is needed for building accurate prediction models.

To bridge the gap between programs' semantics and defect prediction features, this paper proposes to leverage a powerful representation-learning algorithm, deep learning, to learn semantic representation of programs automatically from source code. Specifically, we leverage Deep Belief Network (DBN) to automatically learn semantic features from token vectors extracted from programs' Abstract Syntax Trees (ASTs).

Our evaluation on ten open source projects shows that our automatically learned semantic features significantly improve both within-project defect prediction (WPDP) and cross-project defect prediction (CPDP) compared to traditional features. Our semantic features improve WPDP on average by 14.7% in precision, 11.5% in recall, and 14.2% in F1. For CPDP, our semantic features based approach outperforms the state-of-the-art technique TCA+ with traditional features by 8.9% in F1.

References

[1]

S. Amasaki, Y. Takagi, O. Mizuno, and T. Kikuno. A Bayesian Belief Network for Assessing the Likelihood of Fault Content. In ISSRE'03, pages 215--226.

Digital Library

Google Scholar

[2]

Y. Bengio. Learning Deep Architectures for AI. Foundations and Trends in Machine Learning, 2(1):1--127, 2009.

Digital Library

Google Scholar

[3]

D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent dirichlet allocation. the Journal of machine Learning research, 3:993--1022, 2003.

Digital Library

Google Scholar

[4]

T.-H. Chen, S. W. Thomas, M. Nagappan, and A. E. Hassan. Explaining software defects using topic models. In MSR'12, pages 189--198.

Digital Library

Google Scholar

[5]

S. R. Chidamber and C. F. Kemerer. A Metrics Suite for Object Oriented Design. TSE'94, 20(6):476--493.

Digital Library

Google Scholar

[6]

D. Ciresan, U. Meier, and J. Schmidhuber. Multi-column deep neural networks for image classification. In CVPR'12, pages 3642--3649.

Digital Library

Google Scholar

[7]

F. B. e Abreu and R. Carapuça. Candidate metrics for object-oriented software within a taxonomy framework. JSS'94, 26(1):87--96.

Digital Library

Google Scholar

[8]

K. O. Elish and M. O. Elish. Predicting defect-prone software modules using support vector machines. JSS'08, 81(5):649--660.

Digital Library

Google Scholar

[9]

N. Gayatri, S. Nickolas, A. Reddy, S. Reddy, and A. Nickolas. Feature selection using decision tree induction in class level metrics dataset for software defect predictions. In WCECS'10, pages 124--129.

Google Scholar

[10]

M. H. Halstead. Elements of Software Science (Operating and programming systems series). Elsevier Science Inc., 1977.

Digital Library

Google Scholar

[11]

R. Harrison, S. J. Counsell, and R. V. Nithi. An evaluation of the mood set of object-oriented software metrics. TSE'98, 24(6):491--496.

Digital Library

Google Scholar

[12]

A. E. Hassan. Predicting faults using the complexity of code changes. In ICSE'09, pages 78--88.

Digital Library

Google Scholar

[13]

Z. He, F. Peters, T. Menzies, and Y. Yang. Learning from open-source projects: An empirical study on defect prediction. In ESEM'13, pages 45--54.

Google Scholar

[14]

K. Herzig, S. Just, and A. Zeller. It's not a bug, it's a feature: how misclassification impacts bug prediction. In ICSE'13, pages 392--401.

Digital Library

Google Scholar

[15]

A. Hindle, E. T. Barr, Z. Su, M. Gabel, and P. Devanbu. On the naturalness of software. In ICSE'12, pages 837--847.

Digital Library

Google Scholar

[16]

G. E. Hinton, S. Osindero, and Y.-W. Teh. A fast learning algorithm for deep belief nets. Neural computation'06, 18(7):1527--1554.

Digital Library

Google Scholar

[17]

G. E. Hinton and R. R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science'06, 313(5786):504--507.

Google Scholar

[18]

T. Jiang, L. Tan, and S. Kim. Personalized defect prediction. In ASE'13, pages 279--289.

Google Scholar

[19]

X. Jing, F. Wu, X. Dong, F. Qi, and B. Xu. Heterogeneous cross-company defect prediction by unified metric representation and cca-based transfer learning. In FSE'15, pages 496--507.

Digital Library

Google Scholar

[20]

X.-Y. Jing, S. Ying, Z.-W. Zhang, S.-S. Wu, and J. Liu. Dictionary learning based software defect prediction. In ICSE'14, pages 414--423.

Digital Library

Google Scholar

[21]

T. Khoshgoftaar and N. Seliya. Tree-based software quality estimation models for fault prediction. In Software Metrics'02, pages 203--214.

Digital Library

Google Scholar

[22]

S. Kim, H. Zhang, R. Wu, and L. Gong. Dealing with noise in defect prediction. In ICSE'11, pages 481--490.

Digital Library

Google Scholar

[23]

S. Kim, T. Zimmermann, E. J. Whitehead Jr, and A. Zeller. Predicting faults from cached history. In ICSE'07, pages 489--498.

Digital Library

Google Scholar

[24]

S. Kombrink, T. Mikolov, M. Karafiát, and L. Burget. Recurrent neural network based language modeling in meeting recognition. In INTERSPEECH'11, pages 2877--2880.

Google Scholar

[25]

A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems'12, pages 1097--1105.

Google Scholar

[26]

A. Lam, A. Nguyen, H. Nguyen, and T. Nguyen. Combining deep learning with information retrieval to localize buggy files for bug reports. In ASE'15, pages 476--481.

Google Scholar

[27]

T. Lee, J. Nam, D. Han, S. Kim, and H. P. In. Micro interaction metrics for defect prediction. In FSE'11, pages 311--321.

Digital Library

Google Scholar

[28]

Z. Li and Y. Zhou. Pr-miner: automatically extracting implicit programming rules and detecting violations in large software code. In FSE'05, pages 306--315.

Digital Library

Google Scholar

[29]

Y. Liu, D. Poshyvanyk, R. Ferenc, T. Gyimóthy, and N. Chrisochoides. Modeling class cohesion as mixtures of latent topics. In ICSM'09, pages 233--242.

Google Scholar

[30]

C. D. Manning and H. Schütze. Foundations of statistical natural language processing. MIT press, 1999.

Digital Library

Google Scholar

[31]

T. J. McCabe. A complexity measure. TSE'76, (4):308--320.

Digital Library

Google Scholar

[32]

A. Meneely, L. Williams, W. Snipes, and J. Osborne. Predicting failures with developer networks and social network analysis. In FSE'08, pages 13--23.

Digital Library

Google Scholar

[33]

T. Menzies, J. Greenwald, and A. Frank. Data mining static code attributes to learn defect predictors. TSE'07, 33(1):2--13.

Digital Library

Google Scholar

[34]

T. Menzies, Z. Milton, B. Turhan, B. Cukic, Y. Jiang, and A. Bener. Defect prediction from static code features: current results, limitations, new approaches. ASE'10, 17(4):375--407.

Digital Library

Google Scholar

[35]

A. Mnih and G. E. Hinton. A scalable hierarchical distributed language model. In Advances in neural information processing systems'09, pages 1081--1088.

Google Scholar

[36]

A. Mohamed, D. Yu, and L. Deng. Investigation of full-sequence training of deep belief networks for speech recognition. In INTERSPEECH'10, pages 2846--2849.

Google Scholar

[37]

A.-r. Mohamed, G. E. Dahl, and G. Hinton. Acoustic modeling using deep belief networks. Audio, Speech, and Language Processing, IEEE Transactions on, 20(1):14--22, 2012.

Digital Library

Google Scholar

[38]

R. Moser, W. Pedrycz, and G. Succi. A comparative analysis of the efficiency of change metrics and static code attributes for defect prediction. In ICSE'08, pages 181--190.

Digital Library

Google Scholar

[39]

L. Mou, G. Li, Z. Jin, L. Zhang, and T. Wang. TBCNN: A tree-based convolutional neural network for programming language processing. arXiv preprint arXiv:1409.5718, 2014.

Google Scholar

[40]

N. Nagappan and T. Ball. Using software dependencies and churn metrics to predict field failures: An empirical case study. In ESEM'07, pages 364--373.

Digital Library

Google Scholar

[41]

J. Nam and S. Kim. Heterogeneous defect prediction. In FSE'15, pages 508--519.

Digital Library

Google Scholar

[42]

J. Nam, S. J. Pan, and S. Kim. Transfer defect learning. In ICSE'13, pages 382--391.

Digital Library

Google Scholar

[43]

G. Navarro. A guided tour to approximate string matching. ACM Comput. Surv., 33(1), 2001.

Digital Library

Google Scholar

[44]

A. T. Nguyen and T. N. Nguyen. Graph-based statistical language model for code. In ICSE'15, pages 858--868.

Digital Library

Google Scholar

[45]

A. T. Nguyen, T. T. Nguyen, T. N. Nguyen, D. Lo, and C. Sun. Duplicate bug report detection with a combination of information retrieval and topic modeling. In ASE'12, pages 70--79.

Digital Library

Google Scholar

[46]

T. T. Nguyen, H. A. Nguyen, N. H. Pham, J. M. Al-Kofahi, and T. N. Nguyen. Graph-based mining of multiple object usage patterns. In FSE'09, pages 383--392.

Digital Library

Google Scholar

[47]

T. T. Nguyen, T. N. Nguyen, and T. M. Phuong. Topic-based defect prediction. In ICSE'11, pages 932--935.

Digital Library

Google Scholar

[48]

T. J. Ostrand, E. J. Weyuker, and R. M. Bell. Programmer-based fault prediction. In PROMISE'10, pages 19:1--19:10.

Digital Library

Google Scholar

[49]

S. J. Pan, I. Tsang, J. Kwok, and Q. Yang. Domain adaptation via transfer component analysis. Neural Networks, IEEE Transactions on, pages 199--210, 2011.

Digital Library

Google Scholar

[50]

R. Pascanu, J. W. Stokes, H. Sanossian, M. Marinescu, and A. Thomas. Malware classification with recurrent networks. In ICASSP'15, pages 1916--1920.

Google Scholar

[51]

M. Pinzger, N. Nagappan, and B. Murphy. Can developer-module networks predict failures? In FSE'08, pages 2--12.

Digital Library

Google Scholar

[52]

F. Rahman and P. Devanbu. How, and why, process metrics are better. In ICSE'13, pages 432--441.

Digital Library

Google Scholar

[53]

V. Raychev, M. Vechev, and E. Yahav. Code completion with statistical language models. In PLDI'14, pages 419--428.

Digital Library

Google Scholar

[54]

R. Salakhutdinov and G. Hinton. Semantic hashing. RBM'07, 500(3):500.

Google Scholar

[55]

R. Sarikaya, G. E. Hinton, and A. Deoras. Application of deep belief networks for natural language understanding. Audio, Speech, and Language Processing, IEEE/ACM Transactions on, 22(4):778--784, 2014.

Digital Library

Google Scholar

[56]

A. Tamrawi, T. T. Nguyen, J. M. Al-Kofahi, and T. N. Nguyen. Fuzzy set and cache-based approach for bug triaging. In FSE'11, pages 365--375.

Digital Library

Google Scholar

[57]

M. Tan, L. Tan, S. Dara, and C. Mayeux. Online defect prediction for imbalanced data. In ICSE'15, pages 99--108.

Digital Library

Google Scholar

[58]

C. Tantithamthavorn, S. McIntosh, A. E. Hassan, A. Ihara, and K. ichi Matsumoto. The impact of mislabelling on the performance and interpretation of defect prediction models. In ICSE'15, pages 812--823.

Digital Library

Google Scholar

[59]

W. Tao and L. Wei-hua. Naive bayes software defect prediction model. In CiSE'10, pages 1--4.

Google Scholar

[60]

Z. Tu, Z. Su, and P. Devanbu. On the localness of software. In FSE'14, pages 269--280.

Digital Library

Google Scholar

[61]

B. Turhan, T. Menzies, A. B. Bener, and J. Di Stefano. On the relative value of cross-company and within-company data for defect prediction. Empirical Softw. Engg., 14(5):540--578, 2009.

Digital Library

Google Scholar

[62]

J. Wang, B. Shen, and Y. Chen. Compressed c4. 5 models for software defect prediction. In QSIC'12, pages 13--16.

Digital Library

Google Scholar

[63]

S. Watanabe, H. Kaiya, and K. Kaijiri. Adapting a fault prediction model to allow inter languagereuse. In Proceedings of the 4th International Workshop on Predictor Models in Software Engineering, pages 19--24, 2008.

Digital Library

Google Scholar

[64]

E. J. Weyuker, T. J. Ostrand, and R. M. Bell. Using developer information as a factor for fault prediction. In PROMISE'07, pages 8--8.

Digital Library

Google Scholar

[65]

M. White, C. Vendome, M. Linares-Vásquez, and D. Poshyvanyk. Toward deep learning software repositories. In MSR'15, pages 334--345.

Digital Library

Google Scholar

[66]

I. H. Witten and E. Frank. Data Mining: Practical machine learning tools and techniques. Morgan Kaufmann, 2005.

Digital Library

Google Scholar

[67]

X. Xie, W. Zhang, Y. Yang, and Q. Wang. Dretom: Developer recommendation based on topic models for bug resolution. In PROMISE'12, pages 19--28.

Digital Library

Google Scholar

[68]

X. Yang, D. Lo, X. xia, Y. Zhang, and J. Sun. Deep learning for just-in-time defect prediction. In QRS'15, pages 17--26.

Digital Library

Google Scholar

[69]

Z. Yuan, Y. Lu, Z. Wang, and Y. Xue. Droid-sec: Deep learning in android malware detection. In SIGCOMM'14, pages 371--372.

Digital Library

Google Scholar

[70]

T. Zimmermann, R. Premraj, and A. Zeller. Predicting defects for eclipse. In PROMISE'07, pages 9--9.

Digital Library

Google Scholar

Cited By

View all

Nashaat MMiller J(2025)Refining software defect prediction through attentive neural models for code understandingJournal of Systems and Software10.1016/j.jss.2024.112266220(112266)Online publication date: Feb-2025
https://doi.org/10.1016/j.jss.2024.112266
Groppe JGroppe SSenf DMöller R(2024)There Are Infinite Ways to Formulate Code: How to Mitigate the Resulting Problems for Better Software Vulnerability DetectionInformation10.3390/info1504021615:4(216)Online publication date: 11-Apr-2024
https://doi.org/10.3390/info15040216
Lei TXue JMan DWang YLi MKong Z(2024)SDP-MTF: A Composite Transfer Learning and Feature Fusion for Cross-Project Software Defect PredictionElectronics10.3390/electronics1313243913:13(2439)Online publication date: 21-Jun-2024
https://doi.org/10.3390/electronics13132439
Show More Cited By

Index Terms

Automatically learning semantic features for defect prediction
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Software visualization and deep transfer learning for effective software defect prediction
ICSE '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

Software defect prediction aims to automatically locate defective code modules to better focus testing resources and human effort. Typically, software defect prediction pipelines are comprised of two parts: the first extracts program features, like ...
Heterogeneous defect prediction
ESEC/FSE 2015: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering

Software defect prediction is one of the most active research areas in software engineering. We can build a prediction model with defect data collected from a software project and predict defects in the same project, i.e. within-project defect ...
Cross-project defect prediction: a large scale experiment on data vs. domain vs. process
ESEC/FSE '09: Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering

Prediction of software defects works well within projects as long as there is a sufficient amount of data available to train any models. However, this is rarely the case for new software projects and for many companies. So far, only a few have studies ...

Reviews

Reviewer: Bernard Kuc

Have data Try teaching a machine. Any problem space that has a large amount of data is just asking for someone to apply machine learning to find a solution. Defect detection is one such domain. With an extensive number of open-source projects, coupled with publicly available revision histories, it is unsurprising that there is such a large proliferation of machine learning papers on the topic. The authors take Java source code and create a list of tokens for each file. These lists of tokens are then assigned to discrete numerical values that feed into a deep belief network. The network is then trained to identify semantic details from the token list that form the basis of deciding whether a file will contain defects. What I found very surprising is that the authors did not encode scope delimiters into the token list. Having trained their network of nodes, the authors compare the predictive power of their approach against classification algorithms that base their decisions on traditional features of the code, such as lines of code, operand and operator count, number of methods, position of class in inheritance hierarchy, McCabe complexity, and others. For both, within project defect detection and (with some changes to their algorithm) for cross-project defect detection, the authors show that their approach improves on existing algorithms' defect detection capabilities. The paper unfortunately fails to provide any theoretical justification for why any particular token list is more likely to contain a defect than another slightly different list of tokens.

Access critical reviews of Computing literature here

Become a reviewer for Computing Reviews.

Comments

Information & Contributors

Information

Published In

ICSE '16: Proceedings of the 38th International Conference on Software Engineering

May 2016

1235 pages

ISBN:9781450339001

DOI:10.1145/2884781

General Chair:
Laura Dillon
Michigan State University
,
Program Chairs:
Willem Visser
Stellenbosch University, South Africa
,
Laurie Williams
North Carolina State University

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 May 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

ICSE '16

Sponsor:

ACM
SIGSOFT
IEEE-CS\TCSE
IEEE-CS\DATC

ICSE '16: 38th International Conference on Software Engineering

May 14 - 22, 2016

Texas, Austin

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

454
Total Citations
View Citations
2,758
Total Downloads

Downloads (Last 12 months)298
Downloads (Last 6 weeks)43

Reflects downloads up to 23 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Nashaat MMiller J(2025)Refining software defect prediction through attentive neural models for code understandingJournal of Systems and Software10.1016/j.jss.2024.112266220(112266)Online publication date: Feb-2025
https://doi.org/10.1016/j.jss.2024.112266
Groppe JGroppe SSenf DMöller R(2024)There Are Infinite Ways to Formulate Code: How to Mitigate the Resulting Problems for Better Software Vulnerability DetectionInformation10.3390/info1504021615:4(216)Online publication date: 11-Apr-2024
https://doi.org/10.3390/info15040216
Lei TXue JMan DWang YLi MKong Z(2024)SDP-MTF: A Composite Transfer Learning and Feature Fusion for Cross-Project Software Defect PredictionElectronics10.3390/electronics1313243913:13(2439)Online publication date: 21-Jun-2024
https://doi.org/10.3390/electronics13132439
Bagheri AHegedűs P(2024)Towards a Block-Level ML-Based Python Vulnerability Detection ToolActa Cybernetica10.14232/actacyb.29966726:3(323-371)Online publication date: 22-Jul-2024
https://doi.org/10.14232/actacyb.299667
Gao ZSu YHu XXia X(2024)Automating TODO-missed Methods Detection and PatchingACM Transactions on Software Engineering and Methodology10.1145/370079334:1(1-28)Online publication date: 6-Nov-2024
https://dl.acm.org/doi/10.1145/3700793
Shiri Harzevili NBoaye Belle AWang JWang SJiang ZNagappan N(2024)A Systematic Literature Review on Automated Software Vulnerability Detection Using Machine LearningACM Computing Surveys10.1145/369971157:3(1-36)Online publication date: 11-Nov-2024
https://dl.acm.org/doi/10.1145/3699711
Jiang MJiang JWu TMa ZLuo XZhou Y(2024)Understanding Vulnerability Inducing Commits of the Linux KernelACM Transactions on Software Engineering and Methodology10.1145/367245233:7(1-28)Online publication date: 14-Jun-2024
https://dl.acm.org/doi/10.1145/3672452
Wan YBi ZHe YZhang JZhang HSui YXu GJin HYu P(2024)Deep Learning for Code Intelligence: Survey, Benchmark and ToolkitACM Computing Surveys10.1145/366459756:12(1-41)Online publication date: 18-May-2024
https://dl.acm.org/doi/10.1145/3664597
Xin QWu HTang JLiu XReiss SXuan J(2024)Detecting, Creating, Repairing, and Understanding Indivisible Multi-Hunk BugsProceedings of the ACM on Software Engineering10.1145/36608281:FSE(2747-2770)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660828
Wang XYu HMeng XCao HZhang HSun HLiu XHu C(2024)MTL-TRANSFER: Leveraging Multi-task Learning and Transferred Knowledge for Improving Fault Localization and Program RepairACM Transactions on Software Engineering and Methodology10.1145/365444133:6(1-31)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3654441
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Software visualization and deep transfer learning for effective software defect prediction

Heterogeneous defect prediction

Cross-project defect prediction: a large scale experiment on data vs. domain vs. process

Reviews

Access critical reviews of Computing literature here