research-article

Understanding static code warnings: : An incremental AI approach

Authors:

Tim MenziesAuthors Info & Claims

Volume 167, Issue C

https://doi.org/10.1016/j.eswa.2020.114134

Published: 01 April 2021 Publication History

Highlights

•

Incremental AI tool for Static Warning Identification.

•

A case study to help developers Ignoring falsely reported static warnings.

•

Self-adaptive active learning process.

•

Reducing human inspection cost in software mining on early life cycle.

Abstract

Knowledge-based systems reason over some knowledge base. Hence, an important issue for such systems is how to acquire the knowledge needed for their inference. This paper assesses active learning methods for acquiring knowledge for “static code warnings”.

Static code analysis is a widely-used method for detecting bugs and security vulnerabilities in software systems. As software becomes more complex, analysis tools also report lists of increasingly complex warnings that developers need to address on a daily basis. Such static code analysis tools are usually over-cautious; i.e. they often offer many warnings about spurious issues. Previous research work shows that about 35% to 91 % warnings reported as bugs by SA tools are actually unactionable (i.e., warnings that would not be acted on by developers because they are falsely suggested as bugs).

Experienced developers know which errors are important and which can be safely ignored. How can we capture that experience? This paper reports on an incremental AI tool that watches humans reading false alarm reports. Using an incremental support vector machine mechanism, this AI tool can quickly learn to distinguish spurious false alarms from more serious matters that deserve further attention.

In this work, nine open-source projects are employed to evaluate our proposed model on the features extracted by previous researchers and identify the actionable warnings in a priority order given by our algorithm. We observe that our model can identify over 90% of actionable warnings when our methods tell humans to ignore 70 to 80% of the warnings.

References

[1]

S. Allier, N. Anquetil, A. Hora, S. Ducasse, A framework to compare alert ranking algorithms, in: 2012 19th Working Conference on Reverse Engineering, IEEE, 2012, pp. 277–285.

[2]

Arnold, J., Abbott, T., Daher, W., Price, G., Elhage, N., Thomas, G., & Kaseorg, A. (2009). Security impact ratings considered harmful. arXiv preprint arXiv:0904.4058.

[3]

P. Avgustinov, A.I. Baars, A.S. Henriksen, G. Lavender, G. Menzel, O. de Moor, M. Schäfer, J. Tibble, Tracking static analysis violations over time to capture developer characteristics, in: Proceedings of the 37th International Conference on Software Engineering-Volume 1, IEEE Press, 2015, pp. 437–447.

[4]

N. Ayewah, W. Pugh, D. Hovemeyer, J.D. Morgenthaler, J. Penix, Using static analysis to find bugs, IEEE Software 25 (2008) 22–29.

[5]

P. Bhattacharya, M. Iliofotou, I. Neamtiu, M. Faloutsos, Graph-based analysis and prediction for software evolution, in: 2012 34th International Conference on Software Engineering (ICSE), IEEE, 2012, pp. 419–429.

[6]

C. Boogerd, L. Moonen, Assessing the value of coding standards: An empirical study, in: 2008 IEEE International Conference on Software Maintenance, IEEE, 2008, pp. 277–286.

[7]

J.F. Bowring, J.M. Rehg, M.J. Harrold, Active learning for automatic classification of software behavior, ACM SIGSOFT Software Engineering Notes, ACM, 2004, pp. 195–205.

[8]

Cormack, G. V. & Grossman, M. R. (2015). Autonomy and reliability of continuous active learning for technology-assisted review. arXiv preprint arXiv:1504.06868.

[9]

C. Cortes, V. Vapnik, Support-vector networks, Machine Learning 20 (1995) 273–297.

[10]

S. Ertekin, J. Huang, C.L. Giles, Active learning for class imbalance problem, SIGIR, 2007, pp. 823–824.

[11]

Fahid, F. M., Yu, Z., & Menzies, T. (2019). Better technical debt detection via surveying. arXiv preprint arXiv:1905.08297.

[12]

Feigenbaum, E. A. (1980). Knowledge engineering: the applied side of artificial intelligence. Technical Report. Stanford Univ CA Dept of Computer Science.

[13]

Q. Hanam, L. Tan, R. Holmes, P. Lam, Finding patterns in static analysis alerts: improving actionable alert ranking, in, in: Proceedings of the 11th Working Conference on Mining Software Repositories, ACM, 2014, pp. 152–161.

[14]

Heckman, S. & Williams, L. (2008). On establishing a benchmark for evaluating static analysis alert prioritization and classification techniques. In: Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement, ACM. pp. 41–50.

[15]

S. Heckman, L. Williams, A model building process for identifying actionable static analysis alerts, in: 2009 International Conference on Software Testing Verification and Validation, IEEE, 2009, pp. 161–170.

[16]

S. Heckman, L. Williams, A systematic literature review of actionable alert identification techniques for automated static code analysis, Information and Software Technology 53 (2011) 363–387.

[17]

R. Hoekstra, The knowledge reengineering bottleneck, Semantic Web 1 (2010) 111–115.

[18]

D. Hovemeyer, W. Pugh, Finding bugs is easy, Acm sigplan Notices 39 (2004) 92–106.

[19]

B. Johnson, Y. Song, E. Murphy-Hill, R. Bowdidge, Why don’t software developers use static analysis tools to find bugs?, in: Proceedings of the 2013 International Conference on Software Engineering, IEEE Press, 2013, pp. 672–681.

[20]

Kim, S. & Ernst, M.D. (2007a). Prioritizing warning categories by analyzing software history, in: Proceedings of the Fourth International Workshop on Mining Software Repositories, IEEE Computer Society. p. 27.

[21]

S. Kim, M.D. Ernst, Which warnings should i fix first?, in, in: Proceedings of the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, ACM, 2007, pp. 45–54.

[22]

J. Krall, T. Menzies, M. Davies, Gale: Geometric active learning for search-based software engineering, IEEE Transactions on Software Engineering 41 (2015) 1001–1018.

[23]

T. Kremenek, K. Ashcraft, J. Yang, D. Engler, Correlation exploitation in error ranking, ACM SIGSOFT Software Engineering Notes, ACM, 2004, pp. 83–93.

[24]

R. Krishna, Z. Yu, A. Agrawal, M. Dominguez, D. Wolf, The’bigse’project: Lessons learned from validating industrial text mining, in: 2016 IEEE/ACM 2nd International Workshop on Big Data Software Engineering (BIGDSE), IEEE, 2016, pp. 65–71.

[25]

T.C. Landgrebe, R.P. Duin, Efficient multiclass roc approximation by decomposition via confusion matrix perturbation analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence 30 (2008) 810–822.

[26]

G. Liang, L. Wu, Q. Wu, Q. Wang, T. Xie, H. Mei, Automatic construction of an effective training set for prioritizing static analysis warnings, in, in: Proceedings of the IEEE/ACM international conference on Automated software engineering, ACM, 2010, pp. 93–102.

[27]

A. Liaw, M. Wiener, et al., Classification and regression by randomforest, R News 2 (2002) 18–22.

[28]

M. Miwa, J. Thomas, A. O’Mara-Eves, S. Ananiadou, Reducing systematic review workload through certainty-based screening, Journal of Biomedical Informatics 51 (2014) 242–253.

[29]

S.S. Murtaza, W. Khreich, A. Hamou-Lhadj, A.B. Bener, Mining trends and patterns of software vulnerabilities, Journal of Systems and Software 117 (2016) 218–228.

[30]

P.K. Murukannaiah, M.P. Singh, Platys: An active learning framework for place-aware application development and its evaluation, ACM Transactions on Software Engineering and Methodology (TOSEM) 24 (2015) 19.

[31]

Panichella, S., Arnaoudova, V., Di Penta, M., & Antoniol, G. (2015). Would static analysis tools help developers with code reviews? In: 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), IEEE. pp. 161–170.

[32]

E. Pasolli, F. Melgani, D. Tuia, F. Pacifici, W.J. Emery, Svm active learning approach for image classification using spatial information, IEEE Transactions on Geoscience and Remote Sensing 52 (2013) 2217–2233.

[33]

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, et al., Scikit-learn: Machine learning in python, Journal of Machine Learning Research 12 (2011) 2825–2830.

Digital Library

[34]

F. Rahman, S. Khatri, E.T. Barr, P. Devanbu, Comparing static bug finders and statistical prediction, in, in: Proceedings of the 36th International Conference on Software Engineering, ACM, 2014, pp. 424–434.

[35]

S.R. Safavian, D. Landgrebe, A survey of decision tree classifier methodology, IEEE Transactions on Systems, Man, and Cybernetics 21 (1991) 660–674.

[36]

B. Settles, Active learning literature survey. Technical Report, University of Wisconsin-Madison Department of Computer Sciences, 2009.

[37]

H. Shen, J. Fang, J. Zhao, Efindbugs: Effective error ranking for findbugs, in: 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation, IEEE, 2011, pp. 299–308.

[38]

S. Shivaji, E.J. Whitehead Jr, R. Akella, S. Kim, Reducing features to improve bug prediction, in: 2009 IEEE/ACM International Conference on Automated Software Engineering, IEEE, 2009, pp. 600–604.

[39]

F. Thung, D. Lo, L. Jiang, F. Rahman, P.T. Devanbu, et al., To what extent could we detect field defects? an extended empirical study of false negatives in static bug-finding tools, Automated Software Engineering 22 (2015) 561–602.

[40]

S. Tong, D. Koller, Support vector machine active learning with applications to text classification, Journal of Machine Learning Research 2 (2001) 45–66.

Digital Library

[41]

B.C. Wallace, C.H. Schmid, J. Lau, T.A. Trikalinos, Meta-analyst: software for meta-analysis of binary, continuous and diagnostic data, BMC Medical Research Methodology 9 (2009) 80.

[42]

B.C. Wallace, T.A. Trikalinos, J. Lau, C. Brodley, C.H. Schmid, Semi-automated screening of biomedical citations for systematic reviews, BMC Bioinformatics 11 (2010) 55.

[43]

J. Wang, S. Wang, Q. Cui, Q. Wang, Local-based active classification of test report to assist crowdsourced testing, in: 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE), IEEE, 2016, pp. 190–201.

[44]

J. Wang, S. Wang, Q. Wang, Is there a golden feature set for static warning identification?: an experimental evaluation, in, in: Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ACM, 2018, p. 17.

[45]

S. Wang, T. Liu, L. Tan, Automatically learning semantic features for defect prediction, in: 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE), IEEE, 2016, pp. 297–308.

[46]

D. Wijayasekara, M. Manic, J.L. Wright, M. McQueen, Mining bug databases for unidentified software vulnerabilities, in: 2012 5th International Conference on Human System Interactions, IEEE, 2012, pp. 89–96.

[47]

I.H. Witten, E. Frank, M.A. Hall, C.J. Pal, Data Mining: Practical machine learning tools and techniques, Morgan Kaufmann, 2016.

[48]

D.H. Wolpert, W.G. Macready, et al., No free lunch theorems for optimization, IEEE Transactions on Evolutionary Computation 1 (1997) 67–82.

Digital Library

[49]

M. Yan, X. Zhang, L. Xu, H. Hu, S. Sun, X. Xia, Revisiting the correlation between alerts and software defects: A case study on myfaces, camel, and cxf, in: 2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC), IEEE, 2017, pp. 103–108.

[50]

Yu, Z., Carver, J. C., Rothermel, G., & Menzies, T. (2019a). Searching for better test case prioritization schemes: a case study of ai-assisted systematic literature review. arXiv preprint arXiv:1909.07249.

[51]

Z. Yu, F. Fahid, T. Menzies, G. Rothermel, K. Patrick, S. Cherian, Terminator: Better automated ui test case prioritization, in: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ACM, New York, NY, USA, 2019, pp. 883–894,.

Digital Library

[52]

Z. Yu, N.A. Kraft, T. Menzies, Finding better active learners for faster literature reviews, Empirical Software Engineering 23 (2018) 3161–3186.

[53]

Z. Yu, T. Menzies, Total recall, language processing, and software engineering, in, in: Proceedings of the 4th ACM SIGSOFT International Workshop on NLP for Software Engineering, ACM, 2018, pp. 10–13.

[54]

Z. Yu, T. Menzies, Fast2: An intelligent assistant for finding relevant papers, Expert Systems with Applications 120 (2019) 57–71.

[55]

Z. Yu, C. Theisen, L. Williams, T. Menzies, Improving vulnerability inspection efficiency using active learning, IEEE Transactions on Software Engineering 1–1 (2019),.

[56]

C. Zhang, Y. Ma, Ensemble machine learning: methods and applications, Springer, 2012.

Cited By

Yang XJakubowski MKang LYu HMenzies T(2025)SparseCoder: Advancing source code analysis with sparse attention and learned token pruningEmpirical Software Engineering10.1007/s10664-024-10558-130:1Online publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1007/s10664-024-10558-1
Ge XFang CLi XSun WWu DZhai JLin SZhao ZLiu YChen Z(2024)Machine Learning for Actionable Warning Identification: A Comprehensive SurveyACM Computing Surveys10.1145/369635257:2(1-35)Online publication date: 19-Sep-2024
https://dl.acm.org/doi/10.1145/3696352
Liu ZYan MGao ZLi DZhang XYang DSpinellis DConstantinou EBacchelli A(2024)AW4C: A Commit-Aware C Dataset for Actionable Warning IdentificationProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644885(133-137)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644885
Show More Cited By

Index Terms

Understanding static code warnings: An incremental AI approach

Index terms have been assigned to the content through auto-classification.

Recommendations

Is there a "golden" feature set for static warning identification?: an experimental evaluation
ESEM '18: Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

Background: The most important challenge regarding the use of static analysis tools (e.g., FindBugs) is that there are a large number of warnings that are not acted on by developers. Many features have been proposed to build classification models for ...
A better approach to track the evolution of static code warnings
ICSE '21: Proceedings of the 43rd International Conference on Software Engineering: Companion Proceedings

Static bug detection tools help developers detect code problems. However, it is known that they remain underutilized due to various reasons. Recent advances to incorporate static bug detectors in modern software development workflows can better motivate ...
Evaluating static analysis defect warnings on production software
PASTE '07: Proceedings of the 7th ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering

Static analysis tools for software defect detection are becoming widely used in practice. However, there is little public information regarding the experimental evaluation of the accuracy and value of the warnings these tools report. In this paper, we ...

Comments

Information & Contributors

Information

Published In

cover image Expert Systems with Applications: An International Journal

Expert Systems with Applications: An International Journal Volume 167, Issue C

Apr 2021

1224 pages

ISSN:0957-4174

Issue’s Table of Contents

Elsevier Ltd.

Publisher

Pergamon Press, Inc.

United States

Publication History

Published: 01 April 2021

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 27 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yang XJakubowski MKang LYu HMenzies T(2025)SparseCoder: Advancing source code analysis with sparse attention and learned token pruningEmpirical Software Engineering10.1007/s10664-024-10558-130:1Online publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1007/s10664-024-10558-1
Ge XFang CLi XSun WWu DZhai JLin SZhao ZLiu YChen Z(2024)Machine Learning for Actionable Warning Identification: A Comprehensive SurveyACM Computing Surveys10.1145/369635257:2(1-35)Online publication date: 19-Sep-2024
https://dl.acm.org/doi/10.1145/3696352
Liu ZYan MGao ZLi DZhang XYang DSpinellis DConstantinou EBacchelli A(2024)AW4C: A Commit-Aware C Dataset for Actionable Warning IdentificationProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644885(133-137)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644885
Guo ZTan TLiu SLiu XLai WYang YLi YChen LDong WZhou Y(2023)Mitigating False Positive Static Analysis Warnings: Progress, Challenges, and OpportunitiesIEEE Transactions on Software Engineering10.1109/TSE.2023.332966749:12(5154-5188)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1109/TSE.2023.3329667
Yedida RKang HTu HYang XLo DMenzies T(2023)How to Find Actionable Static Analysis Warnings: A Case Study With FindBugsIEEE Transactions on Software Engineering10.1109/TSE.2023.323420649:4(2856-2872)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1109/TSE.2023.3234206
Yu PWu YPeng XPeng JZhang JXie PZhao WGrundy JPollock LPenta M(2023)ViolationTracker: Building Precise Histories for Static Analysis ViolationsProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00171(2022-2034)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00171
Motwani MBrun YLiu AMuccini H(2023)Understanding Why and Predicting When Developers Adhere to Code-Quality StandardsProceedings of the 45th International Conference on Software Engineering: Software Engineering in Practice10.1109/ICSE-SEIP58684.2023.00045(432-444)Online publication date: 17-May-2023
https://dl.acm.org/doi/10.1109/ICSE-SEIP58684.2023.00045
Choi YNam J(2023)WINEInformation and Software Technology10.1016/j.infsof.2022.107109155:COnline publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1016/j.infsof.2022.107109
Guo YHu QCordy MPapadakis MLe Traon Y(2023)DRE: density-based data selection with entropy for adversarial-robust deep learning modelsNeural Computing and Applications10.1007/s00521-022-07812-235:5(4009-4026)Online publication date: 1-Feb-2023
https://dl.acm.org/doi/10.1007/s00521-022-07812-2
Kang HAw KLo DDwyer MDamian DZeller A(2022)Detecting false alarms from automatic static analysis toolsProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510214(698-709)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510214

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents