research-article

Test case selection and prioritization using machine learning: a systematic literature review

Authors:

Mojtaba Bagherzadeh,

Taher A. Ghaleb,

Lionel BriandAuthors Info & Claims

Empirical Software Engineering, Volume 27, Issue 2

https://doi.org/10.1007/s10664-021-10066-6

Published: 01 March 2022 Publication History

Abstract

Regression testing is an essential activity to assure that software code changes do not adversely affect existing functionalities. With the wide adoption of Continuous Integration (CI) in software projects, which increases the frequency of running software builds, running all tests can be time-consuming and resource-intensive. To alleviate that problem, Test case Selection and Prioritization (TSP) techniques have been proposed to improve regression testing by selecting and prioritizing test cases in order to provide early feedback to developers. In recent years, researchers have relied on Machine Learning (ML) techniques to achieve effective TSP (ML-based TSP). Such techniques help combine information about test cases, from partial and imperfect sources, into accurate prediction models. This work conducts a systematic literature review focused on ML-based TSP techniques, aiming to perform an in-depth analysis of the state of the art, thus gaining insights regarding future avenues of research. To that end, we analyze 29 primary studies published from 2006 to 2020, which have been identified through a systematic and documented process. This paper addresses five research questions addressing variations in ML-based TSP techniques and feature sets for training and testing ML models, alternative metrics used for evaluating the techniques, the performance of techniques, and the reproducibility of the published studies. We summarize the results related to our research questions in a high-level summary that can be used as a taxonomy for classifying future TSP studies.

References

[1]

Almaghairbe R and Roper M Separating passing and failing test executions by clustering anomalies Softw Qual J 2017 25 3 803-840

[2]

Alon U, Zilberstein M, Levy O, and Yahav E code2vec: Learning distributed representations of code Proc ACM Programm Lang 2019 3 POPL 1-29

[3]

Aman H, Amasaki S, Yokogawa T, Kawahara M (2020) A comparative study of vectorization-based static test case prioritization methods. In: 2020 46th Euromicro conference on software engineering and advanced applications (SEAA). IEEE, pp 80–88

[4]

Beller M, Gousios G, Zaidman A (2017) Travistorrent: Synthesizing travis ci and github for full-stack research on continuous integration. In: 2017 IEEE/ACM 14th international conference on mining software repositories (MSR). IEEE, pp 447–450

[5]

Bertolino A, Guerriero A, Miranda B, Pietrantuono R, Russo S (2020) Learning-to-rank vs ranking-to-learn: Strategies for regression testing in continuous integration. In: In 42nd international conference on software engineering (ICSE)

[6]

Busjaeger B, Xie T (2016) Learning for test prioritization: an industrial case study. In: Proceedings of the 2016 24th ACM SIGSOFT International symposium on foundations of software engineering. pp 975–980

[7]

Carlson R, Do H, Denton A (2011) A clustering approach to improving test case prioritization: An industrial case study. In: ICSM, vol 11. pp 382–391

[8]

Chen S, Chen Z, Zhao Z, Baowen X u, Feng Y (2011) Using semi-supervised clustering to improve regression test selection techniques. In: 2011 Fourth IEEE international conference on software testing, verification and validation. IEEE, pp 1–10

[9]

Chen J, Lou Y, Zhang L, Zhou J, Wang X, Hao D, Zhang L u (2018) Optimizing test prioritization via test distribution analysis. In: Proceedings of the 2018 26th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering. pp 656–667

[10]

Cliff N Dominance statistics: Ordinal analyses to answer ordinal questions Psychol Bull 1993 114 3 494

[11]

Clover A (2021) Atlassian Clover. https://bitbucket.org/atlassian/clover. Retrieved March 14, 2021

[12]

Dang V, Zarozinski M (2020) Ranklib. https://sourceforge.net/p/lemur/wiki/RankLib/

[13]

Do H and Rothermel G On the use of mutation faults in empirical assessments of test case prioritization techniques IEEE Trans Softw Eng 2006 32 9 733-752

[14]

Devlin J, Chang M-W, Lee K, Toutanova K (2018) BERT Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805

[15]

Durelli VHS, Durelli RS, Borges SS, Endo AT, Eler MM, Dias DRC, and Guimaraes MP Machine learning applied to software testing: A systematic mapping study IEEE Trans Reliab 2019 68 3 1189-1212

[16]

EclEmma team (2021) JaCoCo: Java code coverage library. https://github.com/jacoco/jacoco. Retrieved March 14, 2021

[17]

Elbaum S, Malishevsky A, Rothermel G (2001) Incorporating varying test costs and fault severities into test case prioritization. In: Proceedings of the 23rd international conference on software engineering, ICSE 2001. IEEE, pp 329–338

[18]

Elbaum S, Malishevsky A, and Rothermel G Test case prioritization: A family of empirical studies IEEE Trans Softw Eng 2002 28 2 159-182

[19]

Feng Z, Guo D, Tang D, Duan N, Feng X, Gong M, Shou L, Qin B, Liu T, Jiang D, et al. (2020) CodeBERT: A pre-trained model for programming and natural languages. arXiv:2002.08155

[20]

Fowler M, Foemmel M (2006) Continuous integration. http://www.dccia.ua.es/dccia/inf/asignaturas/MADS/2013-14/lecturas/10_Fowler_Continuous_Integration.pdf

[21]

Freund Y, Iyer R, Schapire RE, and Singer Y An efficient boosting algorithm for combining preferences J Mach Learn Res 2003 4 Nov 933-969

[22]

Hasnain M, Pasha MF, Lim CH, Ghan I (2019) Recurrent neural network for web services performance forecasting, ranking and regression testing. In: 2019 Asia-pacific signal and information processing association annual summit and conference (APSIPA ASC). IEEE, pp 96–105

[23]

Hochreiter S and Schmidhuber J Long short-term memory Neural Comput 1997 9 8 1735-1780

[24]

Jahan H, Feng Z, Mahmud SM, and Dong P Version specific test case prioritization approach based on artificial neural network J Intell Fuzzy Syst 2019 36 6 6181-6194

[25]

James G, Witten D, Hastie T, and Tibshirani R An introduction to statistical learning, vol 112 2013 Berlin Springer

[26]

Joachims T (2002) Optimizing search engines using clickthrough data. In: Proceedings of the eighth ACM SIGKDD international conference on knowledge discovery and data mining. pp 133–142

[27]

Just R, Jalali D, Ernst MD (2014) Defects4j: A database of existing faults to enable controlled testing studies for java programs. In: Proceedings of the 2014 International Symposium on Software Testing and Analysis. pp 437–440

[28]

Kandil P, Moussa S, and Badr N Cluster-based test cases prioritization and selection technique for agile regression testing J Softw Evol Process 2017 29 6 e1794

[29]

Kazmi R, Jawawi DNA, Mohamad R, and Ghani I Effective regression test case selection: A systematic literature review ACM Comput Surv (CSUR) 2017 50 2 1-32

[30]

Khalid Z, Qamar U (2019) Weight and cluster based test case prioritization technique. In: 2019 IEEE 10th annual information technology, electronics and mobile communication conference (IEMCON). IEEE, pp 1013–1022

[31]

Khatibsyarbini M, Isa MA, Jawawi DNA, and Tumeng R Test case prioritization approaches in regression testing: A systematic literature review Inf Softw Technol 2018 93 74-93

[32]

Kim J-M, Porter A (2002) A history-based test prioritization technique for regression testing in resource constrained environments. In: Proceedings of the 24th international conference on software engineering. pp 119–129

[33]

Kitchenham B Procedures for performing systematic reviews Keele, UK, Keele University 2004 33 2004 1-26

[34]

Kitchenham B, Brereton OP, Budgen D, Turner M, Bailey J, and Linkman S Systematic literature reviews in software engineering–a systematic literature review Inf Softw Technol 2009 51 1 7-15

[35]

Lachmann R, Schulze S, Nieke M, Seidl C, Schaefer I (2016) System-level test case prioritization using machine learning. In: 2016 15th IEEE international conference on machine learning and applications (ICMLA). IEEE, pp 361–368

[36]

Li H Learning to rank for information retrieval and natural language processing Synth Lect Hum Lang Technol 2011 4 1 1-113

[37]

Liem C, Panichella A (2020) Run, forest, run? on randomization and reproducibility in predictive software engineering. arXiv:2012.08387,2020,

[38]

Lima JAP and Vergilio SR Test case prioritization in continuous integration environments A systematic mapping study Inf Softw Technol 2020 121 106268

[39]

Lima JAP, Vergilio SR (2020) Multi-armed bandit test case prioritization in continuous integration environments: A trade-off analysis. In: Proceedings of the 5th Brazilian symposium on systematic and automated software testing. pp 21–30

[40]

Lima JAP, Mendonċa WDF, Vergilio SR, Assunċão WKG (2020) Learning-based prioritization of test cases in continuous integration of highly-configurable software. In: Proceedings of the 24th ACM conference on systems and software product line: Volume A-Volume A. pp 1–11

[41]

Mahdieh M, Mirian-Hosseinabadi Seyed-Hassan, Etemadi K, Nosrati A, and Jalali S Incorporating fault-proneness estimations into coverage-based test case prioritization methods Inf Softw Technol 2020 121 106269

[42]

Manning C and Schutze H Foundations of statistical natural language processing 1999 Cambridge MIT press

[43]

Mattis T, Rein P, Dürsch F, Hirschfeld R (2020) Rtptorrent: An open-source dataset for evaluating regression test prioritization. In: Proceedings of the 17th international conference on mining software repositories. pp 385–396

[44]

McGuire N, Kernel L (2006) GCOV - tool analysis. https://linuxdevices.org/ldfiles/article062/der_herr_gcov.pdf. Retrieved March 14, 2021

[45]

Medhat N, Moussa S, Badr N, and Tolba MF A framework for continuous regression and integration testing in iot systems based on deep learning and search-based techniques IEEE Access 2020 8 215716-215726

[46]

Mirarab S, Tahvildari L (2008) An empirical study on bayesian network-based approach for test case prioritization. In: 2008 1st international conference on software testing, verification, and validation. IEEE, pp 278–287

[47]

Noor TB, Hemmati H (2017) Studying test case failure prediction for test case prioritization. In: Proceedings of the 13th international conference on predictive models and data analytics in software engineering. pp 2–11

[48]

Nuñez-Varela AS, Pérez-Gonzalez HG, Martínez-Perez FE, and Soubervielle-Montalvo C Source code metrics: A systematic mapping study J Syst Softw 2017 128 164-197

[49]

Palma F, Abdou T, Bener A, Maidens J, Liu S (2018) An improvement to test case failure prediction in the context of test case prioritization. In: Proceedings of the 14th international conference on predictive models and data analytics in software engineering. pp 80–89

[50]

do Prado Lima JA, Vergilio SR (2020) A multi-armed bandit approach for test case prioritization in continuous integration environments. IEEE Trans Softw Eng

[51]

Qi L, Moran K, Poshyvanyk D, Penta MD (2018) Assessing test case prioritization on real faults and mutants. In: 2018 IEEE international conference on software maintenance and evolution (ICSME). IEEE, pp 240–251

[52]

Replication package (2021) https://github.com/uOttawa-Nanda-Lab/ML-based-TSP-SLR

[53]

Robbins H Some aspects of the sequential design of experiments Bull Am Math Soc 1952 58 5 527-535

[54]

Rosenbauer L, Stein A, Maier R, Pätzel D, Hähner J (2020) Xcs as a reinforcement learning approach to automatic test case prioritization. In: Proceedings of the 2020 genetic and evolutionary computation conference companion. pp 1798–1806

[55]

Rothermel G, Untch RH, Chu C, and Harrold MJ Prioritizing test cases for regression testing IEEE Trans Softw Eng 2001 27 10 929-948

[56]

Shaheamlung G, Rote K, et al. (2020) A comprehensive review for test case prioritization in software engineering. In: 2020 international conference on intelligent engineering and management (ICIEM). IEEE, pp 331–336

[57]

Sharma MM and Agrawal A Test case design and test case prioritization using machine learning Int J Eng Adv Technol 2019 9 1 2742-2748

[58]

Shi T, Xiao L, Wu K (2020) Reinforcement learning based test case prioritization for enhancing the security of software. In: 2020 IEEE 7th international conference on data science and advanced analytics (DSAA). IEEE, pp 663–672

[59]

Singh A, Bhatia RK, and Singhrova A Machine learning based test case prioritization in object oriented testing Int J Recent Technol Eng 2019 8 3 700-707

[60]

Spieker H, Gotlieb A, Marijan D, Mossige M (2017) Reinforcement learning for automatic test case prioritization and selection in continuous integration. In: Proceedings of the 26th ACM SIGSOFT international symposium on software testing and analysis. pp 12–22

[61]

Stone M Cross-validatory choice and assessment of statistical predictions J R Stat Soc Ser B (Methodol) 1974 36 2 111-133

[62]

Sutton RS and Barto AG Reinforcement learning: An introduction 2018 Cambridge MIT press

[63]

Thomas SW, Hemmati H, Hassan AE, and Blostein D Static test case prioritization using topic models Empir Softw Eng 2014 19 1 182-212

[64]

Tonella P, Avesani P, Susi A (2006) Using the case-based ranking methodology for test case prioritization. In: 2006 22nd IEEE international conference on software maintenance. pp 123–133

[65]

Wang Y, Chen Z, Feng Y, Luo B, Yang Y (2012) Using weighted attributes to improve cluster test selection. In: 2012 IEEE sixth international conference on software security and reliability. IEEE, pp 138–146

[66]

Wilks DS Statistical methods in the atmospheric sciences, vol 100 2011 Cambridge Academic press

[67]

Wilson SW Classifier fitness based on accuracy Evol Comput 1995 3 2 149-175

[68]

Xiao Q u, Cohen MB, Woolf KM (2007) Combinatorial interaction regression testing: A study of test case generation and prioritization. In: 2007 IEEE international conference on software maintenance. IEEE, pp 255–264

[69]

Yan S, Chen Z, Zhao Z, Zhang C, Zhou Y (2010) A dynamic test cluster sampling strategy by leveraging execution spectra information. In: 2010 third international conference on software testing, verification and validation. IEEE, pp 147–154

[70]

Yoo S, Harman M, Tonella P, Susi A (2009) Clustering test cases to achieve effective and scalable prioritisation incorporating expert knowledge. In: Proceedings of the eighteenth international symposium on software testing and analysis. pp 201–212

[71]

Yue Y, Finley T, Radlinski F, Joachims T (2007) A support vector method for optimizing average precision. In: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. pp 271–278

[72]

Zhang C, Zhang Y, Shi X, Almpanidis G, Fan G, and Shen X On incremental learning for gradient boosting decision trees Neural Process Lett 2019 50 1 957-987

Cited By

Wang JWang KNie PFilkov VRay BZhou M(2024)Efficient Incremental Code Coverage Analysis for Regression Test SuitesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695551(1882-1894)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695551
Silva LMachado LEmmendorfer L(2024)A Case and Cluster-Based Framework for Reuse and Prioritization in Software TestingProceedings of the 20th Brazilian Symposium on Information Systems10.1145/3658271.3658312(1-10)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3658271.3658312
Huynh HPham NNguyen TNguyen VChristakis MPradel M(2024)Segment-Based Test Case Prioritization: A Multi-objective ApproachProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680349(1149-1160)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680349
Show More Cited By

Recommendations

Reinforcement learning for automatic test case prioritization and selection in continuous integration
ISSTA 2017: Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis

Testing in Continuous Integration (CI) involves test case prioritization, selection, and execution at each cycle. Selecting the most promising test cases to detect bugs is hard if there are uncertainties on the impact of committed code changes or, if ...
A Systematic Literature Review on Test Case Prioritization in Combinatorial Testing
ICACS '21: Proceedings of the 5th International Conference on Algorithms, Computing and Systems

In recent years, regression testing is widely used and performed in industries during the maintenance phase of software system development. The increases of software regression test cases resulted in a higher complexity of the software system, due to ...
Test Case Prioritization for Continuous Regression Testing: An Industrial Case Study
ICSM '13: Proceedings of the 2013 IEEE International Conference on Software Maintenance

Regression testing in continuous integration environment is bounded by tight time constraints. To satisfy time constraints and achieve testing goals, test cases must be efficiently ordered in execution. Prioritization techniques are commonly used to ...

Comments

Information & Contributors

Information

Published In

cover image Empirical Software Engineering

Empirical Software Engineering Volume 27, Issue 2

Mar 2022

1119 pages

ISSN:1382-3256

Issue’s Table of Contents

© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2021.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 March 2022

Accepted: 05 October 2021

Author Tags

Qualifiers

Research-article

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

26
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang JWang KNie PFilkov VRay BZhou M(2024)Efficient Incremental Code Coverage Analysis for Regression Test SuitesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695551(1882-1894)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695551
Silva LMachado LEmmendorfer L(2024)A Case and Cluster-Based Framework for Reuse and Prioritization in Software TestingProceedings of the 20th Brazilian Symposium on Information Systems10.1145/3658271.3658312(1-10)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3658271.3658312
Huynh HPham NNguyen TNguyen VChristakis MPradel M(2024)Segment-Based Test Case Prioritization: A Multi-objective ApproachProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680349(1149-1160)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680349
Aghababaeyan ZAbdellatif MDadkhah MBriand L(2024)DeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural NetworksACM Transactions on Software Engineering and Methodology10.1145/364438833:6(1-29)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3644388
Torbunova AStrandberg PPorres ISaadatmand MLonetti FBudnik CLi JGuerriero A(2024)Dynamic Test Case Prioritization in Industrial Test Result DatasetsProceedings of the 5th ACM/IEEE International Conference on Automation of Software Test (AST 2024)10.1145/3644032.3644452(154-158)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3644032.3644452
Zeng ZXiao TLamothe MHata HMcintosh SSpinellis DConstantinou EBacchelli A(2024)A Mutation-Guided Assessment of Acceleration Approaches for Continuous Integration: An Empirical Study of YourBaseProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644914(556-568)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644914
Malmir SRigby P(2024)Predicting the Lifetime of Flaky Tests on ChromeProceedings of the 1st International Workshop on Flaky Tests10.1145/3643656.3643899(5-13)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3643656.3643899
Haas RNömmer RJuergens EApel S(2024)Optimization of Automated and Manual Software Tests in Industrial Practice: A Survey and Historical AnalysisIEEE Transactions on Software Engineering10.1109/TSE.2024.341819150:8(2005-2020)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3418191
Esposito MFalessi D(2024)VALIDATEInformation and Software Technology10.1016/j.infsof.2024.107448170:COnline publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1016/j.infsof.2024.107448
Weng SFeng YYin YDai YLiu JZhao Z(2024)Seeing the invisible: test prioritization for object detection systemEmpirical Software Engineering10.1007/s10664-024-10539-429:6Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1007/s10664-024-10539-4
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents