research-article

Open access

Empirical evaluation of pareto efficient multi-objective regression test case prioritisation

Authors:

Michael G. Epitropakis,

Edmund K. BurkeAuthors Info & Claims

ISSTA 2015: Proceedings of the 2015 International Symposium on Software Testing and Analysis

Pages 234 - 245

https://doi.org/10.1145/2771783.2771788

Published: 13 July 2015 Publication History

Abstract

The aim of test case prioritisation is to determine an ordering of test cases that maximises the likelihood of early fault revelation. Previous prioritisation techniques have tended to be single objective, for which the additional greedy algorithm is the current state-of-the-art. Unlike test suite minimisation, multi objective test case prioritisation has not been thoroughly evaluated. This paper presents an extensive empirical study of the effectiveness of multi objective test case prioritisation, evaluating it on multiple versions of five widely-used benchmark programs and a much larger real world system of over 1 million lines of code. The paper also presents a lossless coverage compaction algorithm that dramatically scales the performance of all algorithms studied by between 2 and 4 orders of magnitude, making prioritisation practical for even very demanding problems.

References

[1]

CLOC: count lines of code. http://cloc.sourceforge.net.

[2]

MySQL, Online Bug Repository. http://bugs.mysql.com/.

[3]

SIR: Software-artifact Infrastructure Repository. http://sir.unl.edu/.

[4]

The MySQL Test Framework, Version 2.0. http://bugs.mysql.com/.

[5]

A. Arcuri and L. Briand. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In Proceedings of the 33rd International Conference on Software Engineering, ICSE ’11, pages 1–10, New York, NY, USA, 2011. ACM.

Digital Library

[6]

R. Assi and W. Masri. Lossless reduction of execution profiles using a genetic algorithm. In Proceedings of the 7th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2014, pages 294–297, March 2014.

Digital Library

[7]

R. C. Bryce and C. J. Colbourn. Prioritized interaction testing for pair-wise coverage with seeding and constraints. Journal of Information and Software Technology, 48(10):960–970, 2006.

[8]

J. T. de Souza, C. L. Maia, F. G. de Freitas, and D. P. Coutinho. The human competitiveness of search based software engineering. In Proceedings of 2nd International Symposium on Search based Software Engineering (SSBSE 2010), pages 143–152, Benevento, Italy, 2010. IEEE Computer Society Press.

Digital Library

[9]

K. Deb, S. Agrawal, A. Pratab, and T. Meyarivan. A Fast Elitist Non-Dominated Sorting Genetic Algorithm for Multi-Objective Optimization: NSGA-II. In Proceedings of the Parallel Problem Solving from Nature Conference, pages 849–858. Springer, 2000.

Digital Library

[10]

H. Do, S. G. Elbaum, and G. Rothermel. Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact. Empirical Software Engineering, 10(4):405–435, 2005.

Digital Library

[11]

H. Do, G. Rothermel, and A. Kinneer. Prioritizing junit test cases: An empirical assessment and cost-benefits analysis. Empirical Software Engineering, 11(1):33–70, 2006.

Digital Library

[12]

S. Elbaum, A. Malishevsky, and G. Rothermel. Test case prioritization: a family of empirical studies. IEEE Transactions on Software Engineering, 28(2):159–182, Feb 2002.

Digital Library

[13]

S. G. Elbaum, A. G. Malishevsky, and G. Rothermel. Prioritizing test cases for regression testing. In Proceedings of International Symposium on Software Testing and Analysis, pages 102–112, August 2000.

Digital Library

[14]

S. G. Elbaum, A. G. Malishevsky, and G. Rothermel. Incorporating varying test costs and fault severities into test case prioritization. In Proceedings of the International Conference on Software Engineering (ICSE 2001), pages 329–338. ACM Press, May 2001.

Digital Library

[15]

A. Finkelstein, M. Harman, S. A. Mansouri, J. Ren, and Y. Zhang. ”fairness analysis” in requirements assignments. In Proceedings of the 16th IEEE International Requirements Engineering Conference (RE ’08), Barcelona, Catalunya, Spain, September 2008.

Digital Library

[16]

D. E. Goldberg. Genetic Algorithms in Search, Optimization and Machine Learning. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 1st edition, 1989.

Digital Library

[17]

D. E. Goldberg and R. Lingle, Jr. Alleles loci and the traveling salesman problem. In Proceedings of the 1st International Conference on Genetic Algorithms, pages 154–159, Hillsdale, NJ, USA, 1985. L. Erlbaum Associates Inc.

Digital Library

[18]

R. Grissom and J. Kim. Effect Sizes for Research: A Broad Practical Approach. Lawrence Erlbaum Associates, Inc., Publishers., 2005.

[19]

Q. Gu, B. Tang, and D. Chen. Optimal regression testing based on selective coverage of test requirements. In International Symposium on Parallel and Distributed Processing with Applications (ISPA 10), pages 419 – 426, Sept. 2010.

Digital Library

[20]

M. Harman. Making the case for MORTO: Multi Objective Regression Test Optimization. In Proceedings of the 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops, ICSTW ’11, pages 111–114, Washington, DC, USA, 2011. IEEE Computer Society.

Digital Library

[21]

D. Jeffrey and N. Gupta. Test suite reduction with selective redundancy. In Proceedings of the 21st IEEE International Conference on Software Maintenance 2005 (ICSM’05), pages 549–558. IEEE Computer Society Press, September 2005.

Digital Library

[22]

J.-M. Kim and A. Porter. A history-based test prioritization technique for regression testing in resource constrained environments. In Proceedings of the 24th International Conference on Software Engineering, pages 119–129. ACM Press, May 2002.

Digital Library

[23]

J. Knowles, L. Thiele, and E. Zitzler. A tutorial on the performance assessment of stochastic multiobjective optimizers. 214, Computer Engineering and Networks Laboratory (TIK), ETH Zurich, Switzerland, Feb. 2006. revised version.

[24]

Z. Li, Y. Bian, R. Zhao, and J. Cheng. A fine-grained parallel multi-objective test case prioritization on gpu. In G. Ruhe and Y. Zhang, editors, Search Based Software Engineering, volume 8084 of Lecture Notes in Computer Science, pages 111–125. Springer Berlin Heidelberg, 2013.

[25]

Z. Li, M. Harman, and R. M. Hierons. Search Algorithms for Regression Test Case Prioritization. IEEE Transactions on Software Engineering, 33(4):225–237, 2007.

Digital Library

[26]

A. G. Malishevsky, J. R. Ruthruff, G. Rothermel, and S. Elbaum. Cost-cognizant test case prioritization. Technical Report TR-UNL-CSE-2006-0004, Department of Computer Science and Engineering, University of Nebraska-Lincoln, March 2006.

[27]

N. Nethercote and J. Seward. Valgrind: A program supervision framework. In Proceedings of ACM Conference on Programming Language Design and Implementation, pages 89–100. ACM Press, June 2007.

[28]

Oracle Corporation. http://www.mysql.com.

[29]

K. Praditwong and X. Yao. A new multi-objective evolutionary optimisation algorithm: The two-archive algorithm. In Proceedings of Computational Intelligence and Security, International Conference, volume 4456 of Lecture Notes in Computer Science, pages 95–104, November 2006.

[30]

G. Rothermel, S. Elbaum, A. Malishevsky, P. Kallakuri, and B. Davia. The impact of test suite granularity on the cost-effectiveness of regression testing. In Proceedings of the 24th International Conference on Software Engineering (ICSE 2002), pages 130–140. ACM Press, May 2002.

Digital Library

[31]

G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold. Test case prioritization: An empirical study. In Proceedings of International Conference on Software Maintenance (ICSM 1999), pages 179–188. IEEE Computer Society Press, August 1999.

Digital Library

[32]

G. Rothermel, R. J. Untch, and C. Chu. Prioritizing test cases for regression testing. IEEE Transactions on Software Engineering, 27(10):929–948, October 2001.

Digital Library

[33]

J. P. Royston. An extension of shapiro and wilk’s w test for normality to large samples. Journal of the Royal Statistical Society. Series C (Applied Statistics), 31(2):115–124, 1982.

[34]

P. G. Sapna and M. Hrushikesha. Automated test scenario selection based on levenshtein distance. In T. Janowski and H. Mohanty, editors, 6th Distributed Computing and Internet Technology (ICDCIT’10), volume 5966 of Lecture Notes in Computer Science (LNCS), pages 255–266. Springer-Verlag (New York), Bhubaneswar, India, Feb. 2010.

Digital Library

[35]

A. Vargha and H. D. Delaney. A critique and improvement of the “CL” common language effect size statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics, 25(2):pp. 101–132, 2000.

[36]

D. A. V. Veldhuizen and G. B. Lamont. Multiobjective evolutionary algorithm research: A history and analysis. Technical Report TR-98-03, Department of Electrical and Computer Engineering, Air Force Institute of Technology, 1998.

[37]

S. Yoo and M. Harman. Pareto efficient multi-objective test case selection. In Proceedings of International Symposium on Software Testing and Analysis, pages 140–150. ACM Press, July 2007.

Digital Library

[38]

S. Yoo and M. Harman. Using hybrid algorithm for pareto effcient multi-objective test suite minimisation. Journal of Systems Software, 83(4):689–701, April 2010.

Digital Library

[39]

S. Yoo and M. Harman. Regression testing minimisation, selection and prioritisation: A survey. Software Testing, Verification, and Reliability, 22(2):67–120, March 2012.

Digital Library

[40]

S. Yoo, M. Harman, P. Tonella, and A. Susi. Clustering test cases to achieve effective & scalable prioritisation incorporating expert knowledge. In Proceedings of International Symposium on Software Testing and Analysis (ISSTA 2009), pages 201–211. ACM Press, July 2009.

Digital Library

[41]

S. Yoo, M. Harman, and S. Ur. Highly scalable multi-objective test suite minimisation using graphics card. In LNCS: Proceedings of the 3rd International Symposium on Search-Based Software Engineering, volume 6956 of SSBSE, pages 219–236, September 2011.

Digital Library

[42]

S. Yoo, M. Harman, and S. Ur. Gpgpu test suite minimisation: search based software engineering performance improvement using graphics cards. Empirical Software Engineering, 18(3):550–593, 2013.

[43]

S. Yoo, R. Nilsson, and M. Harman. Faster fault finding at Google using multi objective regression test optimisation. In 8th European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE ’11), Szeged, Hungary, September 5th - 9th 2011.

[44]

Industry Track.

[45]

L. Zhang, S.-S. Hou, C. Guo, T. Xie, and H. Mei. Time-aware test-case prioritization using Integer Linear Programming. In Proceedings of the International Conference on Software Testing and Analysis (ISSTA 2009), pages 212–222. ACM Press, July 2009.

Digital Library

[46]

E. Zitzler and L. Thiele. Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach. IEEE Transactions on Evolutionary Computation, 3(4):257–271, Nov. 1999.

Digital Library

Cited By

Trovato ADe Stefano MPecorelli FDi Nucci DDe Lucia A(2025)Reformulating regression test suite optimization using quantum annealing - an empirical studyInternational Journal on Software Tools for Technology Transfer10.1007/s10009-024-00775-w26:6(767-780)Online publication date: 20-Jan-2025
https://doi.org/10.1007/s10009-024-00775-w
Garg KShekhar S(2024)Fault sensitivity index-based multi-objective testcase prioritizationJournal of Electrical Engineering10.2478/jee-2024-001875:2(151-160)Online publication date: 4-Apr-2024
https://doi.org/10.2478/jee-2024-0018
Laaber CYue TAli S(2024)Evaluating Search-Based Software Microbenchmark PrioritizationIEEE Transactions on Software Engineering10.1109/TSE.2024.338083650:7(1687-1703)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3380836
Show More Cited By

Index Terms

Empirical evaluation of pareto efficient multi-objective regression test case prioritisation
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Pareto efficient multi-objective test case selection
ISSTA '07: Proceedings of the 2007 international symposium on Software testing and analysis

Previous work has treated test case selection as a single objective optimisation problem. This paper introduces the concept of Pareto efficiency to test case selection. The Pareto efficient approach takes multiple objectives such as code coverage, past ...
Multi-objective test case prioritization for GUI applications
SAC '13: Proceedings of the 28th Annual ACM Symposium on Applied Computing

Test case prioritization techniques are proposed to schedule execution of test cases in order to improve testing effectiveness. Various coverage criteria are used as surrogates for test case prioritization. They are expected to improve testing ...
On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques

Regression testing is an important activity in the software life cycle, but it can also be very expensive. To reduce the cost of regression testing, software testers may prioritize their test cases so that those which are more important, by some measure,...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA 2015: Proceedings of the 2015 International Symposium on Software Testing and Analysis

July 2015

447 pages

ISBN:9781450336208

DOI:10.1145/2771783

General Chair:
Michal Young
University of Oregon, USA
,
Program Chair:
Tao Xie
University of Illinois at Urbana-Champaign, USA

Copyright © 2015 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2015

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Engineering and Physical Sciences Research Council

Conference

ISSTA '15

Sponsor:

SIGSOFT

ISSTA '15: International Symposium on Software Testing and Analysis

July 13 - 17, 2015

MD, Baltimore, USA

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

68
Total Citations
View Citations
1,212
Total Downloads

Downloads (Last 12 months)129
Downloads (Last 6 weeks)11

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Trovato ADe Stefano MPecorelli FDi Nucci DDe Lucia A(2025)Reformulating regression test suite optimization using quantum annealing - an empirical studyInternational Journal on Software Tools for Technology Transfer10.1007/s10009-024-00775-w26:6(767-780)Online publication date: 20-Jan-2025
https://doi.org/10.1007/s10009-024-00775-w
Garg KShekhar S(2024)Fault sensitivity index-based multi-objective testcase prioritizationJournal of Electrical Engineering10.2478/jee-2024-001875:2(151-160)Online publication date: 4-Apr-2024
https://doi.org/10.2478/jee-2024-0018
Laaber CYue TAli S(2024)Evaluating Search-Based Software Microbenchmark PrioritizationIEEE Transactions on Software Engineering10.1109/TSE.2024.338083650:7(1687-1703)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3380836
Varun JKarthika R(2024)Improving agility in projects using machine learning algorithmMultimedia Tools and Applications10.1007/s11042-024-19909-y83:38(85987-86005)Online publication date: 17-Sep-2024
https://doi.org/10.1007/s11042-024-19909-y
Viggiato MPaas DBezemer CChandra SBlincoe KTonella P(2023)Prioritizing Natural Language Test Cases Based on Highly-Used Game FeaturesProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613872(1961-1972)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3613872
Yang ZKeung JYu XXiao YJin ZZhang J(2023)On the Significance of Category Prediction for Code-Comment SynchronizationACM Transactions on Software Engineering and Methodology10.1145/353411732:2(1-41)Online publication date: 29-Mar-2023
https://dl.acm.org/doi/10.1145/3534117
Banach R(2023)Graded Refinement, Retrenchment, and SimulationACM Transactions on Software Engineering and Methodology10.1145/353411632:2(1-69)Online publication date: 31-Mar-2023
https://dl.acm.org/doi/10.1145/3534116
Birchler CKhatiri SDerakhshanfar PPanichella SPanichella A(2023)Single and Multi-objective Test Cases Prioritization for Self-driving Cars in Virtual EnvironmentsACM Transactions on Software Engineering and Methodology10.1145/353381832:2(1-30)Online publication date: 4-Apr-2023
https://dl.acm.org/doi/10.1145/3533818
Sun CDai HLiu HChen T(2023)Feedback-Directed Metamorphic TestingACM Transactions on Software Engineering and Methodology10.1145/353331432:1(1-34)Online publication date: 13-Feb-2023
https://dl.acm.org/doi/10.1145/3533314
Wu JClause J(2023)Automated Identification of Uniqueness in JUnit TestsACM Transactions on Software Engineering and Methodology10.1145/353331332:1(1-32)Online publication date: 13-Feb-2023
https://dl.acm.org/doi/10.1145/3533313
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten