article

An experimental study of four typical test suite reduction techniques

Authors:

Hong MeiAuthors Info & Claims

Information and Software Technology, Volume 50, Issue 6

Pages 534 - 546

https://doi.org/10.1016/j.infsof.2007.06.003

Published: 01 May 2008 Publication History

Abstract

In software development, developers often rely on testing to reveal bugs. Typically, a test suite should be prepared before initial testing, and new test cases may be added to the test suite during the whole testing process. This may usually cause the test suite to contain more or less redundancy. In other words, a subset of the test suite (called the representative set) may still satisfy all the test objectives. As the redundancy can increase the cost of executing the test suite, quite a few test suite reduction techniques have been brought out in spite of the NP-completeness of the general problem of finding the optimal representative set of a test suite. In the literature, there have been some experimental studies of test suite reduction techniques, but the limitations of the these experimental studies are quite obvious. Recently proposed techniques are not experimentally compared against each other, and reported experiments are mainly based on small programs or even simulation data. This paper presents a new experimental study of the four typical test suite reduction techniques, including Harrold et al.'s heuristic, and three other recently proposed techniques such as Chen and Lau's GRE heuristic, Mansour and El-Fakin's genetic algorithm-based approach, and Black et al.'s ILP-based approach. Based on the results of this experimental study, we also provide a guideline for choosing the appropriate test suite reduction technique and some insights into why the techniques vary in effectiveness and efficiency.

References

[1]

Aho, A., Hopcroft, J. and Ullman, J., The Design and Analysis of Computer Algorithms. 1974. Addison-Wesley.

[2]

Basili, V.R., Selby, R.W. and Hutchens, D.H., Experimentation in software engineering. IEEE Transaction Software Engineering. v12 i7. 733-743.

[3]

Binkley, D., Semantics guided regression test cost reduction. IEEE Transactions on Software Engineering. v23 i8. 498-516.

[4]

Black, J., Melachrinoudis, E. and Kaeli, D., Bi-criteria models for all-uses test suite reduction. In: Proceedings of 26th International Conference on Software Engineering, IEEE Computer Society, Washington,DC, USA. pp. 106-115.

[5]

Chen, T.Y. and Lau, M., Dividing strategies for the optimization of a test suite. Information Processing Letters. v60 i3. 135-141.

[6]

Chen, T.Y. and Lau, M., A new heuristic for test suite reduction. Information and Software Technology. v40 i5-6. 347-354.

[7]

Chen, T.Y. and Lau, M., A simulation study on some heuristics for test suite reduction. Information and Software Technology. v40 i13. 777-787.

[8]

Elbaum, S., Malishevsky, A. and Rothermel, G., Test case prioritization: a family of empirical studies. IEEE Transactions on Software Engineering. v28 i2. 159-182.

[9]

E. Gizdarski, H. Fujiwara. Spirit: Satisfiability problem implementation for redundancy identification and test generation, in: Proceedings of Ninth Asian Test Symposium (ATS'00), Taiwan, 2000, IEEE Computer Society, p. 171.

[10]

Goldberg, D., Genetic Algorithms in Search, Optimization and Machine Learning. 1989. Addison-Wesley.

[11]

Graves, T., Harrold, M., Kim, J., Porter, A. and Rothermel, G., An empirical study of regression test selection techniques. ACM Transactions on Software Engineering and Methodology (TOSEM). v10 i2. 184-208.

[12]

Gupta, R., Harrold, M.J. and Soffa, M.L., Program slicing-based regression testing techniques. Journal of Software Testing Verification and Reliability. v6 i2. 83-111.

[13]

Harrold, M.J., Gupta, R. and Soffa, M.L., A methodology for controlling the size of a test suite. ACM Transactions on Software Engineering and Methodology. v2 i3. 270-285.

[14]

M.J. Harrold, A. Orso, D. Rosenblum, G. Rothermel, M.L. Soffa, H. Do, Using Component Metadata to Support the Regression Testing of Component-Based Software, 2000.

[15]

Hartmann, J. and Robson, D., Revalidation during the software maintenance phase. Proceedings of International Conference on Software Maintenance. 0-80.

[16]

J. Horgan, S. London, ATAC: a data flow coverage testing tool for C, In: Proc. Symposium of Quality Software Development Tools, 1992, pp. 2-10.

[17]

Hutchins, M., Foster, H., Goradia, T. and Ostrand, T., Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria. In: Proceedings of 16th International Conference on Software Engineering, IEEE Computer Society Press, Los Alamitos, CA, USA. pp. 191-200.

[18]

Johnson, D.S., Approximation algorithms for combinatorial problems. Journal of Computer and System Sciences. v9 i3. 256-278.

[19]

Leung, H. and White, L., Insights into regression testing. Proceedings of the Conference on Software Maintenance. 60-69.

[20]

Li, Z., Harman, M. and Hierons, R., Search algorithms for regression test case prioritization. IEEE Transactions on Software Engineering. v33 i4. 225-237.

[21]

Mansour, N. and El-Fakih, K., Simulated annealing and genetic algorithms for optimal regression testing. Journal of Software Maintenance. v11 i1. 19-34.

[22]

Myers, G., Software Reliability Principles and Practice. 1979. John Wiley and Sons, New York.

[23]

J. Offutt, J. Pan, J. Voas, Procedures for reducing the size of coverage-based test sets, in: Proceedings of 12th International Conference on Testing Computer Software, 1995, pp. 111-123.

[24]

Orso, A., Shi, N. and Harrold, M.J., Scaling regression testing to large software systems. ACM SIGSOFTSoftware Engineering Notes. v29 i6. 241-251.

[25]

T. Ralphs, M. Guzelsoy, The symphony callable library for mixed integer programming, in: Proceedings of Ninth INFORMS Computing Society Conference, 2005, pp. 61-73.

[26]

G. Rothermel, M.J. Harrold, J. Ostrin, C. Hong, An empirical study of the effects of minimization on the fault detection capabilities of test suites, in: Proceedings of 14th International Conference on Software Maintenance, 1998, pp. 34-43.

[27]

Rothermel, G., Harrold, M.J., von Ronne, J. and Hong, C., Empirical studies of test-suite reduction. Software Testing, Verification And Reliability. v12 i4. 219-249.

[28]

Rothermel, G., Untch, R.H., Chu, C. and Harrold, M.J., Prioritizing test cases for regression testing. IEEE Transactions on Software Engineering. v27 i10. 929-948.

[29]

Rusu, V., du Bousquet, L. and Jeron, T., An approach to symbolic test generation. In: Proceedings of Second International Conference on Integrated Formal Methods, Springer-Verlag, London, UK. pp. 338-357.

[30]

P. Thevenod Fosse, H. Waeselynck, Y. Crouzet, An experimental study on software structural testing: deterministic versus random input generation. Fault-Tolerant Computing, 1991. FTCS-21. Digest of Papers, 21st International Symposium, 1991, pp. 410-417.

[31]

F. Vokolos, P. Frankl, Pythia: a regression test selection tool based on textual differencing, in: Proceedings of Third International Conference on Reliability, Quality and Safty of Software-Internsive Systems, 1997.

[32]

F.I. Vokolos, P.G. Frankl, Empirical evaluation of the textual differencing regression testing technique, in: Proceedings of 14th International Conference on Software Maintenance, 1998, pp. 44-53.

[33]

Walcott, K., Soffa, M., Kapfhammer, G. and Roos, R., TimeAware test suite prioritization. Proceedings of the 2006 International Symposium on Software Testing and Analysis. 1-12.

[34]

M.B. Wall, A genetic algorithm for resource-constrained scheduling. Ph.D. Thesis, Massachusetts Institute of Technology, 1996.

[35]

Wong, W.E., Horgan, J.R., London, S. and Mathur, A.P., Effect of test set minimization on fault detection effectiveness. Software: Practice and Experience. v28 i4. 347-369.

[36]

Wong, W.E., Horgan, J.R., Mathur, A.P. and Pasquini, A., Test set size minimization and fault detection effectiveness: a case study in a space application. Journal of Systems and Software. v48 i2. 79-89.

[37]

S. Yoo, M. Harman, Pareto efficient multi-objective test case selection. Proceedings of the 2007 International Symposium on Software Testing and Analysis (2007) 140-150.

[38]

H. Zhong, L. Zhang, H. Mei, An experimental comparison of four test suite reduction techniques, In: Proc. 28th International Conference on Software Engineering, ACM Press, New York, NY, USA, 2006, pp. 636-640.

Digital Library

Cited By

Wu JZhang ZYang DXu JHe JMao X(2024)Knowledge-Augmented Mutation-Based Bug Localization for Hardware Design CodeACM Transactions on Architecture and Code Optimization10.1145/366052621:3(1-26)Online publication date: 22-Apr-2024
https://dl.acm.org/doi/10.1145/3660526
Baral TRahman SChanumolu BBalcı BTuncer TShi ALam WBissyandé TKlein JBird CSarro F(2023)Optimizing Continuous Development By Detecting and Preventing Unnecessary Content GenerationProceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE56229.2023.00216(901-913)Online publication date: 11-Nov-2023
https://dl.acm.org/doi/10.1109/ASE56229.2023.00216
Mahdieh MMirian-Hosseinabadi SMahdieh M(2022)Test case prioritization using test case diversification and fault-proneness estimationsAutomated Software Engineering10.1007/s10515-022-00344-y29:2Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1007/s10515-022-00344-y
Show More Cited By

Index Terms

Recommendations

An experimental comparison of four test suite reduction techniques
ICSE '06: Proceedings of the 28th international conference on Software engineering

As a test suite usually contains redundancy, a subset of the test suite (representative set) may still satisfy all the test objectives. As the redundancy increases the cost of executing the test suite, many test suite reduction techniques have been ...
Improving Fault Detection Capability by Selectively Retaining Test Cases during Test Suite Reduction

Software testing is a critical part of software development. As new test cases are generated over time due to software modifications, test suite sizes may grow significantly. Because of time and resource constraints for testing, test suite minimization ...
Analysis of test suite reduction with enhanced tie-breaking techniques

Test suite minimization techniques try to remove redundant test cases of a test suite. However, reducing the size of a test suite might reduce its ability to reveal faults. In this paper, we present a novel approach for test suite reduction that uses an ...

Comments

Information & Contributors

Information

Published In

cover image Information and Software Technology

Information and Software Technology Volume 50, Issue 6

May, 2008

125 pages

ISSN:0950-5849

Issue’s Table of Contents

Copyright © Elsevier B.V. © 2007.

Publisher

Butterworth-Heinemann

United States

Publication History

Published: 01 May 2008

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

24
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 11 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wu JZhang ZYang DXu JHe JMao X(2024)Knowledge-Augmented Mutation-Based Bug Localization for Hardware Design CodeACM Transactions on Architecture and Code Optimization10.1145/366052621:3(1-26)Online publication date: 22-Apr-2024
https://dl.acm.org/doi/10.1145/3660526
Baral TRahman SChanumolu BBalcı BTuncer TShi ALam WBissyandé TKlein JBird CSarro F(2023)Optimizing Continuous Development By Detecting and Preventing Unnecessary Content GenerationProceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE56229.2023.00216(901-913)Online publication date: 11-Nov-2023
https://dl.acm.org/doi/10.1109/ASE56229.2023.00216
Mahdieh MMirian-Hosseinabadi SMahdieh M(2022)Test case prioritization using test case diversification and fault-proneness estimationsAutomated Software Engineering10.1007/s10515-022-00344-y29:2Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1007/s10515-022-00344-y
Goyal AShyamasundar RSivakumar GJetley RRamaswamy S(2021)Empirical Analysis of Greedy, GE and GRE HeuristicsProceedings of the 14th Innovations in Software Engineering Conference (formerly known as India Software Engineering Conference)10.1145/3452383.3452389(1-11)Online publication date: 25-Feb-2021
https://dl.acm.org/doi/10.1145/3452383.3452389
Chen JWang GHao DXiong YZhang HZhang LXie B(2021)Coverage Prediction for Accelerating Compiler TestingIEEE Transactions on Software Engineering10.1109/TSE.2018.288977147:2(261-278)Online publication date: 1-Feb-2021
https://dl.acm.org/doi/10.1109/TSE.2018.2889771
Agrawal AChoudhary AKaur APandey H(2020)Fault coverage-based test suite optimization method for regression testing: learning from mistakes-based approachNeural Computing and Applications10.1007/s00521-019-04098-932:12(7769-7784)Online publication date: 1-Jun-2020
https://dl.acm.org/doi/10.1007/s00521-019-04098-9
Coviello CRomano SScanniello GOivo MMéndez DMockus A(2018)An empirical study of inadequate and adequate test suite reduction approachesProceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3239235.3240497(1-10)Online publication date: 11-Oct-2018
https://dl.acm.org/doi/10.1145/3239235.3240497
Palomo-Lozano FEstero-Botaro AMedina-Bulo INúñez MTakadama K(2018)Test suite minimization for mutation testing of WS-BPEL compositionsProceedings of the Genetic and Evolutionary Computation Conference10.1145/3205455.3205533(1427-1434)Online publication date: 2-Jul-2018
https://dl.acm.org/doi/10.1145/3205455.3205533
Zhang LChaudron MCrnkovic IChechik MHarman M(2018)Hybrid regression test selectionProceedings of the 40th International Conference on Software Engineering10.1145/3180155.3180198(199-209)Online publication date: 27-May-2018
https://dl.acm.org/doi/10.1145/3180155.3180198
Choi WSen KNecula GWang WChaudron MCrnkovic IChechik MHarman M(2018)DetReduceProceedings of the 40th International Conference on Software Engineering10.1145/3180155.3180173(445-455)Online publication date: 27-May-2018
https://dl.acm.org/doi/10.1145/3180155.3180173
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents