research-article

Automated code-based test case reuse for software product line testing

Authors:

Uicheon LeeAuthors Info & Claims

Volume 166, Issue C

https://doi.org/10.1016/j.infsof.2023.107372

Published: 04 March 2024 Publication History

Abstract

Context

A software product line (SPL) grows in size as a new product is developed. A new product in an SPL should be tested extensively for quality assurance. For the efficient testing, previous studies suggested reusing the existing test cases of a product family. However, either their methods were not efficient because interventions from human experts, specifications, architecture and/or traceabilities for test cases were required.

Objective

To address these limitations, we propose an Automated Code-based Test case reuse for SPLs (ActSPL). ActSPL automatically identifies reusable test cases for new products of a product family using source code and test cases.

Method

ActSPL automatically constructs a hash-based traceability links between test cases and source code of a product family. Using the traceability links, ActSPL selects reusable test cases for a given new product from existing test cases of the product family.

Results

We evaluated ActSPL in terms of the effectiveness and cost reduction of reusing test cases with five open-source SPLs. The evaluation results showed that ActSPL, on average, achieved 100 % precision and 62 % recall. In addition, ActSPL, on average, saved 47.5 % of time required for testing a new product from scratch.

Conclusion

Our study shows the feasibility of ActSPL reusing SPL test cases based on source code and test cases. Our results can be a basis for successive studies for automated code-based SPL testing.

References

[1]

B. Beizer, Software Testing Techniques, International Thomson Computer Press, 1990.

Digital Library

[2]

R. Lachmann, S. Lity, S. Lischke, S. Beddig, S. Schulze, I. Schaefer, Delta-oriented test case prioritization for integration testing of software product lines, in: Proceedings of the 19th International Conference on Software Product Line, 2015, pp. 81–90.

[3]

I. do Carmo Machado, J.D. McGregor, Y.C. Cavalcanti, E.S. De Almeida, On strategies for testing software product lines: a systematic literature review, Inf. Softw. Technol. 56 (10) (2014) 1183–1199.

[4]

K.L. Petry, E. OliveiraJr, A.F. Zorzo, Model-based testing of software product lines: Mapping study and research roadmap, J. Syst. Softw. 167 (2020).

[5]

S. Lity, M. Nieke, T. Thüm, I. Schaefer, Retest test selection for product-line regression testing of variants and versions of variants, J. Syst. Softw. 147 (2019) 46–63.

[6]

M. Lochau, S. Lity, R. Lachmann, I. Schaefer, U. Goltz, Delta-oriented model-based integration testing of large-scale systems, J. Syst. Softw. 91 (2014) 63–84.

[7]

F. Damiani, D. Faitelson, C. Gladisch, S. Tyszberowicz, A novel model-based testing approach for software product lines, Softw. Syst. Model. 16 (4) (2017) 1223–1251.

[8]

E. Uzuncaova, S. Khurshid, D. Batory, Incremental test generation for software product lines, IEEE Trans. Softw. Eng. 36 (3) (2010) 309–322.

[9]

M. Varshosaz, H. Beohar, M.R. Mousavi, Delta-oriented FSM-based testing, in: International Conference on Formal Engineering Methods, Springer, Cham, 2015, pp. 366–381.

[10]

C.S. Gebizli, H. Sözer, Model-based software product line testing by coupling feature models with hierarchical markov chain usage models, in: 2016 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C), 2016, pp. 278–283.

[11]

S. Wang, S. Ali, A. Gotlieb, M. Liaaen, Automated product line test case selection: industrial case study and controlled experiment, Softw. Syst. Model. 16 (2) (2017) 417–441.

[12]

S. Wang, S. Ali, A. Gotlieb, M. Liaaen, A systematic test case selection methodology for product lines: results and insights from an industrial case study, Empir. Softw. Eng. 21 (4) (2016) 1586–1622.

[13]

K. Pohl, G. Böckle, F. Van Der Linden, Software product line engineering: Foundations, Principles and Techniques, 10, Springer Sciences & Business Media, 2005, pp. 3–540.

[14]

P. Jung, S. Kang, J. Lee, Efficient regression testing of software product lines by reducing redundant test executions, Appl. Sci. 10 (23) (2020) 8686.

[15]

M. Gligoric, L. Eloussi, D. Marinov, Practical regression test selection with dynamic file dependencies, in: Proceedings of the 2015 International Symposium on Software Testing and Analysis, 2015, pp. 211–222.

[16]

M. Vasic, Z. Parvez, A. Milicevic, M. Gligoric, File-level vs. module-level regression test selection for. net, in: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, 2017, pp. 848–853.

[17]

L. Zhang, Hybrid regression test selection, in: Proceedings of the 40th International Conference on Software Engineering, 2018, pp. 199–209.

[18]

A. Celik, M. Vasic, A. Milicevic, M. Gligoric, Regression test selection across JVM boundaries, in: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, 2017, pp. 809–820.

[19]

G. Fraser, A. Arcuri, Evosuite: automatic test suite generation for object-oriented software, in: Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering, 2011, pp. 416–419.

[20]

A. Sakti, G. Pesant, Y.G. Guéhéneuc, Instance generator and problem representation to improve object oriented code coverage, IEEE Trans. Softw. Eng. 41 (3) (2014) 294–313.

[21]

C. Pacheco, M.D. Ernst, Randoop: feedback-directed random testing for Java, in: Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion, 2007, pp. 815–816.

[22]

J. Meinicke, C.P. Wong, C. Kästner, T. Thüm, G. Saake, On essential configuration complexity: measuring interactions in highly-configurable systems, in: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, 2016, pp. 483–494.

[23]

F. Angerer, P. Grünbacher, H. Prähofer, L. Linsbauer, An experiment comparing lifted and delayed variability-aware program analysis, in: 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2017, pp. 148–158.

[24]

J. Kim, D. Batory, D. Dig, Refactoring java software product lines, in: Proceedings of the 21st International Systems and Software Product Line Conference-Volume A, 2017, pp. 59–68.

[25]

J. Benesty, J. Chen, Y. Huang, I. Cohen, Pearson correlation coefficient, Noise Reduction Speech Processing, Springer-Verlag, Berlin, Germany, 2009, pp. 1–4.

[26]

M. Perscheid, D. Cassou, R. Hirschfeld, Test quality feedback improving effectivity and efficiency of unit testing, in: 2012 10th International Conference on Creating, Connecting and Collaborating through Computing, IEEE, 2012, pp. 60–67.

[27]

Y. Qi, X. Mao, Z. Dai, Y. Qi, Efficient automatic program repair using function-based part-execution, in: 2013 IEEE 4th International Conference on Software Engineering and Service Science, IEEE, 2013, pp. 235–238.

[28]

D. Paterson, J. Campos, R. Abreu, G.M. Kapfhammer, G. Fraser, P. McMinn, An empirical study on the use of defect prediction for test case prioritization, in: 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST), IEEE, 2019, pp. 346–357.

[29]

J. Bell, O. Legunsen, M. Hilton, L. Eloussi, T. Yung, D. Marinov, DeFlaker: Automatically detecting flaky tests, in: Proceedings of the International Conference on Software Engineering (ICSE’18, 2018, pp. 433–444.

[30]

O. Parry, G.M. Kapfhammer, M. Hilton, P. McMinn, A survey of flaky tests, ACM Trans. Softw. Eng. Methodol. (TOSEM) 31 (1) (2021) 1–74.

[31]

A. Gambi, J. Bell, A. Zeller, Practical test dependency detection, in: 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST), 2018, pp. 1–11.

[32]

C. Zhu, O. Legunsen, A. Shi, M. Gligoric, A framework for checking regression test selection tools, in: 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), 2019, pp. 430–441.

[33]

A. Alshammari, C. Morris, M. Hilton, J. Bell, FlakeFlagger: Predicting flakiness without rerunning tests, in: Proceedings of the International Conference on Software Engineering (ICSE’21), 2021.

[34]

A. Shi, W. Lam, R. Oei, T. Xie, D. Marinov, iFixFlakies: A framework for automatically fixing orderdependent flaky tests, in: Proceedings of the Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’19), 2019, pp. 545–555.

[35]

P. Zhang, Y. Jiang, V.Stodden A.Wei, D. Marinov, A. Shi, Domain-specific fixes for flaky tests with wrong assumptions on underdetermined specifications, in: Proceedings of the International Conference on Software Engineering (ICSE’21), 2021, pp. 50–61.

[36]

G. Rothermel, M.J. Harrold, Analyzing regression test selection techniques, IEEE Trans. Softw. Eng. 22 (8) (1996) 529–551.

Digital Library

[37]

C. Gacek, M. Anastasopoules, Implementing product line variabilities, in: Proceedings of the 2001 symposium on Software reusability: putting software reuse in context, 2001, pp. 109–117.

[38]

F. Kifetew, X. Devroey, U. Rueda, Java unit testing tool competition-seventh round, 2019 IEEE/ACM 12th International Workshop on Search-Based Software Testing (SBST), IEEE, 2019, pp. 15–20.

[39]

S. Schweikl, G. Fraser, A. Arcuri, EvoSuite at the SBST 2022 Tool Competition, 2022 IEEE/ACM 15th International Workshop on Search-Based Software Testing (SBST), IEEE, 2022, pp. 33–34.

[40]

X. Li, W.E. Wong, R. Gao, L. Hu, S. Hosono, Genetic algorithm-based test generation for software product line with the integration of fault localization techniques, Empir. Softw. Eng. 23 (1) (2018) 1–51.

[41]

Z. Xu, M.B. Cohen, W. Motycka, G. Rothermel, Continuous test suite aug- mentation in software product lines, in: Proceedings of the 17th International Software Product Line Conference, 2013, pp. 52–61.

[42]

M. Dukaczewski, I. Schaefer, R. Lachmann, M. Lochau, Requirements-based delta-oriented spl testing, in: 2013 4th International Workshop on Product LinE Approaches in Software Engineering (PLEASE), 2013, pp. 49–52.

[43]

C. Nebut, Y.L. Traon, J.M. Jézéquel, System testing of product lines: From requirements to test cases, Software Product Lines, Springer, Berlin, Heidelberg, 2006, pp. 447–477.

[44]

A. Bertolino, A. Fantechi, S. Gnesi, G. Lami, Product line use cases: Scenario-based specification and testing of requirements, Software Product Lines, Springer, Berlin, Heidelberg, 2006, pp. 425–445.

[45]

A. Reuys, S. Reis, E. Kamsties, K. Pohl, The scented method for testing software product lines, Software Product Lines, Springer, Berlin, Heidelberg, 2006, pp. 479–520.

Index Terms

Automated code-based test case reuse for software product line testing
1. Software and its engineering
  1. Software creation and management
    1. Software development techniques
      1. Reusability
    2. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software notations and tools

Index terms have been assigned to the content through auto-classification.

Recommendations

Systematic Software Product Line Test Case Derivation for Test Data Reuse
COMPSAC '15: Proceedings of the 2015 IEEE 39th Annual Computer Software and Applications Conference - Volume 03

This paper proposes a systematic software product line test case derivation method that reuses test data. With the method, significant test cases reduction can be achieved over the conventional software product line testing. The key to achieving reuse ...
Automated code-based test selection for software product line regression testing
Highlights
- The first code-based regression test selection method for SPLs is presented.
- A ...
Abstract
Regression testing for software product lines (SPLs) is challenging and can be expensive because it must ensure that all the products of a product family are correct whenever changes are made. SPL regression testing can be made ...
Automated product line test case selection: industrial case study and controlled experiment

Automated test case selection for a new product in a product line is challenging due to several reasons. First, the variability within the product line needs to be captured in a systematic way; second, the reusable test cases from the repository are ...

Comments

Information & Contributors

Information

Published In

cover image Information and Software Technology

Information and Software Technology Volume 166, Issue C

Feb 2024

156 pages

Issue’s Table of Contents

Copyright © 2023.

Publisher

Butterworth-Heinemann

United States

Publication History

Published: 04 March 2024

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents