research-article

Multi-objective black-box test case selection for system testing

Authors:

Michael Felderer,

Sandro Schulze,

Christoph Seidl,

Ina SchaeferAuthors Info & Claims

GECCO '17: Proceedings of the Genetic and Evolutionary Computation Conference

Pages 1311 - 1318

https://doi.org/10.1145/3071178.3071189

Published: 01 July 2017 Publication History

Abstract

Testing is a fundamental task to ensure software quality. Regression testing aims to ensure that changes to software do not introduce new failures. As resources are often limited and testing comprises a vast amount of test cases, different regression strategies have been proposed to reduce testing effort by selecting or prioritizing important test cases, e.g., code coverage (to ensure a sufficient testing depth). However, in system testing, source code is often not available creating a black-box system. In this paper, we introduce an automated, multi-objective test case selection technique in black-box systems using genetic algorithms. We define seven different objectives, based on meta-data, allowing a flexible test case selection for a variety of systems. For evaluation, we apply our technique on two different subject systems assessing the feasibility and suitability of our test case selection approach. Results indicate that our approach is applicable based on different data available and is able to outperform random test case selection and retest-all.

References

[1]

S. Amland. Risk-based testing: Risk analysis fundamentals and metrics for software testing including a financial application case study. Journal of Systems and Software, 53(3):287--295, 2000.

Digital Library

[2]

P. Ammann and J. Offutt. Introduction to Software Testing. Cambridge University Press, 2008.

Digital Library

[3]

A. Baresel, H. Sthamer, and M. Schmidt. Fitness function design to improve evolutionary structural testing. In Proc. Genetic and Evol. Comp. Conf., pages 1329--1336. Morgan Kaufmann Publishers Inc., 2002.

Digital Library

[4]

S. Biswas, R. Mall, M. Satpathy, and S. Sukumaran. Regression test selection techniques: A survey. Informatica, 35(3), 2011.

[5]

L. C. Briand, Y. Labiche, and M. Shousha. Stress testing real-time systems with genetic algorithms. In Proc. Genetic and Evol. Comp. Conf., pages 1021--1028, 2005.

Digital Library

[6]

A. P. Conrad, R. S. Roos, and G. M. Kapfhammer. Empirically studying the role of selection operators during search-based test suite prioritization. In Proc. Genetic and Evol. Comp. Conf., pages 1373--1380, 2010.

Digital Library

[7]

L. De Souza, P. de Miranda, R. Prudencio, and F. de Barros. A multi-objective particle swarm optimization for test case selection based on functional requirements coverage and execution effort. In Proc. Int'l Conf. Tools Artifi. Intel., pages 245--252, 2011.

Digital Library

[8]

K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan. A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE Transactions on Evolutionary Computation, 6(2):182--197, 2002.

Digital Library

[9]

J. J. Durillo and A. J. Nebro. jmetal: A Java framework for multi-objective optimization. Adv. Eng. Softw., 42(10):760--771, 2011.

Digital Library

[10]

E. D. Ekelund and E. Engström. Efficient regression testing based on test history: An industrial evaluation, page 9. IEEE Computer Society, 2015.

Digital Library

[11]

E. Engström, P. Runeson, and A. Ljung. Improving regression testing transparency and efficiency with history-based prioritization - an industrial case study. In Proc. Int'l Conf. Software Testing, Verification, and Validation, pages 367--376. IEEE, 2011.

Digital Library

[12]

E. Engström, P. Runeson, and M. Skoglund. A systematic review on regression test selection techniques. Info. and Softw. Techn., 52:14--30, 2010.

Digital Library

[13]

G. Erdogan, Y. Li, R. K. Runde, F. Seehusen, and K. StØlen. Approaches for the combined use of risk analysis and testing: A systematic literature review. Int. J. Softw. Tools Technol. Transf., pages 627--642, 2014.

Digital Library

[14]

Y. Fazlalizadeh, A. Khalilian, M. A. Azgomi, and S. Parsa. Prioritizing test cases for resource constraint environments using historical test case performance data. In Proc. Int'l Conf. on Comp. Science Information Techn., pages 190--195. IEEE, 2009.

[15]

M. Felderer, C. Haisjackl, V. Pekar, and R. Breu. A risk assessment framework for software testing. In Proc. Int'l Symposium Leveraging Applications of Formal Methods, Verification and Validation, pages 292--308. 2014.

[16]

M. Felderer, C. Haisjackl, V. Pekar, and R. Breu. An exploratory study on risk estimation in risk-based testing approaches. In Software Quality Days (SWQD) 2015, pages 32--43. 2015.

[17]

M. Felderer and I. Schieferdecker. A taxonomy of risk-based testing. Int. J. Softw. Tools Technol. Transf., 16(5):559--568, 2014.

Digital Library

[18]

M. Harman and P. McMinn. A theoretical and empirical study of search-based testing: Local, global, and hybrid search. IEEE Trans. Soft. Eng., 36(2):226--247, 2010.

Digital Library

[19]

M. J. Harrold. Testing: A roadmap. In Proc. Int'l Conf. Software Engineering, pages 61--72. ACM, 2000.

Digital Library

[20]

H. Hemmati, A. Arcuri, and L. Briand. Achieving scalable model-based testing through test case diversity. ACM Trans. Soft. Eng. and Methods, 22(1):6:1--6:42, 2013.

Digital Library

[21]

K. Herzig, M. Greiler, J. Czerwonka, and B. Murphy. The art of testing less without sacrificing quality. In Proc. Int'l Conf. Software Engineering. IEEE, 2015. to appear.

Digital Library

[22]

Y.-C. Huang, K.-L. Peng, and C.-Y. Huang. A history-based cost-cognizant test case prioritization technique in regression testing. J. Sys. and Soft., 85(3):626 -- 637, 2012.

Digital Library

[23]

J. Knowles and D. Corne. The pareto archived evolution strategy: a new baseline algorithm for pareto multiobjective optimisation. In Proc. Congress on Evol. Comp., volume 1, page 105, 1999.

[24]

P. Larrañaga, C. M. H. Kuijpers, R. H. Murga, I. Inza, and S. Dizdarevic. Genetic algorithms for the travelling salesman problem: A review of representations and operators. Art. Int. Rev., 13(2):129--170, 1999.

Digital Library

[25]

S. Lity, R. Lachmann, M. Lochau, and I. Schaefer. Delta-oriented software product line test models - the body comfort system case study. Technical report, TU Braunschweig, 2013.

[26]

P. McMinn. Search-based software testing: Past, present and future. In Proc. Int'l Conf. Softw. Testing, Verification and Validation Workshops, pages 153--163. IEEE Computer Society, 2011.

Digital Library

[27]

D. Mondal, H. Hemmati, and S. Durocher. Exploring test suite diversification and code coverage in multi-objective test case selection. In Proc. Int'l Conf. Software Testing, Verification, and Validation, pages 1--10, 2015.

[28]

S. Nidhra and J. Dondeti. Black box and white box testing techniques-a literature review. 2(2):29--50, 2012.

[29]

G. Rothermel and M. Harrold. Analyzing regression test selection techniques. IEEE Trans. Soft. Eng., 22(8):529--551, 1996.

Digital Library

[30]

S. Shashank, P. Chakka, and D. Kumar. A systematic literature survey of integration testing in component-based software engineering. In Proc. Int'l Conf. Comp. and Comm. Tech., pages 562--568, 2010.

[31]

M. W. Whalen, A. Rajan, M. P. Heimdahl, and S. P. Miller. Coverage metrics for requirements-based testing. In Proc. Int'l Symposium Software Testing and Analysis, pages 25--36. ACM, 2006.

Digital Library

[32]

S. Yoo and M. Harman. Pareto efficient multi-objective test case selection. In Proc. Int'l Symposium Software Testing and Analysis, pages 140--150. ACM, 2007.

Digital Library

[33]

S. Yoo and M. Harman. Regression testing minimization, selection and prioritization: A survey. Softw. Test. Verif. Reliab., 22(2):67--120, 2007.

Digital Library

Cited By

Ugarte MValle PIllarramendi MArrieta A(2025)Enhancing multi-objective test case selection through the mutation operatorAutomated Software Engineering10.1007/s10515-025-00489-632:1Online publication date: 30-Jan-2025
https://doi.org/10.1007/s10515-025-00489-6
Yao QZhang YLi YLiu FYang S(2024)Multimodal Multi-Objective Test Data Generation Method based on Particle Swarm Optimization2024 IEEE 24th International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS62785.2024.00016(61-71)Online publication date: 1-Jul-2024
https://doi.org/10.1109/QRS62785.2024.00016
Arrieta AValle PAgirre JSagardui G(2023)Some Seeds Are Strong: Seeding Strategies for Search-based Test Case SelectionACM Transactions on Software Engineering and Methodology10.1145/353218232:1(1-47)Online publication date: 13-Feb-2023
https://dl.acm.org/doi/10.1145/3532182
Show More Cited By

Index Terms

Multi-objective black-box test case selection for system testing
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
2. Theory of computation
  1. Design and analysis of algorithms
    1. Mathematical optimization
      1. Discrete optimization
        Optimization with randomized search heuristics
        Evolutionary algorithms

Recommendations

Black-box Test Case Selection by Relating Code Changes with Previously Fixed Defects
EASE '22: Proceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering

Software continuously changes to address new requirements and to fix defects. Regression testing is performed to ensure that the applied changes do not adversely affect existing functionality. The increasing number of test cases makes it infeasible to ...
Selection and Prioritization of Test Cases by Combining White-Box and Black-Box Testing Methods
ECBS-EERC '13: Proceedings of the 2013 3rd Eastern European Regional Conference on the Engineering of Computer Based Systems

In this paper, we present a methodology that combines both white-box and black-box testing, in order to improve testing quality for a given class of embedded systems. The goal of this methodology is generation of test cases for the new functional ...
Reducing the cost of model-based testing through test case diversity
ICTSS'10: Proceedings of the 22nd IFIP WG 6.1 international conference on Testing software and systems

Model-based testing (MBT) suffers from two main problems which in many real world systems make MBT impractical: scalability and automatic oracle generation. When no automated oracle is available, or when testing must be performed on actual hardware or a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

GECCO '17: Proceedings of the Genetic and Evolutionary Computation Conference

July 2017

1427 pages

ISBN:9781450349208

DOI:10.1145/3071178

General Chair:
Peter A. N. Bosman
Centrum Wiskunde & Informatica (CWI)

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGEVO: ACM Special Interest Group on Genetic and Evolutionary Computation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 July 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

GECCO '17

Sponsor:

SIGEVO

GECCO '17: Genetic and Evolutionary Computation Conference

July 15 - 19, 2017

Berlin, Germany

Acceptance Rates

GECCO '17 Paper Acceptance Rate 178 of 462 submissions, 39%;

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
360
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)0

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ugarte MValle PIllarramendi MArrieta A(2025)Enhancing multi-objective test case selection through the mutation operatorAutomated Software Engineering10.1007/s10515-025-00489-632:1Online publication date: 30-Jan-2025
https://doi.org/10.1007/s10515-025-00489-6
Yao QZhang YLi YLiu FYang S(2024)Multimodal Multi-Objective Test Data Generation Method based on Particle Swarm Optimization2024 IEEE 24th International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS62785.2024.00016(61-71)Online publication date: 1-Jul-2024
https://doi.org/10.1109/QRS62785.2024.00016
Arrieta AValle PAgirre JSagardui G(2023)Some Seeds Are Strong: Seeding Strategies for Search-based Test Case SelectionACM Transactions on Software Engineering and Methodology10.1145/353218232:1(1-47)Online publication date: 13-Feb-2023
https://dl.acm.org/doi/10.1145/3532182
Ling XMenzies T(2023)What Not to Test (For Cyber-Physical Systems)IEEE Transactions on Software Engineering10.1109/TSE.2023.327230949:7(3811-3826)Online publication date: Jul-2023
https://doi.org/10.1109/TSE.2023.3272309
Krupalija ECogo EBećirović ŠPrazina IPozderac DBešić I(2023)Optimization of the test case minimization algorithm based on forward-propagation in cause-effect graphs2023 22nd International Symposium INFOTEH-JAHORINA (INFOTEH)10.1109/INFOTEH57020.2023.10094154(1-6)Online publication date: 15-Mar-2023
https://doi.org/10.1109/INFOTEH57020.2023.10094154
Wuersching RElsner DLeinen FPretschner AGrueneissl GNeumeyr TVosseler T(2023)Severity-Aware Prioritization of System-Level Regression Tests in Automotive Software2023 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST57152.2023.00044(398-409)Online publication date: Apr-2023
https://doi.org/10.1109/ICST57152.2023.00044
Arrieta AIllarramendi M(2023)A Novel Mutation Operator for Search-Based Test Case SelectionSearch-Based Software Engineering10.1007/978-3-031-48796-5_6(84-98)Online publication date: 4-Dec-2023
https://doi.org/10.1007/978-3-031-48796-5_6
Arrieta AWagner M(2022)Is the revisited hypervolume an appropriate quality indicator to evaluate multi-objective test case selection algorithms?Proceedings of the Genetic and Evolutionary Computation Conference10.1145/3512290.3528717(1317-1326)Online publication date: 8-Jul-2022
https://dl.acm.org/doi/10.1145/3512290.3528717
Rosenbauer LStein AHähner J(2022)A Germinal Center Artificial Immune System for Black Box Test SelectionSN Computer Science10.1007/s42979-022-01474-64:1Online publication date: 12-Nov-2022
https://doi.org/10.1007/s42979-022-01474-6
Gerber DRosenbauer LLindner PMaier JStein AHähner J(2022)Evolutionary Touch Filter Chain CalibrationSN Computer Science10.1007/s42979-022-01375-84:1Online publication date: 21-Oct-2022
https://doi.org/10.1007/s42979-022-01375-8
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten