research-article

An enhanced test case selection approach for model-based testing: an industrial case study

Authors:

Shaukat AliAuthors Info & Claims

FSE '10: Proceedings of the eighteenth ACM SIGSOFT international symposium on Foundations of software engineering

Pages 267 - 276

https://doi.org/10.1145/1882291.1882331

Published: 07 November 2010 Publication History

Abstract

In recent years, Model-Based Testing (MBT) has attracted an increasingly wide interest from industry and academia. MBT allows automatic generation of a large and comprehensive set of test cases from system models (e.g., state machines), which leads to the systematic testing of the system. However, even when using simple test strategies, applying MBT in large industrial systems often leads to generating large sets of test cases that cannot possibly be executed within time and cost constraints. In this situation, test case selection techniques are employed to select a subset from the entire test suite such that the selected subset conforms to available resources while maximizing fault detection. In this paper, we propose a new similarity-based selection technique for state machine-based test case selection, which includes a new similarity function using triggers and guards on transitions of state machines and a genetic algorithm-based selection algorithm. Applying this technique on an industrial case study, we show that our proposed approach is more effective in detecting real faults than existing alternatives. We also assess the overall benefits of model-based test case selection in our case study by comparing the fault detection rate of the selected subset with the maximum possible fault detection rate of the original test suite.

References

[1]

Utting, M. and Legeard, B., Practical Model-Based Testing: A Tools Approach, Morgan-Kaufmann, 2006.

Digital Library

[2]

Pender, T., UML Bible, Wiley, 2003.

Digital Library

[3]

Ali, S., Hemmati, H., Holt, N. E., Arisholm, E. and Briand, L., Model Transformations as a Strategy to Automate Model-Based Testing -- A Tool and Industrial Case Studies, Simula Research Laboratory, Technical Report(2010-01), 2010.

[4]

Binder, R. V., Testing Object-Oriented Systems: Models, Patterns, and Tools, Addison-Wesley Professional, 1999.

Digital Library

[5]

Mathur, A. P., Foundations of Software Testing, Addison-Wesley Professional, 2008.

Digital Library

[6]

Jones, J. A. and Harrold, M. J., Test-Suite Reduction and Prioritization for Modified Condition/Decision Coverage, IEEE Transactions on Software Engineering, 29(3), 2003, 195--209.

Digital Library

[7]

Cartaxo, E. G., Machado, P. D. L. and Neto, F. G. O., On the use of a similarity function for test case selection in the context of model-based testing, Software Testing, Verification and Reliability, Published Online: 22 Jul 2009.

Digital Library

[8]

Goldberg, D. E., Genetic Algorithms in Search, Optimization, and Machine Learning, Addison-Wesley Professional, 2001.

[9]

Ledru, Y., Petrenko, A. and Boroday, S., Using String Distances for Test Case Prioritisation, In Proceedings of the 24th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2009, 510--514.

Digital Library

[10]

Rothermel, G., Harrold, M. J., Ronne, J. v. and Hong, C., Empirical studies of test-suite reduction, Software Testing, Verification and Reliability, 12(4), 2002, 219--249.

[11]

Elbaum, S. G., Malishevsky, A. G. and Rothermel, G., Test Case Prioritization: A Family of Empirical Studies, IEEE Transactions on Software Engineering, 28(2), 2002, 159--182.

Digital Library

[12]

Li, Z., Harman, M. and Hierons, R. M., Search Algorithms for Regression Test Case Prioritization, IEEE Transactions on Software Engineering, 33(4), 2007, 225--237.

Digital Library

[13]

Yoo, S., Harman, M., Tonella, P. and Susi, A., Clustering test cases to achieve effective and scalable prioritisation incorporating expert knowledge, In Proceedings of the 18th international symposium on Software testing and analysis, 2009, 201--212.

Digital Library

[14]

Orso, A., Do, H., Rothermel, G., Harrold, M. J. and Rosenblum, D. S., Using component metadata to regression test component-based software, Software Testing, Verification and Reliability, 17(2), 2007, 61--94.

Digital Library

[15]

McMaster, S. and Memon, A., Call-Stack Coverage for GUI Test Suite Reduction, IEEE Transactions on Software Engineering, 34(1), 2008, 99--115.

Digital Library

[16]

Chen, Y., Probert, R. L. and Ural, H., Regression test suite reduction based on SDL models of system requirements, Journal of Software Maintenance and Evolution: Research and Practice, 21(6), 2009, 379--405.

Digital Library

[17]

Farooq, U. and Lam, C. P., A Max--Min Multiobjective Technique to Optimize Model Based Test Suite, In Proceedings of the 10th ACIS International Conference on Software Engineering, Artificial Intelligences, Networking and Parallel/Distributed Computing, 2009, 569--574.

Digital Library

[18]

Farooq, U. and Lam, C. P., Evolving the Quality of a Model Based Test Suite, In Proceedings of the Proceedings of the IEEE International Conference on Software Testing, Verification, and Validation Workshops, 2009, 141--149

Digital Library

[19]

Ma, X. Y., Sheng, B. K. and Ye, C. Q., Test-Suite Reduction Using Genetic Algorithm, Advanced Parallel Processing Technologies, Springer Berlin / Heidelberg, 3756, 2005.

Digital Library

[20]

Chen, T. Y. and Lau, M. F., A simulation study on some heuristics for test suite reduction, Information and Software Technology, 40(13), 1998, 777--787.

[21]

Leon, D. and Podgurski, A., A Comparison of Coverage-Based and Distribution-Based Techniques for Filtering and Prioritizing Test Cases, In Proceedings of the IEEE International Symposium on Software Reliability Engineering, 2003, 442--456.

Digital Library

[22]

Harman, M., The Current State and Future of Search Based Software Engineering, In Proceedings of the Future of Software Engineering, 2007, IEEE Computer Society, 342--357.

Digital Library

[23]

Simão, A. d. S., Mello, R. F. d. and Senger, L. J., A Technique to Reduce the Test Case Suites for Regression Testing Based on a Self-Organizing Neural Network Architecture, In Proceedings of the COMPSAC, 2006, 93--96.

Digital Library

[24]

Jiang, B., Zhang, Z., Chan, W. K. and Tse, T. H., Adaptive random test case prioritization, In Proceedings of the IEEE/ACM International Conference on Automated Software Engineering (ASE 2009), 2009, 233--244.

Digital Library

[25]

Masri, W., Podgurski, A. and Leon, D., An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows, IEEE Transactions on Software Engineering, 33(7), 2007.

Digital Library

[26]

Ramanathan, M. K., Koyutürk, M., Grama, A. and Jagannathan, S., PHALANX: a graph--theoretic framework for test case prioritization. In Proceedings of the 23rd Annual ACM Symposium on Applied Computing, 2008, 667--673.

Digital Library

[27]

http://www.levenshtein.net/

[28]

Whitley, D., The genitor algorithm and selective pressure: Why rank-based allocation of reproductive trials is best, In Proceedings of the Third International Conference on Genetic Algorithms (ICGA--89), 1989, 116--121.

Digital Library

[29]

Haupt, R. L. and Haupt, S. E., Practical Genetic Algorithms, Wiley-Interscience, 1998.

Digital Library

[30]

Gusfield, D., Algorithms on Strings, Trees and Sequences: Computer Science and Computational Biology, Cambridge University Press, 1997.

Digital Library

[31]

Ali, S., Briand, L. C., Hemmati, H. and Panesar-Walawege, R. K., A Systematic Review of the Application and Empirical Investigation of Search-based Test-Case Generation, Accepted for publication in IEEE Transactions on Software Engineering, Special issue on Search-Based Software Engineering (SBSE), 2009.

Digital Library

Cited By

Shu THe ZYin XDing ZZhou M(2024)Model-based diversity-driven learn-to-rank test case prioritizationExpert Systems with Applications10.1016/j.eswa.2024.124768255(124768)Online publication date: Dec-2024
https://doi.org/10.1016/j.eswa.2024.124768
Querejeta MJee ELiu LValle PArrieta ARezabal M(2023)Search-based Test Case Selection for PLC Systems using Functional Block Diagram Programs2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE59848.2023.00040(228-239)Online publication date: 9-Oct-2023
https://doi.org/10.1109/ISSRE59848.2023.00040
Perez FZiadi TCetina C(2022)Utilizing Automatic Query Reformulations as Genetic Operations to Improve Feature Location in Software ModelsIEEE Transactions on Software Engineering10.1109/TSE.2020.300052048:2(713-731)Online publication date: 1-Feb-2022
https://doi.org/10.1109/TSE.2020.3000520
Show More Cited By

Index Terms

An enhanced test case selection approach for model-based testing: an industrial case study
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Achieving scalable model-based testing through test case diversity

The increase in size and complexity of modern software systems requires scalable, systematic, and automated testing approaches. Model-based testing (MBT), as a systematic and automated test case generation technique, is being successfully applied to ...
Reducing the cost of model-based testing through test case diversity
ICTSS'10: Proceedings of the 22nd IFIP WG 6.1 international conference on Testing software and systems

Model-based testing (MBT) suffers from two main problems which in many real world systems make MBT impractical: scalability and automatic oracle generation. When no automated oracle is available, or when testing must be performed on actual hardware or a ...
On the use of a similarity function for test case selection in the context of model-based testing

Test case selection in model-based testing is discussed focusing on the use of a similarity function. Automatically generated test suites usually have redundant test cases. The reason is that test generation algorithms are usually based on structural ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

FSE '10: Proceedings of the eighteenth ACM SIGSOFT international symposium on Foundations of software engineering

November 2010

302 pages

ISBN:9781605587912

DOI:10.1145/1882291

General Chair:
Gruia-Catalin Roman
Washington University in St. Louis, USA
,
Program Chair:
André van der Hoek
University of California, Irvine, USA

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 November 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGSOFT/FSE'10

Sponsor:

SIGSOFT

SIGSOFT/FSE'10: 18th ACM SIGSOFT Symposium on the Foundations of Software Engineering

November 7 - 11, 2010

New Mexico, Santa Fe, USA

Acceptance Rates

Overall Acceptance Rate 17 of 128 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

31
Total Citations
View Citations
880
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)4

Reflects downloads up to 03 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Shu THe ZYin XDing ZZhou M(2024)Model-based diversity-driven learn-to-rank test case prioritizationExpert Systems with Applications10.1016/j.eswa.2024.124768255(124768)Online publication date: Dec-2024
https://doi.org/10.1016/j.eswa.2024.124768
Querejeta MJee ELiu LValle PArrieta ARezabal M(2023)Search-based Test Case Selection for PLC Systems using Functional Block Diagram Programs2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE59848.2023.00040(228-239)Online publication date: 9-Oct-2023
https://doi.org/10.1109/ISSRE59848.2023.00040
Perez FZiadi TCetina C(2022)Utilizing Automatic Query Reformulations as Genetic Operations to Improve Feature Location in Software ModelsIEEE Transactions on Software Engineering10.1109/TSE.2020.300052048:2(713-731)Online publication date: 1-Feb-2022
https://doi.org/10.1109/TSE.2020.3000520
Singhal SJatana NSuri BMisra SFernandez-Sanz L(2021)Systematic Literature Review on Test Case Selection and Prioritization: A Tertiary StudyApplied Sciences10.3390/app11241212111:24(12121)Online publication date: 20-Dec-2021
https://doi.org/10.3390/app112412121
Feng SFeng YYu CZhang YLiu H(2021)Testing Scenario Library Generation for Connected and Automated Vehicles, Part I: MethodologyIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2020.297221122:3(1573-1582)Online publication date: Mar-2021
https://doi.org/10.1109/TITS.2020.2972211
Silva LVilain PCavalcante EDantas FBatista TPinto G(2020)LCCSSProceedings of the 14th Brazilian Symposium on Software Components, Architectures, and Reuse10.1145/3425269.3425283(91-100)Online publication date: 19-Oct-2020
https://dl.acm.org/doi/10.1145/3425269.3425283
Aman HAmasaki SYokogawa TKawahara M(2020)A Comparative Study of Vectorization-Based Static Test Case Prioritization Methods2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)10.1109/SEAA51224.2020.00023(80-88)Online publication date: Aug-2020
https://doi.org/10.1109/SEAA51224.2020.00023
Hajri IGoknil APastore FBriand L(2020)Automating system test case classification and prioritization for use case-driven testing in product linesEmpirical Software Engineering10.1007/s10664-020-09853-4Online publication date: 18-Aug-2020
https://doi.org/10.1007/s10664-020-09853-4
Alkhazi BAbid CKessentini MLeroy DWimmer M(2020)Multi-criteria test cases selection for model transformationsAutomated Software Engineering10.1007/s10515-020-00271-wOnline publication date: 12-Apr-2020
https://doi.org/10.1007/s10515-020-00271-w
(2019)Mutation-based genetic algorithm for efficiency optimisation of unit testingInternational Journal of Advanced Intelligence Paradigms10.5555/3324436.332444012:3-4(254-265)Online publication date: 1-Jan-2019
https://dl.acm.org/doi/10.5555/3324436.3324440
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents