research-article

Test suite generation for boolean conditions with equivalence class partitioning

Author:

Sylvain HalléAuthors Info & Claims

FormaliSE '22: Proceedings of the IEEE/ACM 10th International Conference on Formal Methods in Software Engineering

Pages 23 - 33

https://doi.org/10.1145/3524482.3527659

Published: 21 July 2022 Publication History

Abstract

Boolean test input generation is the process of finding sets of values for variables of a logical expression such that a given coverage criterion is achieved. This paper presents a formal framework in which evaluating an expression produces a tree structure, and where a coverage criterion is expressed as equivalence classes induced by a particular transformation over these trees. It then defines many well-known coverage criteria as particular cases of this framework. The paper describes an algorithm to generate test suites by a reduction through a graph problem; this algorithm works in the same way regardless of the criterion considered. An experimental evaluation of this technique shows that it produces test suites that are in many cases smaller than existing tools.

References

[1]

P. Ammann and J. Offutt. Introduction to Software Testing, 2nd Ed. Cambridge University Press, 2016.

[2]

Z. Awedikian, K. Ayari, and G. Antoniol. MC/DC automatic test input data generation. In F. Rothlauf, editor, GECCO, pages 1657--1664. ACM, 2009.

Digital Library

[3]

R. Bloem, K. Greimel, R. Koenighofer, and F. Roeck. Model-based MCDC testing of complex decisions for the Java Card applet firewall. In VALID, pages 1--6. IARIA, 2013.

[4]

J. Chang and C. Huang. A study of enhanced MC/DC coverage criterion for software testing. In COMPSAC, pages 457--464. IEEE Comp. Soc., 2007.

Digital Library

[5]

T. Chen and M. Lau. An empirical study on the effectiveness of the greedy MUTP strategy. In SEEP, pages 338--344. IEEE, 1998.

[6]

T. Chen, M. Lau, and Y. Yu. MUMCUT: a fault-based strategy for testing Boolean specifications. In APSEC, pages 606--613, Takamatsu, Japan, 1999. IEEE Comp. Soc.

[7]

T. Y. Chen and M. F. Lau. Dividing strategies for the optimization of a test suite. Inf. Process. Lett., 60(3):135--141, 1996.

Digital Library

[8]

J. J. Chilenski. An investigation of three forms of the modified condition decision coverage (MCDC) criterion. Technical Report DOT/FAA/AR-01/18, Office of Aviation Research, Federal Aviation Administration, April 2001.

[9]

J. J. Chilenski and S. P. Miller. Applicability of modified condition/decision coverage to software testing. Softw. Eng. J., 9(5):193--200, 1994.

[10]

J. Flum and M. Grohe. Parameterized Complexity Theory. Springer, 2006.

[11]

S. Hallé, E. L. Chance, and S. Gaboury. Graph methods for generating test cases with universal and existential constraints. In K. El-Fakih, G. D. Barlas, and N. Yevtushenko, editors, ICTSS, volume 9447 of LNCS, pages 55--70. Springer, 2015.

[12]

S. Hallé, R. Khoury, and M. Awesso. Streamlining the inclusion of computer experiments in a research paper. Computer, 51(11):78--89, 2018.

Digital Library

[13]

S. Hallé. Boolean condition coverage with evaluation trees (LabPal bundle), 2022.

[14]

E. Halperin. Improved approximation algorithms for the vertex cover problem in graphs and hypergraphs. SIAM J. Comput., 31(5):1608--1623, 2002.

Digital Library

[15]

L. Hu, W. E. Wong, D. R. Kuhn, and R. Kacker. MCDC-Star: A white-box based automated test generation for high MC/DC coverage. In DSA, pages 102--112. IEEE, 2018.

[16]

G. K. Kaminski and P. Ammann. Reducing logic test set size while preserving fault detection. Softw. Test. Verification Reliab., 21(3):155--193, 2011.

Digital Library

[17]

G. K. Kaminski, G. Williams, and P. Ammann. Reconciling perspectives of software logic testing. Softw. Test. Verification Reliab., 18(3):149--188, 2008.

Digital Library

[18]

S. Kangoye, A. Todoskoff, and M. Barreau. Practical methods for automatic MC/DC test case generation of Boolean expressions. In Proc. AUTOTESTCON, pages 203--212. IEEE, 2015.

Digital Library

[19]

K. Kapoor and J. P. Bowen. A formal analysis of MCDC and RCDC test criteria. Softw. Test. Verification Reliab., 15(1):21--40, 2005.

Digital Library

[20]

G. Karakostas. A better approximation ratio for the vertex cover problem. In L. Caires, G. F. Italiano, L. Monteiro, C. Palamidessi, and M. Yung, editors, ICALP, volume 3580 of LNCS, pages 1043--1050. Springer, 2005.

[21]

T. Kitamura, Q. Maissonneuve, E. Choi, C. Artho, and A. Gargantini. Optimal test suite generation for modified condition decision coverage using SAT solving. In B. Gallina, A. Skavhaug, and F. Bitsch, editors, SAFECOMP, volume 11093 of LNCS, pages 123--138. Springer, 2018.

[22]

D. Kuhn, D. Wallace, and A. Gallo. Software fault interactions and implications for software testing. IEEE Trans. on Softw. Eng., 30(6):418--421, June 2004.

Digital Library

[23]

R. Kuhn. Automated combinatorial testing for software. https://csrc.nist.gov/Projects/automated-combinatorial-testing-for-software/downloadable-tools, Accessed February 27th, 2021.

[24]

M. F. Lau and Y. Yu. On comparing testing criteria for logical decisions. In F. Kordon and Y. Kermarrec, editors, Ada-Europe, volume 5570 of LNCS, pages 44--58. Springer, 2009.

[25]

Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, and J. Lawrence. Ipog: A general strategy for t-way software testing. In ECBS, pages 549--556. IEEE, 2007.

Digital Library

[26]

D. Lopez-Escogido, J. Torres-Jimenez, E. Rodriguez-Tello, and N. Rangel-Valdez. Strength two covering arrays construction using a SAT representation. In MICAI, pages 44--53. Springer, 2008.

Digital Library

[27]

E. J. McCluskey, Jr. Minimization of Boolean functions. Bell System Tech. J., 35(6):1417--1444, November 1956.

[28]

K. J. Nurmela. Upper bounds for covering arrays by tabu search. Discrete applied mathematics, 138(1--2):143--152, 2004.

[29]

T. K. Paul and M. F. Lau. A systematic literature review on modified condition and decision coverage. In Y. Cho, S. Y. Shin, S. Kim, C. Hung, and J. Hong, editors, SAC, pages 1301--1308. ACM, 2014.

[30]

R. K. Singh, P. Chandra, and Y. Singh. An evaluation of boolean expression testing techniques. ACM SIGSOFT Softw. Eng. Notes, 31(5):1--6, 2006.

Digital Library

[31]

C. Sun, Y. Dong, R. Lai, K. Sim, and T. Chen. Analyzing and extending MUMCUT for fault-based testing of general boolean expressions. In CIT, pages 184--184. IEEE, 2006.

Digital Library

[32]

C.-a. Sun, K. Y. Sim, T. Tse, and T. Chen. An Empirical Evaluation and Analysis of the Fault-Detection Capability of MUMCUT for General Boolean Expressions. In ICS, pages 926--932, 2004.

[33]

J. Torres-Jiménez and E. Rodriguez-Tello. New bounds for binary covering arrays using simulated annealing. Inf. Sci., 185(1):137--152, 2012.

Digital Library

[34]

S. Vilkomir and D. Anderson. Relationship between pair-wise and MC/DC testing: Initial experimental results. In ICST Workshops. IEEE, 2015.

[35]

S. A. Vilkomir and J. P. Bowen. From MC/DC to RC/DC: formalization and analysis of control-flow testing criteria. Formal Aspects Comput., 18(1):42--62, 2006.

Digital Library

[36]

E. J. Weyuker, T. Goradia, and A. Singh. Automatically generating test data from a boolean specification. IEEE Trans. Software Eng., 20(5):353--363, 1994.

Digital Library

[37]

Y. Yu and M. F. Lau. A comparison of MC/DC, MUMCUT and several other coverage criteria for logical decisions. J. Syst. Softw., 79(5):577--590, 2006.

Digital Library

[38]

Y. Yu, M. F. Lau, and T. Y. Chen. Automatic generation of test cases from Boolean specifications using the MUMCUT strategy. J. Syst. Softw., 79(6):820--840, 2006.

Digital Library

[39]

Yu Lei and K. C. Tai. In-parameter-order: a test generation strategy for pairwise testing. In HASE, pages 254--261, Nov 1998.

Digital Library

Cited By

Fei ZGe JLi CWang TLi YZhang HHuang LLuo B(2024)Patch Correctness Assessment: A SurveyACM Transactions on Software Engineering and Methodology10.1145/370297234:2(1-50)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3702972
Basile DTer Beek MGöttmann HLochau M(2023)Mutant Equivalence as Monotonicity in Parametric Timed Games2023 IEEE/ACM 11th International Conference on Formal Methods in Software Engineering (FormaliSE)10.1109/FormaliSE58978.2023.00014(55-65)Online publication date: May-2023
https://doi.org/10.1109/FormaliSE58978.2023.00014

Index Terms

Test suite generation for boolean conditions with equivalence class partitioning
1. Security and privacy
  1. Formal methods and theory of security
    1. Logic and verification
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Neuron Semantic-Guided Test Generation for Deep Neural Networks Fuzzing
In recent years, significant progress has been made in testing methods for deep neural networks (DNNs) to ensure their correctness and robustness. Coverage-guided criteria, such as neuron-wise, layer-wise, and path-/trace-wise, have been proposed for DNN ...
A Novel Approach of Equivalence Class Partitioning for Numerical Input

Software testing is an indispensible phase of software development that supports not only productivity but also quality assurance from small to large scale software products. Software Systems essentially need an effective way to develop a strategy for ...
A detailed investigation of the effectiveness of whole test suite generation

A common application of search-based software testing is to generate test cases for all goals defined by a coverage criterion (e.g., lines, branches, mutants). Rather than generating one test case at a time for each of these goals individually, whole ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

FormaliSE '22: Proceedings of the IEEE/ACM 10th International Conference on Formal Methods in Software Engineering

May 2022

137 pages

ISBN:9781450392877

DOI:10.1145/3524482

General Chairs:
Stefania Gnesi
ISTI-CNR, Italy
,
Nico Plat
Thanos, The Netherlands
,
Program Chairs:
Arnd Hartmanns
University of Twente, The Netherlands
,
Ina Schaefer
Technische Universität Braunschweig, Germany

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 July 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Canada Research Chairs

Conference

FormaliSE '22

Sponsor:

SIGSOFT

FormaliSE '22: International Conference on Formal Methods in Software Engineering

May 18 - 22, 2022

Pennsylvania, Pittsburgh

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
69
Total Downloads

Downloads (Last 12 months)21
Downloads (Last 6 weeks)0

Reflects downloads up to 23 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Fei ZGe JLi CWang TLi YZhang HHuang LLuo B(2024)Patch Correctness Assessment: A SurveyACM Transactions on Software Engineering and Methodology10.1145/370297234:2(1-50)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3702972
Basile DTer Beek MGöttmann HLochau M(2023)Mutant Equivalence as Monotonicity in Parametric Timed Games2023 IEEE/ACM 11th International Conference on Formal Methods in Software Engineering (FormaliSE)10.1109/FormaliSE58978.2023.00014(55-65)Online publication date: May-2023
https://doi.org/10.1109/FormaliSE58978.2023.00014

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents