Towards Model-Driven Test Case Concretization for End-to-end Combinatorial Testing

Combinatorial testing can be used to automatically generate relevant sets of combinations of abstract test data for a System Under Test (SUT). It requires defining a combinatorial model with possible abstract values for the SUT input parameters, from which relevant abstract test cases can be generated to reach a chosen coverage criterion. However, concretizing abstract test cases into concrete test cases, and writing corresponding test scripts, is a tedious and error-prone manual process. With a focus on Java unit testing, we present in this paper a first end-to-end approach where combinatorial testing is supplemented with a model-driven concretization step for abstract test cases. To produce concrete test cases out of abstract test cases, the process requires a context-specific data generator provided by the test engineer, which can be implemented using constraint solving techniques. A code generator is used to produce the working JUnit test script for each concrete test case. The approach is implemented and integrated with the PICT combinatorial testing tool, the Choco-Solver Java library for data generation, and the Eclipse Modeling Framework (EMF) for model management. While the approach is currently limited to primitive data types, an initial evaluation on five Java methods shows that the testing effort can be greatly reduced when the combinatorial complexity is high.