An empirical study of regression test selection techniques

Reviewer: Mauro Pezze

Regression testing, i.e., the process of validating modified software, is an expensive and industrially relevant process. One of the main problems in regression testing is the selection of a suitable subset of test cases to be executed on the new version under test. Many solutions have been proposed and independently evaluated, but the choice of a suitable method for a specific application domain has not been thoroughly investigated yet. This paper proposes a set of parameters for comparing different techniques for selecting regression test suites, describes an experimental framework for their evaluation, and proposes a first evaluation of the main classes of regression testing techniques. Techniques are evaluated by comparing effectiveness, i.e., ability to reveal defects) versus costs. The experimental framework allows evaluating effectiveness and costs of different techniques. The paper evaluates: minimization techniques, that attempt to select a minimal set of test cases, dataflow techniques, that select test cases that exercise data interactions affected by modifications, safe techniques, that guarantee that no faults that could be revealed can go undetected, random techniques, that randomly select a subset of tests, and retest-all techniques, that require all tests to be executed. Some results confirm intuitive expectations, providing experimental data, e.g., they confirm that minimization techniques have the lowest fault detection effectiveness. Some others are less intuitive and thus even more valuable. For example, the experiments show that test reduction depends dramatically on the size of the program (From 5% for small programs up to 99% for large programs) This paper can be of interest both for researchers working in software testing and experimental software engineering, and for practitioners.