Regression Testing

1
Software Evolution
• Change happens throughout the software development life
cycle.
• Before and after delivery
• Change can happen to every aspect of software
• Changes can affect unchanged areas
• break code, introduce new bugs
• uncover previous unknown bugs
• reintroduce old bugs

2
Regression Test

• Testing of a previously tested program following

modification to ensure that new defects have not been
introduced or uncovered in unchanged areas of the
software, as a result of the changes made.
• It should be performed whenever the software or its
environment is changed.
• It applies to testing at all levels.

3
Regression Testing Process

4
Regression Testing
• It is a type of software testing that intends to ensure that changes
(enhancements or defect fixes) to the software have not adversely
affected it.

• Rerunning of tests can be on both functional and non-functional tests.

• It is a type of change-related testing to detect whether defects have been

introduced or uncovered in unchanged areas of the software.

5
Regression Test
• Keep a test suite
• Use the test suite after every change
• Compare output with previous tests
• Understand all changes
• If new tests are needed, add to the test suite.

6
Regression Testing?

System Acceptance Regression

Test for ... Correctness, Usefulness, Accidental
completion satisfaction changes

Test by ... Development Test group with Development

test group users test group

Verification Validation Verification

7
Regression Test
• Yesterday it worked, today it doesn’t
• I was fixing X, and accidentally broke Y
• That bug was fixed, but now it’s back
• Tests must be re-run after any change
• Adding new features
• Changing, adapting software to new conditions
• Fixing other bugs
• Regression testing can be a major cost of software maintenance

8
Reasons
• Regression testing is conducted to make sure that fixing one thing has not
broken another thing.
• The need for Regression Testing could arise due to any of the changes
below:
• Defect fix
• New feature
• Change in an existing feature
• Code refactoring
• Change in technical design / architecture
• Change in configuration / environment (hardware, software, network)

9
Basic Problems of Regression Test
• Maintaining test suite
• If I change feature X, how many test cases must be revised because
they use feature X?
• Which test cases should be removed or replaced? Which test cases
should be added?
• Cost of re-testing
• Often proportional to product size, not change size
• Big problem if testing requires manual effort

10
Test Case Maintenance

• Some maintenance is inevitable

• If feature X has changed, test cases for feature X will require updating
• Some maintenance should be avoided
• Example: Trivial changes to user interface or file format should not invalidate
large numbers of test cases
• Test suites should be modular!
• Avoid unnecessary dependence
• Generating concrete test cases from test case specifications can help

11
Obsolete and Redundant

• Obsolete: A test case that is not longer valid

• Tests features that have been modified, substituted, or removed
• Should be removed from the test suite
• Redundant: A test case that does not differ significantly from others
• Unlikely to find a fault missed by similar test cases
• Has some cost in re-execution
• Has some (maybe more) cost in human effort to maintain
• May or may not be removed, depending on costs

12
Selecting and Prioritizing Regression Test Cases
• Should we re-run the whole regression test suite? If so, in what order?
• Maybe you don’t care. If you can re-rerun everything automatically over lunch break, do it.
• Sometimes you do care ...
• Selection matters when
• Test cases are expensive to execute
• Prioritization matters when
• A very large test suite cannot be executed every day

13
Select All Test Cases

1.Select All Test Cases

 We want to run all test cases for any change in the program.
 This is the safest technique, without any risk.
 A program may fail many times and every time we will execute the entire
test suite.
 This technique is practical only when the size of the test suite is small.
 For any reasonable or large sized test suite, it becomes impractical to
execute all test cases
14
2.Select Test Cases Randomly
 We may select test cases randomly to reduce the size of the test suite.
 We decide how many test cases are required to be selected depending upon
time and available resources.
 When we decide the number, the same number of test cases is selected
randomly. If the number is large, we may get a good number of test cases
for execution and testing may be of some use.

15
3. Select Modification Traversing Test
Cases

 We select only those test cases that execute the modified portion of the program
and the portion which is affected by the modification(s).
 We want to select all those test cases that reveal faults in the modified program.
These test cases are known as fault revealing test cases.
 Select those test cases that reveal the difference in the output of the original
program and the modified program. These test cases are known as modification
revealing test cases.
 These test cases target that portion of the source code which makes the output of
the original program and the modified program differs.

16
 Test Case Prioritization Methods

Version specific test case prioritization – Prioritize the test cases in T, when P is
modified to P’, with the knowledge of the changes that have been made in P. We consider the
version specific regression test selection algorithm whose objective is to execute the modified
lines of code with maximum number of test cases.

General Test Case Prioritization – For a given program, P, and test suite T, we prioritize
the test cases in T that will be useful over a succession of subsequent modified version of P
without any knowledge of modified version.

17
•The simplest priority category scheme is to assign a priority code to every test case.
•The priority code may be based on the assumption that “test case of priority code 1 is more important than
test case of priority code 2.”
• We may have priority codes as follows:
•Priority code 1 : Essential test case
•Priority code 2 : Important test case
• Priority code 3 : Execute, if time permits
•Priority code 4 : Not important test case
•Priority code 5 : Redundant test case
There may be other ways for assigning priorities based on customer requirements or market
conditions like:
•Priority code 1 : Important for the customer
•Priority code 2 : Required to increase customer satisfaction
•Priority code 3 : Help to increase market share of the product

18
Code-Based Regression Test Selection

• Observation: A test case can’t find a fault in code

it doesn’t execute
• In a large system, many parts of the code are
untouched by many test cases
• So: Only execute test cases that execute changed
or new code

19
Specification-Based Regression Test Selection

• Like code-based regression test case selection

• Pick test cases that test new and changed functionality
• Difference: No guarantee of independence
• A test case that isn’t “for” changed or added feature X might find a bug
in feature X anyway
• Typical approach: Specification-based prioritization
• Execute all test cases, but start with those that related to changed and
added features

20
Prioritized Rotating Selection
• Basic idea:
• Execute all test cases, eventually
• Execute some sooner than others
• Possible priority schemes:
• Round Robin:
• Priority to least-recently-run test cases
• Track record:
• Priority to test cases that have detected faults before
• Structural:
• Priority for executing elements that have not been recently executed

21
Risk Analysis
•Risk is a problem that may cause some loss or threaten the success of the project, but, which has not happened yet.
•Risk is defined as the “probability of occurrence of an undesirable event and the impact of
occurrence of that event.”
•Risks may delay and over-budget a project. Risky projects may also not meet specified quality levels.
Hence, there are two things associated with risk as given below:
(i) Probability of occurrence of a problem (i.e. an event)
(ii) Impact of that problem
Risk analysis is a process of identifying the potential problems and then assigning a
‘probability of occurrence of the problem’ value and ‘impact of that problem’ value for each
identified problem.
Both of these values are assigned on a scale of 1 (low) to 10 (high).
A factor ‘risk exposure’ is calculated for every problem which is the product of ‘probability of
occurrence of the problem’ value and ‘impact of that problem’ value.
•The risks may be ranked on the basis of its risk exposure.
•A risk analysis table may be prepared .
•These values may be calculated on the basis of historical data, past experience, intuition and
criticality of the problem.
•Here, the scale of 1 to 10 is used for assigning values to both the components of the risk exposure.
22
23
Risk Matrix
Risk matrix is used to capture:
•Identified problems
•Estimate their probability of occurrence with impact and rank the risks based on this information.
•The risk matrix has four quadrants.
•Each quadrant represents a priority category.

24
The priority category is defined as:
Priority category 1 (PC-1) = High probability value and high impact value
Priority category 2 (PC-2) = High probability value and low impact value
Priority category 3 (PC-3) = Low probability value and high impact value
Priority category 4 (PC-4) = Low probability value and low impact value

25