04/03/2017

Dhirubhai Ambani Institute of Information and Communication

Technology (DA-IICT), Gandhinagar

Software Testing
&
Applications, Tools and Technologies

Saurabh Tiwari, PhD

DA-IICT Gandhinagar

A Self Assessment Test

Write a set of test cases specific set of data to properly test a

relatively Simple program.

Create a set of test data for the program .

The program reads three integer values from an input dialog. The three
value represent the lengths of the sides of a triangle. The program
displays a message that states whether the triangle is scalene,
isosceles, or equilateral.

Test Action and Data Expected Result

2, 5, 10 ??

1
 04/03/2017

A Self Assessment Test

The function triangle takes three integer parameters that are

interpreted as the lengths of the sides of a triangle. It returns whether
the triangle is equilateral (three lengths equal), isosceles (two lengths
equal), scalene (no lengths equal), or invalid (impossible lengths).

Solution?
Tester Action and Data Expected Result
Erroneous input partition
a, b, or c a non-number an error message

Scalene triangle partition

2, 3, 4 scalene
1, 2, 3 an error message
Isosceles triangle partition
2, 2, 1 isosceles
2, 2, 0 an error message
Equilateral triangle partition
1, 1, 1 equilateral
1, 1, -1 an error message
User interface tests
Try a leading space for a, b, c an error message
Many more

2
 04/03/2017

Why Testing?

Uncover as many as errors as possible in a given timeline.

Demonstrate a given software product matching its

requirements specification.

Validate the quality of a software

Software testing is arguably the least understood part of the development

process. Through a four-phase approach, the author shows why
eliminating bugs is tricky and why testing is a constant trade-off
[Whittaker]

Testing Purpose

1. Avoid Redundancy
Source Code
Requirement/Design/Maintenance/Change
Management

2. Reducing Cost
Cost of detection, Cost of Prevention
Internal Failure (at the time of development or before
you release the software product)
External Failure (finding bugs after deploying the
software product)

3. Correctness CONFIDENCE

4. Quality Assurance

3
 04/03/2017

Trivial example

Try to move a file from folder A to another folder B

What are the possible scenarios?

Trying to move the file when it is open

You do not have the security rights to paste the file in folder B
Folder B is on a shared drive and storage capacity is full
Folder B already has a file with the same name

Taxonomy of Bugs

Verification?
Error?
Validation?
Defects ? Doesnt work as defined

Fault?
Failure?- Non functioning of system

Testing? Bugs? error in the system

Debugging?

Quality?

4
 04/03/2017

BUG

Bugs

The word bug, originated in engineering.

The term's application to computing has been attributed to the

pioneer programmer, Grace Hopper. In 1944, Hopper, who was then
a young Naval Reserve officer, went to work on the Mark I computer
at Harvard.

He later described an incident in which a technician is said to have

pulled an actual bug (a moth, in fact) from between two electrical
relays in the Mark II computer.

5
 04/03/2017

Error, Defects, Fault and Failure

Example:
You are driving a car and you are on road while on driving now
there is two way on the road
1. left--> Mumbai
2. right--> Delhi

Now you have to go to Delhi, it means you have to turn the

steering to the right, but by mistake you turn the steering to
the left, from that position that is called as "Error

and now Fault is there till you will not reach the Mumbai, but
when you reach Mumbai that is a final stage which is called
"Failure" because you had to reach Delhi but now you are in
Mumbai.
A mistake in coding is called Error, error found by tester is called
defect, defect accepted by development team then it is called bug, build
does not meet the requirements then it Is failure.

Coding: Error, Defects, Fault and Failure

The program is required to add two numbers
1 #include<stdio.h>
2
3 int main ()
4 { 5 + 3 should be 8, but the result is 2. There could
5 int value1, value2, ans; be various reasons as to why the program displays
6 the answer 2 instead of 8. For now we have
7 value1 = 5; detected a failure.
8 value2 = 3;
9
10 ans = value1 - value2;
11
12 printf("The addition of 5 + 3 = %d.", ans);
13
14 return 0;
15 }

As the failure has been detected a defect can be raised.

6
 04/03/2017

The program is required to add two numbers

1 #include<stdio.h>
2
3 int main () Now lets go back to the program and analyze what
4 { was the fault in the program.
5 int value1, value2, ans;
6 There is a - sign present instead of + sign. So
7 value1 = 5; the fault in the program is the - sign.
8 value2 = 3;
9
10 ans = value1 - value2; // Bug, Defect
11
12 printf("The addition of 5 + 3 = %d.", ans);
13
14 return 0;
15 }

Error is the mistake I made by typing - instead

of + sign.

Error, Defects, Fault and Failure

A tester does not necessarily have access to the code and may be just
testing the functionality of the program. In that case the tester will
realize the output is faulty and will raise a defect.

Error: A mistake made by a programmer

Example: Misunderstood the requirements.

Defect/fault/bug: Manifestation of an error in a program.

Failure: Manifestation of one or more faults

in the observed program behavior

7
 04/03/2017

Testing and Debugging

Testing: Finding inputs that cause the software to fail

Finding and localization of a defect
Intention behind is to find as many as defects possible

Debugging: The process of finding a fault given a failure

Fixing that defect
Intention is to remove those defects

...testing can be a very effective way to show the presence of bugs, but
is hopelessly inadequate for showing their absence. The only effective
way to raise the confidence level of a program significantly is to give a
convincing proof of its correctness.
Edsger Dijkstra

http://www.cs.utexas.edu/users/EWD/transcriptions/EWD03xx/EWD340.html

Who Tests the Software?

Developer
Independent Tester
Understands the system but, Must learn about the
will test "gently and, is driven system, but, will attempt to
by "delivery" break it and, is driven by
quality

8
 04/03/2017

Why Testing is necessary

Suppose you have a 15 input fields each one having 5

possible values.

How many combinations to be tested?

5^15 = 30517578125!!!

Why Testing is necessary

Why dont we test everything ?

System has 20 screens
Average 4 menus / screen
Average 3 options / menu
Average of 10 fields / screen
2 types of input per field
Around 100 possible values

Approximate total for exhaustive testing

20 x 4 x 3 x 10 x 2 x 100 = 480,000 tests
Test length = 1 sec then test duration = 17.7 days
Test length = 10 sec then test duration = 34 weeks
Test length = 1 min then test duration = 4 years
Test length = 10 mins then test duration = 40 years!

It is not a matter of time. But,time is money ( salary is taken by hour. So second

is valuable for software houses)

9
 04/03/2017

Testing Technique Hierarchies

Software Testing
Techniques

Execution-based Non-execution based

testing testing

Program-based Combined Specification- Inspections

testing Testing based testing

Ad-hoc Checklist Scenario-

based
Structure-based Fault-based Error-based
criterion criterion criterion

19

Testing Strategy Test Case Design

Testing Strategy

Black-box White-box
(Specification based) Static Analysis
(Code based)
Requirement
Design
Equivalence class Boundary Value Cause-Effect Implementation
Partitioning Analysis Graphing

Model-Based Data Flow Mutation Control Flow

Testing based based based
Statement coverage, Decision coverage, Condition
coverage, Decision-condition coverage, Multiple-condition
coverage

20

10
 04/03/2017

New : Test Coverage Criteria

A testers job is simple : Define a model of the software,

then find ways to cover it

Test Requirements : Specific things that must be satisfied or

covered during testing

Test Criterion : A collection of rules and a process that define

test requirements

Testing researchers have defined dozens of criteria, but

they are all really just a few criteria on four types of
structures

Criteria Based on Structures

Structures : Four ways to model software

1. Graphs

2. Logical Expressions
(not X or not Y) and A and B

A: {0, 1, >1}
3. Input Domain B: {600, 700, 800}
Characterization C: {swe, cs, isa, infs}

if (x > y)
z = x - y;
4. Syntactic Structures else
z = 2 * x;

11
 04/03/2017

Black-box Testing

Black-Box : Test to Specification

Inputs causing
anomalous
Input test data I behaviour
e

System

Outputs which reveal

the presence of
Output test results Oe defects

24

12
 04/03/2017

Equivalence Class Partitioning

Partition the program input domain into equivalence

classes (according to the specifications)

The rationale is that test of a representative value of

each class is equivalent to a test of any other value
of the same class.

Identify valid as well as invalid equivalence classes

One test case from each equivalence class

25

A Set of Guidelines

If an input condition specifies a range of values (e.g., the item count can
be from 1 to 999), identify one valid equivalence class (1<item count<999)
and two invalid classes (item count <1 and item count >999)

If an input condition specifies the number of values (e.g., one through six
owners can be listed for the automobile), identify one valid equivalence
class and two invalid equivalence classes (no owners and more than six
owners)

If an input condition specifies a set of input values and there is a reason to

believe that the program handles each differently (type of vehicle must be
BUS, TRUCK, TAXICAB, PASSENGER, or MOTORCYCLE), identify a valid
equivalence class for each and one invalid equivalence class (e.g.,
"TRAILER)

If an input condition specifies a must be situation such as first character

of the identifier must be a letter, identify one valid equivalence class (it is
a letter) and one invalid equivalence class (it is not a letter)
26

13
 04/03/2017

A Summary: Set of Guidelines

Input Condition Valid Eq. Classes Invalid Eq. Classes

Range of values (1-200) One valid (value two invalid
within the range)
Number N valid values One valid Two invalid (none,
more than N)
Set of input values One valid Eq. class One
each handled differently by for each value (e.g. any value not
the (total n) in valid input set)
program (e.g. A, B, C, (n))
Must be condition (e.g., id One (e.g., it is a One (e.g., it is not
name must begin with a letter) a letter)
letter)

27

Example

If there is any reason to believe that the program does not

handle elements in an equivalence class identically, split the
equivalence class into smaller equivalence classes.

Example:

Input condition 0 <= x <= Max

Valid Equivalence Classes 0 <= x <= Max
Invalid Equivalence Classes x < 0; x > Max

3 Test Cases

28

14
 04/03/2017

Example

Example of Equivalence class partitioning

A text field permits only numeric characters

Length must be 6-10 characters long

Partition according to the requirement should be like this:

29

Identifying Test cases

Assign a unique number to each equivalence class

Until all valid equivalence classes have been covered by test

cases, write a new test case covering as many of the uncovered
valid equivalence classes as possible.

Each invalid equivalence class cover by a separate test case.

30

15
 04/03/2017

Identifying Test cases

A individual test case could not cover invalid cases.

For example, if the specification is

enter book type (HARDCOVER, SOFTCOVER, or LOOSE) and

amount (1-999)

The test case, XYZ, 0, expressing two error conditions (invalid book
type and amount) will probably not exercise the check for the
amount.

(Program may say XYZ is UNKNOWN BOOK TYPE, and not bother
to examine the remainder of the input.

31

Question?

Consider that wordCount method takes a word w and a

filename f as input and returns the number of occurrences of
w in the text contained in the file named f. An exception is
raised if there is no file with name f. Using the partitioning
method, we obtain the following equivalence classes.

Equivalence Class W F
E1 Non-null Exists, not empty
E2 Non-null Does not exist
E3 Non-null Exists, empty
E4 Null Exists, not empty
E5 Null Does not exist
E6 Null Exists, empty

32

16
 04/03/2017

Question?

Consider an application that requires two integer inputs x and y. Each

of these inputs is expected to lie in the following ranges:
3 <= x <= 7 and 5 <= y <= 9

E1: x<3 E2: 3<=x<=7 E3: x>7 (y ignored)

Six equivalent Classes
E4: y<5 E5: 5<=y<=9 E6: y>9 (x ignored)

E1: x<3, y<5

E2: x<3, 5<=y<=9
E3: x<3, y>9
E4: 3<=x<=7, y<5 Nine equivalent Classes
E5: 3<=x<=7, 5<=y<=9 X x Y
E6: 3<=x<=7, y>9
E7: x>7, y<5
E8: x>7, 5<=y<=9
E9: x>7, y>9

33

Boundary-Value Analysis

Boundary value analysis is a test selection technique that targets

faults in applications at the boundaries of equivalence classes.

Generally combined with Equivalence Class Partitioning

Design test cases that exercise values that lie at the boundaries
of an input equivalence class.
Also identify output equivalence classes, and write test cases to
generate o/p at the boundaries of the output equivalence
classes.

Example: input condition 0 <= x <= max

Test for values : 0,1, x, max-1, max ( valid inputs)
-1, max+1 (invalid inputs)

34

17
 04/03/2017

3 11
4 7 10

Less than 4 Between 4 and 10 More than 10

Number of input values

9999 100000
10000 50000 99999

Less than 10000 Between 10000 and 99999 More than 99999

Input values

35

Example 1

Suppose you have very important tool at office, accepts valid User
Name and Password field to work on that tool, and accepts
minimum 8 characters and maximum 12 characters. Valid range 8-
12, Invalid range 7 or less than 7 and Invalid range 13 or more than
13.

Write Test Cases for Valid partition value, Invalid partition value and
exact boundary value.

36

18
 04/03/2017

Test Cases 1: Consider password length less than 8.

Test Cases 2: Consider password of length exactly 8.
Test Cases 3: Consider password of length between 9 and 11.
Test Cases 4: Consider password of length exactly 12.
Test Cases 5: Consider password of length more than 12.

37

Exercise??

Consider a program for determining the previous date. Its input is

triple of day, month and year with the following ranges 1 <= month
<= 12, 1 <= day <= 31, 1900 <= year <= 2015.The possible output
dates would be previous date or invalid date. Design the
equivalence class test cases?
Equivalence classes:-
1. year<1900
2. year>2015
3. 1900<=year<=2015
a. Month<1
b. Month>12
c. 1<=Month<=12
i. Day<1
ii. Month has 30 days a.)Day>30 b.)1<=Day<=30
iii. Month has 31 days a.)Day>31 b.)1<=Day<=31
iv. Month is February
1. Year is leap year a.) Day>29 b.)1<=Day<=29
2. Year is common year a.)Day>28 b.)1<=Day<=28

38

19
 04/03/2017

White-Box Testing

39

White-Box Testing

Testing based on analysis of internal logic (design, code,

etc.). (But expected results still come from requirements.)

Also know as structural testing.

White-box testing concerns techniques for designing tests; it

is not a level of testing.

White-box testing techniques apply primarily to lower levels

of testing (e.g., unit and component).

20
 04/03/2017

Control-flow-based Testing

A traditional form of white-box testing

Step 1: From the source, create a graph describing the flow of

control
Called the control flow graph
The graph is created (extracted from the source code)
manually or automatically

Step 2: Design test cases to cover certain elements of this

graph
Nodes, edges, paths

Control-flow-Graph

21
 04/03/2017

Elements of a CFG

Three kinds of nodes:

Statement nodes: represent single-entry-single-exit

sequences of statements
Predicate nodes: represent conditions for branching
Auxiliary nodes: (optional) for easier understanding (e.g.,
join points for IF, etc.)

Edges: represents possible flow of control

It is relatively easy to map standard constructs from

programming languages to elements of CFGs

IF-THEN, IF-THEN-ELSE, SWITCH

if (c) if (c) switch (c)

then then case 1:
// join point else case 2:
// join point // join point

..

22
 04/03/2017

Example

switch (position)
case CASHIER
if (empl_yrs > 5)
bonus := 1;
else
bonus := 0.7; .
case MANAGER .
.
bonus := 1.5;
if (retiring_soon) .
bonus := 1.2 * bonus
case
endswitch

Mapping for Loops

while (c) {

Note: other loops (e.g., FOR, DO-WHILE,) are mapped

similarly. Figure out how this is done.

23
 04/03/2017

Example

int f1(int x,int y){

1 while (x != y){
2 if (x>y) then
3 x=x-y;
4 else y=y-x;
5}
6 return x; }

Example Control Flow Graph

1
2
3 4
5
6

24
 04/03/2017

White-Box Test Case Design

Statement coverage
write enough test cases to execute every statement at least once

Decision coverage (Branch coverage)

write test cases to exercise the true and false outcomes of every
decision

Condition coverage (Predicate coverage)

write test cases such that each condition in a decision takes on
all possible outcomes at least once

may not always satisfy decision coverage

Example

void function eval (int A, int B, int X )

{
if ( A > 1) and ( B = 0 )
then X = X / A;
if ( A = 2 ) or ( X > 1)
then X = X + 1;
}

Statement coverage test cases:

1) A = 2, B = 0, X = ? ( X can be
assigned any value)

25
 04/03/2017

Example

void function eval (int A, int B, int X )

{
if ( A > 1) and ( B = 0 )
then X = X / A;
if ( A = 2 ) or ( X > 1)
then X = X + 1;
}

Decision coverage test cases:

1) A = 3 B = 0 X = 1 (acd)
2) A = 2, B = 1, X = ? (abe)

Example

Condition coverage test cases must

cover conditions

A > 1, A <=1, B = 0 , B!=0

A = 2, A!=2, X > 1, X <=1

Test Cases:
1) A = 1, B = 0, X = 3 (abe)
2) A =2, B = 1, X = 1 (abe)

Doesnt satisfy decision coverage

26
 04/03/2017

Exercise

Consider the following function which determines whether a given

code is a valid Canadian postal code or not. The Canadian postal
code is a seven-character uniformly structured, alphanumeric code in
the form ANA NAN where `A' represents an alphabetic character
and `N' represents a numeric character.

Exercise

[a] Statement Coverage; [b] Branch

Coverage; [c] Condition Coverage; [d]
Equivalence Class Partitioning

27
 04/03/2017

JUnit Testing Framework

Example: Old way vs. New way

int max(int a, int b) { @Test

if (a > b) { void testMax() {
return a; assertEquals(7, max(3, 7));
} else { assertEquals(3, max(3, -7));
return b;
} }
}
void testMax() {
int x = max(3, 7);
if (x != 7) {
System.out.println("max(3, 7) gives " + x);
}
x = max(3, -7);
if (x != 3) {
System.out.println("max(3, -7) gives " + x);
}
}
public static void main(String[] args) {
new MyClass().testMax();
}

28
 04/03/2017

Junit, In PICTURES

test suite test runner

another unit test
test case (for one method) A unit test tests the methods in a
another test case single class
A test case tests (insofar as
another unit test possible) a single method
another test case You can have multiple test cases
another test case
for a single method
another test case A test suite combines unit tests
The test fixture provides
software support for all this
unit test (for one class)
test case (for one method)
The test runner runs unit tests or
an entire test suite
another test case
Integration testing (testing that it
all works together) is not well
test fixture
supported by JUnit

Writing a JUnit test class, I

Start by importing these JUnit 4 classes:

import org.junit.*;
import static org.junit.Assert.*; // note static import

Declare your test class in the usual way

public class MyProgramTest {

Declare an instance of the class being tested

You can declare other variables, but dont give them initial
values here
public class MyProgramTest {
MyProgram program;
int someVariable;

29
 04/03/2017

Writing a JUnit test class, II

Define a method (or several methods) to be executed before
each test
Initialize your variables in this method, so that each test starts
with a fresh set of values
@Before
public void setUp() {
program = new MyProgram();
someVariable = 1000;
}

You can define one or more methods to be executed after each

test
Typically such methods release resources, such as files
Usually there is no need to bother with this method
@After
public void tearDown() {
}

A Simple Example

Suppose you have a class Arithmetic with methods int multiply(int x, int y), and
boolean isPositive(int x)

import org.junit.*;
import static org.junit.Assert.*;

public class ArithmeticTest {

@Test
public void testMultiply() {
assertEquals(4, Arithmetic.multiply(2, 2));
assertEquals(-15, Arithmetic.multiply(3, -5));
}

@Test
public void testIsPositive() {
assertTrue(Arithmetic.isPositive(5));
assertFalse(Arithmetic.isPositive(-5));
assertFalse(Arithmetic.isPositive(0));
}
}

30