Introduction to Testing as an Engineering Activity

Software systems are becoming more challenging to build. They are playing an increasingly
important role in society. Pressure for software professionals to focus on quality issues. Poor quality
software that can cause loss of life or property is no longer acceptable to society. Highest quality
attributes are reliability, correctness, usability, and the ability to meet all user requirements

Using an engineering approach to software development implies

 The development process is well understood;

 Projects are planned;
 Life cycle models are defined and adhered to;
 Standards are in place for product and process;
 Measurements are employed to evaluate product and process quality;
 Components are reuse
 Validation and verification processes play a key role in quality determination;
 Engineers have proper education, training, and certification

Testing as a Process

Process, in the software engineering domain, is the set of methods, practices, standards, documents,
activities, policies, and procedures that software engineers use to develop and maintain a software
system and its associated artifacts, such as project and test plans, design documents, code, and manuals.

Components of an engineered process.

Embedded within the software development process are several other processes including testing.
Testing itself is related to two other processes called verification and validation
.

Processes embedded in the software development process.

Validation is the process of evaluating a software system or component during, or at the end of, the
development cycle in order to determine whether it satisfies specified requirements

Verification is the process of evaluating a software system or component to determine whether the
products of a given development phase satisfy the conditions imposed at the start of that phase

Testing as a process has economic, technical and managerial aspects.

Economic aspects are related to the reality that resources and time are available to the testing group on a
limited basis.

The technical aspects of testing relate to the techniques, methods, measurements, and tools used to
insure that the software under test is as defect-free and reliable as possible for the conditions and
constraints under which it must operate

The managerial aspects means that an organizational policy for testing must be defined and documented.

TMM Levels

As in the case of the CMM, the TMM also follows what is called a staged architecture for process
improvement models. It contains stages or levels through which an organization passes as its testing
process evolves from one that is ad hoc and unmanaged to one that is managed, defined, measured, and
optimizable.

The maturity goals at each level of the TMM are

Level 1—Initial: (No maturity goals)

At TMM level 1, testing is a chaotic process; it is ill-defined, and not distinguished from debugging.

Level 2—Phase Definition:

Goal 1: Develop testing and debugging goals;

Goal 2: Initiate a testing planning process;

Goal 3: Institutionalize basic testing techniques and methods

Level 3—Integration:

Goal 1: Establish a software test organization;

Goal 2: Establish a technical training program;

Goal 3: Integrate testing into the software life cycle; Goal 4: Control and monitor testing
Level 4—Management and Measurement:

Goal 1: Establish an organization wide review program;

Goal 2: Establish a test measurement program;

Goal 3: Software quality evaluation

Level 5—Optimization/Defect Prevention/Quality Control:

Goal 1: Defect prevention;

Goal 2: Quality control;

Goal 3: Test process optimization

Basic definitions

Errors

An error is a mistake, misconception, or misunderstanding on the part of a software developer.

Faults

A fault (defect) is introduced into the software as the result of an error.

Failures

A failure is the inability of a software system or component to perform its required functions
within specified performance requirements

Test Cases

A test case in a practical sense is a test-related item which contains

1. A set of test inputs

2. Execution conditions

3. Expected outputs

Test

A test is a group of related test cases, or a group of related test cases and test procedures

Test Oracle

A test oracle is a document, or piece of software that allows testers to determine whether a test
has been passed or failed.

Test Bed

A test bed is an environment that contains all the hardware and software needed to test a
software component or a software system

Software Quality

Quality relates to the degree to which a system, system component, or process meets specified
requirements.

Quality metrics

A metric is a quantitative measure of the degree to which a system, system component, or

process possesses a given attribute
 quality attributes of a quality metric are

1. Correctness

2. Reliability

3. Usability

4. Integrity

5. Portability

6. Maintainability

7. interoperability

Software Testing Principles

Testing principles are important to test specialists/engineers because they provide the foundation for
developing testing knowledge and acquiring testing skills.

In the software domain, principles may also refer to rules or codes of conduct relating to professionals
who design, develop, test, and maintain software systems

Principle 1.

Testing is the process of exercising a software component using a selected set of test cases, with
the intent of (i) revealing defects (ii) evaluating quality.

Defects do occur, and they have a negative impact on software quality.Testers need to detect
these defects before the software becomes operational

Principle 2.

When the test objective is to detect defects, then a good test case is one that has a high
probability of revealing a yetundetected defect(s)

It requires the tester to consider the goal for each test case, which specific type of defect is to be
detected by the test case.the tester approaches testing in the same way a scientist approaches an
experiment

Principle 3.

Test results should be inspected meticulously

A failure may be overlooked, and the test may be granted a “pass” status. The defect may be revealed at
some later stage of testing, but in that case it may be more costly and difficult to locate and repair The
outcome of a quality test results in unnecessary rework, or oversight of a critical problem

Principle 4. A test case must contain the expected output or result

Tester need to determine

(i) whether a defect has been revealed

(ii) pass/fail status for the test

Principle 5. Test cases should be developed for both valid and invalid input conditions
Software users may have misunderstandings, or lack information about the nature of the inputs, often
make typographical errors. Use of test cases that are based on invalid inputs since they may exercise
the code in unexpected ways and identify unexpected software behavior

Principle 6. The probability of the existence of additional defects in a software component is proportional
to the number of defects already detected in that component

if there are two components A and B, and testers have found 20 defects in A and 3 defects in B, then the
probability of additional defects in A is higher than B

Principle 7. Testing should be carried out by a group that is independent of the development group.

It is difficult for a developer to admit that a software created and developed can be faulty. Developers
often have difficulty in locating the defects since their mental model of the code may overshadow as it
exists in actuality

Principle 8. Tests must be repeatable and reusable

Experiments in the testing domain to require recording of the exact conditions of the test.It is useful for
tests that need to be repeated after defect repair.

Principle 9. Testing should be planned

Test plans should be developed for each level of testing, and objectives for each level should be described
in the associated plan. The objectives should be stated as quantitatively as possible

Principle 10. Testing activities should be integrated into the software life cycle.

Test planning activities should be integrated into the software life cycle starting as early as in the
requirements analysis phase, and continue on throughout the software life cycle in parallel with
development activities

Principle 11. Testing is a creative and challenging task

A tester needs to have knowledge from both experience and education as to how software is specified,
designed, and developed. A tester needs to have knowledge of where faults of a certain type might occur
a tester needs to have a good grasp of the problem domain of the software being tested

The Tester’s Role in a Software Development Organization

Developers, analysts, and marketing staff need to realize that testers add value to a software product in
that they detect defects and evaluate quality as early as possible in the software life cycle.

The tester’s job is to reveal defects, find weak points, inconsistent behaviour. It is difficult for developers
to effectively test their own code (spend lot of time in developing)

The goal of a tester is to work with the developers to produce high-quality software that meets the
customers’ requirements. Testers also need to work alongside with requirements engineers to ensure
that requirements are testable. Test managers will need to cooperate with project managers in order to
develop reasonable test plans

At TMM levels 1 or 2, you may find that there is no independent software test function in your
organization. At levels 3 and higher of the TMM the testers should always have managerial independence
from developers

When defects are detected during testing, software should be returned to the developers who locate the
defect and repair the code
Origins of Defects

The term defect and its relationship to the terms error and failure in the context of the software
development domain.

Defects stem from the following sources

Education:
The software engineer did not have the proper educational background to prepare the software artefact
Communication:
The software engineer was not informed about something by a colleague
Oversight:
The software engineer omitted to do something.
Transcription:
The software engineer knows what to do, but makes a mistake in doing it
Process:
The process used by the software engineer misdirected her actions.

Origins of defects.

Defect Classes

Defects, are classified in to four major classes reflecting their point of origin in the software life cycle

 requirements/ specifications defects,

 design defects
 code defects
 testing defects
These defect classes are summarized as

Requirements and Specification Defects

Functional Description Defects

The overall description of what the product does, and how it should behave (inputs/outputs), is incorrect,
ambiguous, and/or incomplete.

Feature Defects

Features refer to functional aspects. Feature defects are due to feature descriptions that are missing,
incorrect, incomplete, or superfluous.

Feature Interaction Defects

Incorrect description of how the features should interact

Interface Description Defects

Defects that occur in the description of how the target software is to interface with external software,
hardware, and users

Design Defects

Algorithmic and Processing Defects

Processing steps in the algorithm as described by the pseudo code are incorrect

Eg: e processing steps in the algorithm written in the pseudo code language may not be in the correct
order

Control, Logic, and Sequence Defects

Control defects occur when logic flow in the pseudo code is not correct

Eg: Logic defects usually relate to incorrect use of logic operators, such as less than (), greater than (), etc.
Data Defects

Incorrect design of data structures. a record may be lacking a field, an incorrect type is assigned to a
variable or a field in a record, an array may not have the proper number of elements assigned, or storage
space may be allocated incorrectly

Module Interface Description Defects

These are defects derived from incorrect number of parameters, or an incorrect ordering of parameters.

Functional Description Defects

this category include incorrect, missing, and/or unclear design elements.

Eg: the design may not properly describe the correct functionality of a module

External Interface Description Defects

These are derived from incorrect design descriptions for interfaces with components, external software
systems, databases, and hardware devices

Eg: improper sequences of commands, lack of proper messages

Coding defects

Coding defects are derived from errors in implementing the code

Algorithmic and Processing Defects

code-related algorithmic and processing defects would now include unchecked overflow and underflow
conditions, comparing inappropriate data types, converting one data type to another, incorrect ordering
of arithmetic operators.

Control, Logic and Sequence Defects

eg incorrect expression of case statements, incorrect iteration of loops

Typographical Defects

These are principally syntax errors. Eg: incorrect spelling of a variable name that are usually detected by a
compiler

Initialization Defects

These occur when initialization statements are omitted or are incorrect

Data-Flow Defects

Certain reasonable operational sequences that data should flow through. Eg: a variable should be
initialized, before it is used, It should not be initialized twice before there is an intermediate use.

Data Defects

These are indicated by incorrect implementation of data structures. Eg:the programmer may omit a field
in a record, an incorrect type or constants set incorrectly

Module Interface Defects

interface defects in the code may be due to using incorrect or inconsistent parameter types, an incorrect
number of parameters, or improper ordering of the parameters

Code Documentation Defects

When the code documentation does not reflect what the program actually does, or is incomplete or
ambiguous, this is called a code documentation defect

External Hardware, Software Interfaces Defects

These defects arise from problems related to system calls, links to databases, input/output sequences,
memory usage, resource usage, interrupts and exception handling, data exchanges with hardware,
protocols, formats, interfaces with build files, and timing sequences (race conditions may result).

Testing Defects

Test Harness Defects

Auxiliary code must be developed in order to test software at the unit and integration levels is called the
test harness

Test harnesses are subject to the same types of code and design defects that can be found in all other
types of software

Test Case Design and Test Procedure Defects

These would encompass incorrect, incomplete, missing, inappropriate test cases, and test procedures.
These defects are again best detected in test plan reviews

Defect Examples

The following example illustrate some instances of the defect classes.

A simple program that calculates the total monetary value of a set of coins.

Functional description defects

Functional description is ambiguous and incomplete.

It does not state that the input, number_of_coins, and the output, number_of_dollars and number
_of_cents, should all have values of zero or greater.The values in number_of_coins, dollars and cents
cannot be negative in the real-world domain

Allowing the final program to accept negative values for the input number_of_coins for each of the
denominations, and consequently it may calculate an invalid value for the results.
Interface description defects

Interface description defects relate to the ambiguous and incomplete description of user–software
interaction. It is not clear from the specification how the user interacts with the program to provide input,
and how the output is to be reportedBecause of ambiguities in the user interaction description the
software may be difficult to use.

Design defects

Below shows the specification transformed in to a design description.

Design defects include the following

Design Description for Program calculate_coin_values

Program calculate_coin_values
number_of_coins is integer
total_coin_value is integer
number_of_dollars is integer
number_of_cents is integer
coin_values is array of six integers representing
each coin value in cents
initialized to: 1,5,10,25,25,100
begin
initialize total_coin_value to zero
initialize loop_counter to one
while loop_counter is less then six
begin
output "enter number of coins"
read (number_of_coins )
total_coin_value = total_coin_value +
number_of_coins * coin_value[loop_counter]
increment loop_counter
end
number_dollars = total_coin_value/100
number_of_cents = total_coin_value - 100 * number_of_dollars
output (number_of_dollars, number_of_cents)
end

Control, logic, and sequencing defects.

The defect in this subclass arises from an incorrect “while” loop condition (should be less than or equal to
six)

Algorithmic, and processing defects.

These arise from the lack of error checks for incorrect and/or invalid inputs, lack of a path where users
can correct erroneous inputs, lack of a path for recovery from input errors

Data defects.

This defect relates to an incorrect value for one of the elements of the integer array, coin_values, which
should read 1, 5,10,25,50,100.
External interface description defects.

These are defects arising from the absence of input messages or prompts that introduce the program to
the user and request inputs. The user has no way of knowing in which order the number of coins for each
denomination must be input, and when to stop inputting values.

There is an absence of help messages, and feedback for user if he wishes to change an input or learn the
correct format and order for inputting the number of coins

Coding Defects

Control, logic, and sequence defects:

These include the loop variable increment step which is out of the scope of the loop. Note that incorrect
loop condition (i<6) is carried over from design and should be counted as a design defect.

Algorithmic and processing defects:

The division operator may cause problems if negative values are divided, although this problem could be
eliminated with an input check.

Data Flow defects:

The variable total_coin_value is not initialized. It is used before it is defined. (This might also be
considered a data defect.)

Data Defects:

The error in initializing the array coin_values is carried over from design and should be counted as a
design defect.

External Hardware, Software Interface Defects:

The call to the external function “scanf” is incorrect. The address of the variable must be provided

(&number_of_coins).

Code Documentation Defects.

The documentation that accompanies this code is incomplete and ambiguous. It reflects the deficiencies
in the external interface description and other defects that occurred during specification and design

Even though it implements a simple function the program is unusable because of the nature of the
defects it contains. Such software is not acceptable to users.

Developer/Tester Support for Developing a Defect Repository

It is important to illustrate the benefits of developing a defect repository to store defect information.
A requirement for repository development should be a part of testing and/or debugging policy
statements Forms and templates will need to be designed to collect the data

he defect data is useful for test planning It helps you to select applicable testing techniques, design (and
reuse) the test cases you need, and allocate the amount of resources you will need to devote to detecting
and removing these defects
The defect repository, and support for TMM maturity goals.