CODING AND TESTING

Coding is undertaken once the design phase is complete and the design documents have been
successfully reviewed. In the coding phase, every module specified in the design document is
coded and unit tested. After all the modules of a system have been coded and unit tested, the
integration and system testing phase is undertaken. Integration and testing of modules is carried
out according to an integration plan. The integration plan, according to which different modules
are integrated together, usually done integration of modules through a number of steps. During
each integration step, a number of modules are added to the partially integrated system and the
resultant system is tested.

CODING
The input to the coding phase is the design document produced at the end of the design phase.
The detailed design is usually documented in the form of module specifications where the data
structures and algorithms for each module are specified. During the coding phase, different
modules identified in the design document are coded according to their respective module
specifications. The objective of the coding phase is to transform the design of a system into code in
a high-level language, and then to unit test this code.
Normally, good software development organisations require their programmers to adhere to some
well-defined and standard style of coding which is called their coding standard. A coding
standard lists several rules to be followed during coding, such as the way variables are to be
named, the way the code is to be laid out, the error return conventions, etc.

CODE REVIEW
Testing is an effective defect removal mechanism. However, testing is applicable to only
executable code. Review is a very effective technique to remove defects from source code. Code
review for a module is undertaken after the module successfully compiles. That is, all the syntax
errors have been eliminated from the module. Code review has been recognised as an extremely
cost-effective strategy for eliminating coding errors and for producing high quality code as it
reviews directly to detect errors.
Normally, the following two types of reviews are carried out on the code of a module:
 Code Inspection.
 Code Walkthrough.
Code Walkthrough
Code walkthrough is an informal code analysis technique. In this technique, a module is taken up
for review after the module has been coded, successfully compiled, and all syntax errors have been
eliminated. The main objective of code walkthrough is to discover the algorithmic and logical
errors in the code.
Some of the guidelines for Code Walkthrough are as follow:
 The team performing code walkthrough should not be either too big or too small. Ideally, it
should consist of between three to seven members.
 Discussions should focus on discovery of errors and avoid deliberations on how to fix the
discovered errors.
 In order to foster co-operation and to avoid the feeling among the engineers that they are
being watched and evaluated in the code walkthrough meetings, managers should not
attend the walkthrough meetings.
Code Inspection
During code inspection, the code is examined for the presence of some common programming
errors. The principal aim of code inspection is to check for the presence of some common types of
errors that usually creep into code due to programmer mistakes and oversights and to check
whether coding standards have been adhered to.
The inspection process has several beneficial side effects, other than finding errors. The
programmer usually receives feedback on programming style, choice of algorithm, and
programming techniques. The other participants gain by being exposed to another programmer’s
errors.

Clean Room Testing

Clean Room testing relies heavily on walkthroughs, inspection, and formal verification. This
technique reportedly produces documentation and code that is more reliable and maintainable
than other development methods relying heavily on code execution-based testing. The main
problem with this approach is that testing effort is increased as walkthroughs, inspection, and
verification are time consuming for detecting all simple errors.

SOFTWARE DOCUMENTATION
When a software is developed, in addition to the executable files and the source code, several
kinds of documents such as users’ manual, software requirements specification (SRS) document,
design document, test document, installation manual, etc., are developed as part of the software
engineering process. Good documents are helpful in the following ways:
 Good documents help enhance understandability of code.
 Documents help the users to understand and effectively use the system.
 Good documents help to effectively tackle the manpower turnover problem.
 Production of good documents helps the manager to effectively track the progress of the
project.
Different types of software documents can broadly be classified into the following:
 Internal documentation: These are provided in the source code itself.
 External documentation: These are the supporting documents such as SRS document,
installation document, user manual, design document, and test document.
Internal Documentation
Internal documentation is the code comprehension features provided in the source code itself.
Internal do cumentation can be provided in the code in several forms. The important types of
internal documentation are the following:
 Comments embedded in the source code.
 Use of meaningful variable names.
 Module and function headers.
 Code indentation.
 Code structuring (i.e., code decomposed into modules and functions).
 Use of enumerated types.
 Use of constant identifiers.
 Use of user-defined data types.
Careful experiments suggest that out of all types of internal documentation, meaningful variable
names is most useful while trying to understand a piece of code. Good software development
organisations usually ensure good internal documentation by appropriately formulating their
coding standards and coding guidelines.
External Documentation
External documentation is provided through various types of supporting documents such as
users’ manual, software requirements specification document, design document, test document,
etc. An important feature that is required of any good external documentation is consistency with
the code. In other words, all the documents developed for a product should be up-to-date and
every change made to the code should be reflected in the relevant external documents. Another
important feature required for external documents is proper understandability by the category of
users for whom the document is designed. For achieving this, Gunning’s fog index is very useful.
Gunning’s fog index: Gunning’s fog index (developed by Robert Gunning in 1952) is a metric that
has been designed to measure the readability of a document. The computed metric value (fog
index) of a document indicates the number of years of formal education that a person should have,
in order to be able to comfortably understand that document. That is, if a certain document has a
fog index of 12, anyone who has completed his 12th class would not have much difficulty in
understanding that document.

TESTING
The aim of program testing is to help identify all defects in a program. However, in practice, even
after satisfactory completion of the testing phase, it is not possible to guarantee that a program is
error free. This is because the input data domain of most programs is very large, and it is not
practical to test the program exhaustively with respect to each value that the input can assume.

Verification Versus Validation

The objectives of both verification and validation techniques are very similar since both these
techniques are designed to help remove errors in a software.
 Verification is the process of determining whether the output of one phase of software
development conforms to that of its previous phase; whereas validation is the process of
determining whether fully developed software conforms to its requirements specification.
 The primary techniques used for verification include review, simulation, formal
verification, and testing. Review, simulation, and testing are usually considered as informal
verification techniques. Formal verification usually involves use of theorem proving
techniques or use of automated tools such as a model checker. On the other hand,
validation techniques are primarily based on product testing.
 Verification does not require execution of the software, whereas validation requires
execution of the software.
 Verification is carried out during the development process to check if the development
activities are proceeding alright, whereas validation is carried out to check if the right as
required by the customer has been developed.
 Verification techniques can be viewed as an attempt to achieve phase containment of errors.
The principle of detecting errors as close to their points of commitment as possible is known
as phase containment of errors. While verification is concerned with phase containment of
errors, the aim of validation is to check whether the deliverable software is error free.
To achieve high product reliability in a cost-effective manner, a development team needs to
perform both verification and validation activities. Therefore,
Error detection techniques = Verification techniques + Validation techniques

Testing Activities
Testing involves performing the following main activities:
Test suite design: The set of test cases using which a program is to be tested is designed possibly
using several test case design techniques.
Running test cases and checking the results to detect failures: Each test case is run and the
results are compared with the expected results. A mismatch between the actual result and
expected results indicates a failure and debugged later.
Locate error: In this activity, the failure symptoms are analysed to locate the errors.
Error correction: After the error is located during debugging, the code is appropriately changed to
correct the error.
 Testing process
UNIT TESTING
Unit testing is undertaken after a module has been coded and reviewed. This activity is typically
undertaken by the coder of the module himself in the coding phase. Before carrying out unit
testing, the unit test cases have to be designed and the test environment for the unit under test has
to be developed. The environment needed to perform unit testing are:
Driver and stub modules
In order to test a single module, we need a complete environment to provide all relevant code that
is necessary for execution of the module. That is, besides the module under test, the following are
needed to test the module:
 The procedures belonging to other modules that the module under test calls.
 Non-local data structures that the module accesses.
 A procedure to call the functions of the module under test with appropriate parameters.
In this context, stubs and drivers are designed to provide the complete environment for a module
so that testing can be carried out.
Stub: A stub procedure is a dummy procedure that has the same I/O parameters as the function
called by the unit under test but has a highly simplified behaviour.
Driver: A driver module should contain the non-local data structures accessed by the module
under test. Additionally, it should also have the code to call the different functions of the unit
under test with appropriate parameter values for testing.

Unit testing with the help of driver and stub modules

BLACK-BOX TESTING
In black-box testing, test cases are designed from an examination of the input/output values only
and no knowledge of design or code is required. The following are the two main approaches
available to design black box test cases:
  Equivalence class partitioning
 Boundary value analysis
Equivalence Class Partitioning
In the equivalence class partitioning approach, the domain of input values to the program under
test is partitioned into a set of equivalence classes. The partitioning is done such that for every
input data belonging to the same equivalence class, the program behaves similarly.
Equivalence classes for a unit under test can be designed by examining the input data and output
data. The following are two general guidelines for designing the equivalence classes:
1. If the input data values to a system can be specified by a range of values, then one valid and
two invalid equivalence classes need to be defined. For example, if the equivalence class is the
set of integers in the range 1 to 10 (i.e., [1,10]), then the invalid equivalence classes are [−∞,0],
[11,+∞].
2. If the input data assumes values from a set of discrete members of some domain, then one
equivalence class for the valid input values and another equivalence class for the invalid input
values should be defined.
Boundary Value Analysis
A type of programming error that is frequently committed by programmers is missing out on the
special consideration that should be given to the values at the boundaries of different equivalence
classes of inputs. For example, programmers may improperly use < instead of <=, or conversely
<= for <, etc.
Boundary value analysis-based test suite design involves designing test cases using the values at
the boundaries of different equivalence classes.
To design boundary value test cases, it is required to examine the equivalence classes to check if
any of the equivalence classes contains a range of values. For those equivalence classes that are not
a range of values (i.e., consist of a discrete collection of values) no boundary value test cases can be
defined.

WHITE-BOX TESTING
White-box testing is an important type of unit testing. A large number of white-box testing
strategies exist. Each testing strategy essentially designs test cases based on analysis of some
aspect of source code.
Basic Concepts
A white-box testing strategy can either be coverage-based or faultbased.
Fault-based testing
A fault-based testing strategy targets to detect certain types of faults. These faults that a test
strategy focuses on constitutes the fault model of the strategy.
Coverage-based testing
A coverage-based testing strategy attempts to execute (or cover) certain elements of a program.
Popular examples of coverage-based testing strategies are statement coverage, branch coverage,
multiple condition coverage, and path coverage-based testing.
Testing criterion for coverage-based testing
A coverage-based testing strategy typically targets to execute (i.e., cover) certain program elements
for discovering failures.
The set of specific program elements that a testing strategy targets to execute is called the testing
criterion of the strategy.
For example, if a testing strategy requires all the statements of a program to be executed at least
once, then we say that the testing criterion of the strategy is statement coverage.
Statement Coverage
The statement coverage strategy aims to design test cases so as to execute every statement in a
program at least once. The principal idea governing the statement coverage strategy is that unless
a statement is executed, there is no way to determine whether an error exists in that statement.
It can however be pointed out that a weakness of the statement- coverage strategy is that executing
a statement once and observing that it behaves properly for one input value is no guarantee that it
will behave correctly for all input values.
Branch Coverage
A test suite satisfies branch coverage, if it makes each branch condition in the program to assume
true and false values in turn. In other words, for branch coverage each branch in the CFG (Control
Flow Graph) representation of the program must be taken at least once, when the test suite is
executed.
Multiple Condition Coverage
In the multiple condition (MC) coverage-based testing, test cases are designed to make each
component of a composite conditional expression to assume both true and false values.
Path Coverage
A test suite achieves path coverage if it executes each linearly independent paths ( or basis paths )
at least once. A linearly independent path can be defined in terms of the control flow graph (CFG)
of a program.

Control flow graph (CFG)

A control flow graph describes how the control flows through the program. We can define a
control flow graph as the following:
A control flow graph describes the sequence in which the different instructions of a program get
executed.
In order to draw the control flow graph of a program, we need to first number all the statements of
a program. The different numbered statements serve as nodes of the control flow graph.

There exists an edge from one node to another, if the execution of the statement representing the
first node can result in the transfer of control to the other node.
Path
A path through a program is any node and edge sequence from the start node to a terminal node
of the control flow graph of a program. Writing test cases to cover all paths of a typical program is
impractical since there can be an infinite number of paths through a program in presence of loops.
Linearly independent set of paths (or basis path set)
A set of paths for a given program is called linearly independent set of paths (or the set of basis
paths or simply the basis set), if each path in the set introduces at least one new edge that is not
included in any other path in the set.
McCabe’s Cyclomatic Complexity Metric
McCabe obtained his results by applying graph-theoretic techniques to the control flow graph of a
program. McCabe’s cyclomatic complexity defines an upper bound on the number of independent
paths in a program. There are three different ways to compute the cyclomatic complexity:
Method 1: Given a control flow graph G of a program, the cyclomatic complexity V(G) can be
computed as:
V(G) = E – N + 2
where, N is the number of nodes of the control flow graph and E is the number of edges in the
control flow graph. For the CFG of example shown in figure above, E = 7 and N = 6. Therefore,
the value of the Cyclomatic complexity = 7 – 6 + 2 = 3.
Method 2: An alternate way of computing the cyclomatic complexity of a program is based on a
visual inspection of the control flow graph is as follows —In this method, the cyclomatic
complexity V (G) for a graph G is given by the following expression:
V(G) = Total number of non-overlapping bounded areas + 1
In the program’s control flow graph G, any region enclosed by nodes and edges can be called as a
bounded area. This is an easy way to determine the McCabe’s cyclomatic complexity. Consider
the CFG example shown in figure above. From a visual examination of the CFG the number of
bounded areas is 2. Therefore the cyclomatic complexity, computed with this method is also
2+1=3.
Method 3: The cyclomatic complexity of a program can also be easily computed by computing the
number of decision and loop statements of the program. If N is the number of decision and loop
statements of a program, then the McCabe’s metric is equal to N + 1.

Data Flow-based Testing

Data flow based testing method selects test paths of a program according to the definitions and
uses of different variables in a program.
Consider a program P . For a statement numbered S of P , let
DEF(S) = {X /statement S contains a definition of X } and
USES(S)= {X /statement S contains a use of X }
For the statement S: a=b+c;, DEF(S)={a}, USES(S)={b, c}. The definition of variable X at statement S
is said to be live at statement S1 , if there exists a path from statement S to statement S1 which does
not contain any definition of X .

Mutation Testing
All white-box testing strategies that we have discussed so far, are coverage-based testing
techniques. In contrast, mutation testing is a fault-based testing technique in the sense that
mutation test cases are designed to help detect specific types of faults in a program.
The idea behind mutation testing is to make a few arbitrary changes to a program at a time. Each
time the program is changed, it is called a mutated program and the change effected is called a
mutant. An underlying assumption behind mutation testing is that all programming errors can be
expressed as a combination of simple errors. A mutation operator makes specific changes to a
program. For example, one mutation operator may randomly delete a program statement. A
mutant may or may not cause an error in the program. If a mutant does not introduce any error in
the program, then the original program and the mutated program are called equivalent programs.
DEBUGGING
After a failure has been detected, it is necessary to first identify the program statement(s) that are
in error and are responsible for the failure, the error can then be fixed.
Debugging Approaches
The following are some of the approaches that are popularly adopted by the programmers for
debugging:
Brute force method
This is the most common method of debugging but is the least efficient method. In this approach,
print statements are inserted throughout the program to print the intermediate values with the
hope that some of the printed values will help to identify the statement in error.
Backtracking
This is also a fairly common approach. In this approach, starting from the statement at which an
error symptom has been observed, the source code is traced backwards until the error is
discovered.
Cause elimination method
In this approach, once a failure is observed, the symptoms of the failure (i.e., certain variable is
having a negative value though it should be positive, etc.) are noted. Based on the failure
symptoms, the causes which could possibly have contributed to the symptom are developed and
tests are conducted to eliminate each.
Program slicing
This technique is similar to back tracking. A slice of a program for a particular variable and at a
particular statement is the set of source lines preceding this statement that can influence the value
of that variable. Program slicing makes use of the fact that an error in the value of a variable can be
caused by the statements on which it is data dependent.

Debugging Guidelines
Debugging is often carried out by programmers based on their ingenuity and experience. The
following are some general guidelines for effective debugging:
 Many times debugging requires a thorough understanding of the program design.
 Debugging may sometimes even require full redesign of the system.
 One must be beware of the possibility that an error correction may introduce new errors.

PROGRAM ANALYSIS TOOLS

A program analysis tool usually is an automated tool that takes either the source code or the
executable code of a program as input and produces reports regarding several important
characteristics of the program, such as its size, complexity, adequacy of commenting, adherence to
programming standards, adequacy of testing, etc. We can classify various program analysis tools
into the following two broad categories:
 Static analysis tools
 Dynamic analysis tools
Static Analysis Tools
Static program analysis tools assess and compute various characteristics of a program without
executing it. Typically, static analysis tools analyse the source code to compute certain metrics
characterising the source code (such as size, cyclomatic complexity, etc.) and also report certain
analytical conclusions. Code review techniques such as code walkthrough and code inspection
discussed can be considered as static analysis methods since those target to detect errors based on
analysing the source code.
A major practical limitation of the static analysis tools lies in their inability to analyse run-time
information such as dynamic memory references using pointer variables and pointer arithmetic,
etc.
Dynamic Analysis Tools
Dynamic program analysis tools can be used to evaluate several program characteristics based on
an analysis of the run time behaviour of a program. These tools usually record and analyse the
actual behaviour of a program while it is being executed. A dynamic program analysis tool (also
called a dynamic analyser) usually collects execution trace information by instrumenting the code.
Code instrumentation is usually achieved by inserting additional statements to print the values of
certain variables into a file to collect the execution trace of the program. The instrumented code
when executed, records the behaviour of the software for different test cases.
Normally the dynamic analysis results are reported in the form of a histogram or pie chart to
describe the structural coverage achieved for different modules of the program. The output of a
dynamic analysis tool can be stored and printed easily to provide evidence that thorough testing
has been carried out.

INTEGRATION TESTING
The objective of integration testing is to detect the errors at the module interfaces (call parameters).
For example, it is checked that no parameter mismatch occurs when one module invokes the
functionality of another module. The objective of integration testing is to check whether the
different modules of a program interface with each other properly.
During integration testing, different modules of a system are integrated in a planned manner
using an integration plan.
Any one (or a mixture) of the following approaches can be used to develop the test plan:
 Big-bang approach to integration testing
 Top-down approach to integration testing
 Bottom-up approach to integration testing
 Mixed (also called sandwiched ) approach to integration testing
Big-bang approach to integration testing
Big-bang testing is the most obvious approach to integration testing. In this approach, all the
modules making up a system are integrated in a single step. However, this technique can
meaningfully be used only for very small systems. The main problem with this approach is that
once a failure has been detected during integration testing, it is very difficult to localise the error as
the error may potentially lie in any of the modules.
Bottom-up approach to integration testing
Large software products are often made up of several subsystems. A subsystem might consist of
many modules which communicate among each other through well-defined interfaces. In bottom-
up integration testing, first the modules for the each subsystem are integrated. Thus, the
subsystems can be integrated separately and independently. The primary purpose of carrying out
the integration testing a subsystem is to test whether the interfaces among various modules
making up the subsystem work satisfactorily. The test cases must be carefully chosen to exercise
the interfaces in all possible manners.
The principal advantage of bottom- up integration testing is that several disjoint subsystems can
be tested simultaneously. Another advantage of bottom-up testing is that the low-level modules
get tested thoroughly, since they are exercised in each integration step. A disadvantage of bottom-
up testing is the complexity that occurs when the system is made up of a large number of small
subsystems that are at the same level.
Top-down approach to integration testing
Top-down integration testing starts with the root module in the structure chart and one or two
subordinate modules of the root module. After the top-level ‘skeleton’ has been tested, the
modules that are at the immediately lower layer of the ‘skeleton’ are combined with it and
tested. Top-down integration testing approach requires the use of program stubs to simulate the
effect of lower-level routines that are called by the routines under test. An advantage of top-down
integration testing is that it requires writing only stubs, and stubs are simpler to write compared to
drivers. A disadvantage of the top-down integration testing approach is that in the absence of
lower-level routines, it becomes difficult to exercise the top-level routines in the desired manner
since the lower level routines usually perform input/output (I/O) operations.
Mixed approach to integration testing
The mixed (also called sandwiched ) integration testing follows a combination of top-down and
bottom-up testing approaches. The mixed approach overcomes this shortcoming of the top-down
and bottom-up approaches. In the mixed testing approach, testing can start as and when modules
become available after unit testing. Therefore, this is one of the most commonly used integration
testing approaches. In this approach, both stubs and drivers are required to be designed.

SYSTEM TESTING
After all the units of a program have been integrated together and tested, system testing is taken
up. System tests are designed to validate a fully developed system to assure that it meets its
requirements. The test cases are therefore designed solely based on the SRS document.
There are essentially three main kinds of system testing depending on who carries out testing:
1. Alpha Testing: Alpha testing refers to the system testing carried out by the test team within the
developing organisation.
2. Beta Testing: Beta testing is the system testing performed by a select group of friendly
customers.
3. Acceptance Testing: Acceptance testing is the system testing performed by the customer to
determine whether to accept the delivery of the system.
In each of the above types of system tests, the test cases can be the same, but the difference is with
respect to who designs test cases and carries out testing.
The system test cases can be classified into functionality and performance test cases. Before a fully
integrated system is accepted for system testing, smoke testing is performed. Smoke testing is
done to check whether at least the main functionalities of the software are working properly.
The functionality tests are designed to check whether the software satisfies the functional
requirements as documented in the SRS document. The performance tests, on the other hand, test
the conformance of the system with the non-functional requirements of the system.

Smoke Testing
Smoke testing is carried out before initiating system testing to ensure that system testing would be
meaningful, or whether many parts of the software would fail. The idea behind smoke testing is
that if the integrated program cannot pass even the basic tests, it is not ready for a vigorous
testing. For smoke testing, a few test cases are designed to check whether the basic functionalities
are working.

Performance Testing
Performance testing is an important type of system testing. Performance testing is carried out to
check whether the system meets the non-functional requirements identified in the SRS document.
There are several types of performance testing corresponding to various
types of non-functional requirements. All performance tests can be considered as black-box tests.
Stress testing
Stress testing is also known as endurance testing. Stress testing evaluates system performance
when it is stressed for short periods of time. Stress tests are black-box tests which are designed to
impose a range of abnormal and even illegal input conditions so as to stress the capabilities of the
software.
Volume testing
Volume testing checks whether the data structures (buffers, arrays, queues, stacks, etc.) have been
designed to successfully handle extraordinary situations.
Configuration testing
Configuration testing is used to test system behaviour in various hardware and software
configurations specified in the requirements. The system is configured in each of the required
configurations and depending on the specific customer requirements, it is checked if the system
behaves correctly in all required configurations.
Compatibility testing
This type of testing is required when the system interfaces with external systems (e.g., databases,
servers, etc.). Compatibility aims to check whether the interfaces with the external systems are
performing as required.
Regression testing
This type of testing is required when a software is maintained to fix some bugs or enhance
functionality, performance, etc.
Recovery testing
Recovery testing tests the response of the system to the presence of faults, or loss of power,
devices, services, data, etc.
Maintenance testing
This addresses testing the diagnostic programs, and other procedures that are required to help
maintenance of the system.
Documentation testing
It is checked whether the required user manual, maintenance manuals, and technical manuals
exist and are consistent.
Usability testing
Usability testing concerns checking the user interface to see if it meets all user requirements
concerning the user interface. During usability testing, the display screens, messages, report
formats, and other aspects relating to the user interface requirements are tested.
Security testing
Security testing is essential for software that handle or process confidential data that is to be
guarded against pilfering. It needs to be tested whether the system is fool-proof from security
attacks such as intrusion by hackers.

Error Seeding
Sometimes customers specify the maximum number of residual errors that can be present in the
delivered software. The error seeding technique can be used to estimate the number of residual
errors in a software. Error seeding, as the name implies, it involves seeding the code with some
known errors. In other words, some artificial errors are introduced (seeded) into the program. The
number of these seeded errors that are detected in the course of standard testing is determined.
These values in conjunction with the number of unseeded errors detected during testing can be
used to predict the following aspects of a program:
 The number of errors remaining in the product.
 The effectiveness of the testing strategy.

SOME GENERAL ISSUES ASSOCIATED WITH TESTING

There are two general issues associated with testing. These are—how to document the results of
testing and how to perform regression testing.
Test documentation
A piece of documentation that is produced towards the end of testing is the test summary report.
This report normally covers each subsystem and represents a summary of tests which have been
applied to the subsystem and their outcome. It normally specifies the following:
 What is the total number of tests that were applied to a subsystem.
 Out of the total number of tests how many tests were successful.
 How many were unsuccessful, and the degree to which they were unsuccessful, e.g.,
whether a test was an outright failure or whether some of the expected results of the test
were actually observed.
Regression testing
Regression testing is the practice of running an old test suite after each change to the system or
after each bug fix to ensure that no new bug has been introduced due to the change or the bug fix.
However, if only a few statements are changed, then the entire test suite need not be run —
only those test cases that test the functions and are likely to be affected by the change need to be
run.