Software Testing and Quality Assurance

SEng5441

Chapter 2: Software Testing Process

KIoT
Department of Software Engineering
 Software Development Life Cycle (SDLC)
◦ It is a systematic approach to develop software products.
◦ It creates a structure for the developer to design, create and deliver
high quality software according to the requirements of end users.
◦ It also provides a methodology for improving the quality of the
desired product.
◦ The purpose of SDLC is to provide help in producing a product
that is resource efficient and of high quality.
2

2
SDL C 1. Planning

7.
Maintenance
2. Analysis

6.
Deployment
3. Design

5. Testing 4. Implementation

3
 Basics of Software Testing
• Software Testing is a method to check whether the actual software product
matches expected requirements and to ensure that software product is defect
free.

• It involves execution of software/system components using manual or automated

tools to evaluate one or more properties.

• The purpose of software testing is to identify errors, gaps or missing requirements

in contrast to actual requirements.

4
 Cont…
• IEEE 1059, Testing in Software Engineering is a process of evaluating a
software product to find whether the current software product meets the
required conditions or not.

• The testing process involves evaluating the features of the software product for
requirements in terms of any missing requirements, bugs or errors, security,
reliability and performance.

• Properly tested software product ensures reliability, security and high

performance which further results in time saving, cost effectiveness and
customer satisfaction.
5
 Benefits of Software Testing
• Cost-Effective: It is one of the important advantages of software testing. Testing any SW
project on time helps you to save your money for the long term. In case if the bugs caught in
the earlier stage of software testing, it costs less to fix.

• Security: It is the most vulnerable and sensitive benefit of software testing. People are looking
for trusted products. It helps in removing risks and problems earlier.

• Product quality: It is an essential requirement of any software product. Testing ensures a

quality product is delivered to customers.

• Customer Satisfaction: The main aim of any product is to give satisfaction to their customers.
UI/UX Testing ensures the best user experience.

6
Software Testing Life Cycle (STLC)/process
◦ Software testing life cycle defines the stages in testing of software.
◦ It is executed in a systematic and planned manner.
◦ In STLC process, different activities are carried out to improve the quality of
the product.
◦ STLC, in general, comprises of the following phases:

7
 Software Testing Process…

The Concept of a Process

• A process is a series of activities performed to fulfill a purpose and

produce a tangible output based on a given input.

• Processes can be described and hence monitored and improved. A

process description must always include:
 A definition of the input
 A list of activities—the procedure
 A description of the output

8
 The Concept of a Process…
For a more comprehensive and more useful process description the following
information could also be included:
Entry criteria—What must be in place before we can start?
 Purpose—A description of what must be achieved ?
 Role—Who is going to perform the activities?
 Methods, techniques, tools—How exactly are we going to perform the activities?
 Measurements—What metrics are we going to collect for the process?
Templates—What should the output look like?
 Verification points—Are we on the right track?
 Exit criteria—What do we need to fulfill before we can say that we have finished?
9
The Concept of a Process…
The inputs on which this process is based are:
• Test strategy
• Project plan
• Master test plan
• Information about how the testing is progressing
The activities are:
• Test planning and control
• Test development
• Test analysis and design
• Test implementation and execution
• Evaluating exit criteria and reporting
• Test closure activities
10
The Concept of a Process…

The output consists of:

• Level test plan
• Test specification in the form of test conditions, test design,
test cases, and test procedures and/or test scripts
• Test environment design and specification and actual test
environment including test data
• Test logs
• Progress reports
• Test summary report
• Test experience report
11
Processes Dependency

• The input to a process must be the output from one or more

proceeding process(es)
• The output from a process must be the input to one or more
other processes—even the final product, which is the input
to the maintenance process.

12
 Generic Test Process
• Testing is a process rather than a single activity. Testing must be planned and
it requires discipline to act upon it.
• The quality and effectiveness of software testing are primarily determined by
the quality of the test processes used.
• The activities of testing can be divided into the following basic steps:
1. Planning and Control
2. Analysis and Design
3. Implementation and Execution
4. Evaluating exit criteria and Reporting
5. Test Closure activities
13
 Generic Test Process…
• The generic test process is applicable for each of the dynamic
test levels to be included in the course of the development
and maintenance of a product.
• So the process should be used in testing such as:
• Component testing
• Integration testing
• System testing
• Acceptance testing

14
 Generic Test Process…
• The generic test process is iterative—not a simple
straightforward process.

Figure 1. generic test process

15
 Generic Test Process…
• The test activities need not be performed in strict sequential
order.
• It is an iterative.
• The resulting iterations may be:
1. The defect is in the test object.
2. The defect is in the test procedure
3. More test cases must be specified to increase
coverage, and these must then be executed.
4. The exit criteria are relaxed or strengthened in the
test plan.

16
 Generic Test Process…
1) Planning and Control
Test Planning :
• Test planning involves producing a document that describes an overall approach and test
objectives.
• It involves reviewing the test basis, identifying the test conditions based on analysis of test items,
writing test cases and Designing the test environment.
• Completion or exit criteria must be specified so that we know when testing (at any stage) is
complete.
• Purpose
 To determine the scope and risks and identify the objectives of testing.
 To determine the required test resources like people, test environments etc.
 To schedule test analysis and design tasks, test implementation, execution and evaluation.
17
 Cont…
Test plan have sufficient information on:
1. How the testing will be done?
2. The test strategy to be followed?
3. What test methods will be followed?
4. When to stop testing? and
5. Its resource requirements:
◦ hardware,
◦ software,
◦ money &
◦ time

18
 Generic Test Process…

Control

• This is the activity of comparing actual progress against the plan,

and reporting the status, including deviations from the plan.

• It involves taking actions necessary to meet the mission and

objectives of the project.

19
Generic Test Process…
2) Analysis and Design
The purpose of the test analysis and design activities is to produce test designs
with test conditions and tests cases and the necessary test environment based on
the test basis and the test goals and approach outlined in the test plan.
• After reading and understanding the software requirement, the tester writes.
◦ the test scenarios- on the basis of requirements
◦ the test cases and then
◦ the test data (test data can be provided by the client also)
◦ the test scripts(a line-by-line description of all the actions that are necessary to perform and test on specific user need)
◦ the requirements traceability matrix
20
 Analysis and Design

Test analysis and Test Design has the following major tasks:

 To review the test basis: The test basis is the information on which test cases are based,
such as requirements, design specifications, product risk analysis, architecture and interfaces.

 To identify test conditions: is a specification that a tester must follow for testing a
software application. Example: When User Name and Password are valid then an application
will move forward.

 To design the tests: a process that describes “how” testing should be done.

 To design the test environment set-up and identify the required infrastructure and tools.

21
Analysis and Design

• The input from the level test plan that we need for this process is:

 Test objectives

 Scheduling and staffing for the activities

 Definition of test object(s)

 Approach—especially test case design techniques to use and structure

and contents of the test specification

 Completion criteria, not least required coverage

 Deliverables
22
 Analysis and Design

• The result of the test analysis and design should be documented in the test
specification.

• This document or series of documents encompasses

 The test designs—also called test groups

 The test cases—many test cases per test design

 Test procedures—often many-to-many relationship with test cases

23
 Activities in Test Analysis and Design

 Defining Test Designs

The number of test groups we can define depends on the test level and the
nature, size, and architecture of the test object:

In component testing we usually have one test group per component

For integration testing there are usually a few groups per interface

For system and acceptance testing we typically have many test groups

24
 Activities in Test Analysis and Design…
 Identification of Test Conditions
The features to be tested mentioned in the test design can be expressed as
test conditions or test requirements. A test condition is a verifiable item or
element.

The nature of a test condition depends on the nature of the test basis
documentation. It may for example be a function, a transaction, a feature, a
requirement, or a structural element like an interface parameter or a statement in the code.

The test conditions are based on or identical to our coverage items. They
are the items we are covering when we test the test object.

25
Activities in Test Analysis and Design…

26
 Creation of Test Cases

Based on the test conditions, high-level test cases and low-level test cases are
produce.

A high-level test case is a test case without specific values for input data

and expected results, but with logical operators or other means of defining

what to test in general terms.

27
• High level test case example

28
• A low-level test case is a test case with specific values defined for both
input and expected result.

29
 Generic Test Process
3) Implementation and Execution/ Environment Setup

Test execution involves actually running the specified test on a computer system either manually or by using an automated test
tool. It is a Fundamental Test Process in which actual work is done.

◦ Every software is developed by considering hardware and software requirement.

◦ Here we setup the testing environment (e. g. server/client/network, or install the setup if it is window’s application etc.)
with the goal of replicating the end-users’ environment.

◦ This step decides on which Platform/OS the tester needs to perform testing of project.

◦ Example: Use only Internet Explorer, Resolution must be like 136 x 768, or in many large applications they indicate the minimum hardware
and software requirement like Photoshop, latest version of Matlab, Oracle, large games etc.

◦ In this phase, according to requirements, they configure required hardware and software.

◦ Tester may or may not involve in setting the configuration.

30
 Cont…

Test implementation has the following major task:

 To develop and prioritize test cases by using d/t techniques and create test
data for those tests.
 To create test suites from the test cases for efficient test execution. Test
suite is a collection of test cases that are used to test a software program
 To re-execute the tests that previously failed in order to confirm a fix.
 To log the outcome of the test execution. A test log is the status of the test
case (pass/fail).
 To Compare actual results with expected result.
31
The purpose of the test implementation is to organize the test cases in
procedures and/or scripts and to perform the physical test in the correct
environment.

The inputs on which this process is based are:

• Level test plan
• Test conditions and test design
• Other relevant documents
• The test object

32
The activities are:
• Organizing test procedures
• Design and verify the test environment
• Execute the tests
• Record the testing
• Check the test results
The output consists of:
• Test specification
• Test environment
• Test logs
• Incident reports
• Tested test object

33
Generic Test Process
4) Evaluating Exit criteria and Reporting
Evaluating exit criteria is a process defining when to stop testing. It depends on coverage of
code, functionality or risk.
Basically it also depends on business risk, cost and time and vary from project to project. Exit
criteria come into picture, when:
 Maximum test cases are executed with certain pass percentage
 Bug rate falls below certain level
 When we achieve the deadlines
Evaluating exit criteria has the following major tasks:
 To assess if more test are needed or if the exit criteria specified should be changed
 To write a test summary report for stakeholders

34
The inputs on which this process is based are:
• Test plan
• Measurements from the test development and execution processes
The activities are:
• Comparing actual measurements with estimates and planned values
• Reporting test results
The output consists of:
• Presentation of test progress
• Test report

35
Generic Test Process
5)Test Closure activities:
Test closure activities are done when software is ready to be delivered. The
testing can be closed for the other reasons also like:
 When a project is cancelled
 When some target is achieved
 When a maintenance release or update is done

Test closure activities have the following major tasks:

 To check which planned deliverables are actually delivered and to ensure that all incident reports
have been resolved
 To finalize and archive test ware such as scripts, test environments, etc. for later reuse
 To handover the test ware to the maintenance organization. They will give support to the software
 To evaluate how the testing went and learn lessons for future releases and projects
36
The inputs on which this process is based are:
• Level test plan
• Test ware, including test environment
The overall procedure consists of the activities:
• Final check of deliveries and incident reports
• Secure storage/handover of test ware
• Retrospection/observations
The output generated in this process is:
• Test experience report
• Configuration management documentation
37