Software Testing

22518
Chapter 3
Test Management
(R-04, U-04, A-06 =14 Marks)

3.1 Test Planning:

Preparing a Test Plan,
Deciding Test Approach,
Setting Up Criteria for Testing,
Identifying Responsibilities, Staffing, Resource Requirements,
Test Deliverables,
Testing Tasks
3.2 Test Management:
Test Infrastructure Management,
Test People Management.
3.3 Test Process:
Base Lining a Test Plan,
Test Case Specification.
3.4 Test Reporting:
Executing Test Cases,
Preparing Test Summary Report.
Mrs. Shubhangi Chintawar
Lecturer
VESP
Software Testing (22518) CO5I

Test Plan
A TEST PLAN is a detailed document that describes the test strategy, objectives,
schedule, estimation and deliverables and resources required for testing. Test Plan helps
us determine the effort needed to validate the quality of the application under test. The
test plan serves as a blueprint to conduct software testing activities as a defined process
which is minutely monitored and controlled by the test manager.

Importance of Test Plan

• It serves as a roadmap to the testing process to ensure your testing project is
successful and helps you control risk.
• It clearly defines the roles and responsibilities of every team member, outlines the
resource requirements which are essential to carry out the testing process
• Planning and a test plan encourages better communication with other project
team members, testers, peers, managers, and other stakeholders
• Help people outside the test team such as developers, business managers,
customers understand the details of testing.
• Important aspects like test estimation, test scope, Test Strategy are documented
in Test Plan, so it can be reviewed by Management Team and re-used for other
projects.

How to write a Test Plan

1. Analyze the product

2. Design the Test Strategy
3. Define the Test Objectives
4. Define Test Criteria
5. Resource Planning
6. Plan Test Environment
7. Schedule & Estimation
8. Determine Test Deliverables

1
Software Testing (22518) CO5I

1. Analyze the product

It’s not possible to test a product without any information about it. So, the first step
towards creating a test plan is to analyze the product, its features, and functionalities.
Need to have a proper understanding of user requirements and expectations.

2. Develop Test Strategy

Test Strategy is a critical step in making a Test Plan. A Test Strategy document, is a high-
level document, which is usually developed by Test Manager.
This document defines:
The project’s testing objectives and the means to achieve them
Determines testing effort and costs

2.1) Define Scope of Testing

The components of the system to be tested (hardware, software, middleware, etc.) are
defined as "in scope “. The components of the system that will not be tested also need to
be clearly defined as being "out of scope. "Defining the scope of your testing project is
very important for all stakeholders.
A precise scope helps:
Give everyone a confidence & accurate information of the testing you are doing; all
project members will have a clear understanding about what is tested and what is not
How do you determine scope your project?
Precise customer requirement
Project Budget
Product Specification
Skills & talent of your test team

2.2) Identify Testing Type

A Testing Type is a standard test procedure that gives an expected test outcome.
Each testing type is formulated to identify a specific type of product bugs. But all Testing
Types are aimed at achieving one common goal “Early detection of all the defects before
releasing the product to the customer”
Unit test
Integration test
System test
Acceptance test
2
Software Testing (22518) CO5I

2.3) Document Risk & Issues

Risk is future’s uncertain event with a probability of occurrence and a potential for loss.
When the risk actually happens, it becomes the ‘issue’.

2.4) Create Test Logistics

The Test Manager should answer the following questions:
Who will test?
When will the test occur?

3.Define Test Objective

To release a high-quality bug-free software product to market on time, So, as a part of a
test plan, need to clearly define the testing scope and its boundaries.
You could just follow these two simple steps to define test objectives:
• List all the software features which may need testing
• Define the target of testing, based on features listed previously

4. Define Test Criteria

Test Criteria is a standard or rule on which a test procedure or test judgment can be based
There’re 2 types of test criteria as following
Suspension Criteria
If the suspension criteria are met during testing, the active test cycle will
be suspended until the criteria are resolved.
Exit Criteria
It specifies the criteria that denote a successful completion of a test phase. The exit
criteria are the targeted results of the test and are necessary before proceeding to the
next phase of development.

5.Resource Planning
The resource plan is basically a detailed summary of all types of resources required to
complete the project task. The resources here could be human, equipment and materials
needed to complete a project. It is an important step which helps you determine the
number of resources to be used for the project.

6.Plan Test Environment

A testing environment is a setup of software and hardware on which the testing team is
going to execute test cases. The test environment consists of real business and user
environment, as well as physical environments, such as server, front end running
environment.

3
Software Testing (22518) CO5I

7.Schedule & Estimation

Creating a schedule helps you control the progress of the testing process.
While drawing up a schedule, you should consider factors like:
a. Project estimate
b. Project risk estimate
c. Resource estimate
d. Employee roles & responsibilities
e. Test activity deadlines

8.Test Deliverables
Test Deliverables is a list of all the documents, tools and other components that has to be
developed and maintained in support of the testing effort.
There are different test deliverables at every phase of the software development
lifecycle.

4
Software Testing (22518) CO5I

Test Plan Template

Parameter Description
Test Plan Identifier Uniquely identifies the test plan & may include the version
number
Introduction Sets objectives, scope, goals, resource & budget constraints
Test Items Lists the systems and subsystems which are to be tested
Features to be Tested All the features & functionalities to be tested are listed here

Features not to be Lists the features of the software product that need not be
Tested tested
Approach Has sources of test data, inputs and outputs, testing priorities
Item pass/fail Describes a success criterion for evaluating the test results
Suspension Criteria Has criteria that may result in suspending testing activities
Test Deliverables Includes test cases, sample data, test report, issue log
Testing Tasks Describes dependencies between tasks & resources needed
Environmental Needs Lists software, hardware or other testing requirements

Responsibilities Lists roles and responsibilities assigned to the testing team

Staffing Needs Describes the additional training needs for the staff
Schedule Details on when the testing activities will take place are listed
Risks Lists overall risk of the project as it pertains to testing
Approvals Contains signature of approval from stakeholders

Test Plan Guidelines

• Make the plan concise.
• Avoid redundancy and superfluous.
• Be specific.
• Make use of lists and tables wherever possible.
• Avoid lengthy paragraphs.
• Have the test plan reviewed a number of times prior to baselining it or sending it
for approval.
• Update the plan as and when necessary.

Deciding Test Approach/Strategy

What needs to be tested, to enable estimation of size, effort, and schedule. This
includes identifying:
1. What type of testing would you use for testing the functionality?
2. What are the configurations or scenarios for testing the features?
3. What integration testing would you do to ensure these features work together?
4.What localization validations would be needed?
5.What “non-functional” tests would you need to do?
5
Software Testing (22518) CO5I

Setting up Criteria for Testing

Entry and exit criteria: The entry criteria for a test specify threshold criteria for each phase
or type of test. The completion/exit criteria specify when a test cycle or a testing activity
can be deemed complete.
Pass or Fail criteria- Specify the criteria that will use to determine whether a test item
has passed or failed testing
Suspended criteria - Specify criteria to be used to suspend the testing activity.
Resume criteria- Specified testing activity which must be redone when testing is
resumed.

Identifying Responsibilities, Staffing, and Training Needs

A testing project requires different people to play different roles. The different role
definitions should Ensure there is clear accountability for a given task, so that each person
knows what he or she has to do; Clearly list the responsibilities for various functions to
various people, so that everyone knows how his or her work fits into the entire project;
Complement each other, ensuring no one steps on an others’ toes; and Supplement each
other, so that no task is left unassigned.

Staffing
Staffing is done based on estimation of effort involved and the availability of time for
release. People are assigned to tasks that achieve the best possible fit between the
requirements of the job and skills and experience levels needed to perform that function.
In case there are gaps between the requirements and availability of skills, they should be
addressed with appropriate training programs.
It is important to plan for such training programs upfront as they are usually are de-
prioritized under project pressures.

Identifying Resource Requirements

The following factors need to be considered:
Machine configuration needed to run the product under test.
Overheads required by the test automation tool, if any.
Supporting tools such as compilers, test data generators, configuration management
tools, and so on. The different configurations of the supporting software that must be
present. Special requirements for running machine-intensive tests such as load tests and
performance tests. Appropriate number of licenses of all the software.

Identifying Test Deliverables

The deliverables include the following, all reviewed and approved by the appropriate
people.
The test plan itself (master test plan, and various other test plans for the project)
Test case design specifications
Test cases, including any automation that is specified in the plan

6
Software Testing (22518) CO5I

Test logs produced by running the tests

Test summary reports

Testing Tasks: Size and Effort Estimation

Size estimate quantifies the actual amount of testing that needs to be done. Effort
estimate is given in person days, person months, or person years. The effort estimate is
then translated to a schedule estimate.

Advantages of test planning are:

1. Work involved in test planning and setup pays in the long term. It gives insight testing
activity completely. One knows scope and deliverables of test plan execution.
2. Test plan describes the way in which testing team will show whether software work
correctly as per requirements and the acceptance criteria as defined by customer or
development team with customer. It defines various objectives for testing to measure its
performance and coverage offered.
3. Test plan addresses various levels of testing such as unit testing module testing, System
testing, integration testing, black box testing as well as white box testing. Some time
there may be single master test plan with several child test plan at each level for a
number of a small plans or one monolithic test plan covering every aspect of testing.
4. Test plan explain who does testing. Why test are performed how test are conducted
and
when tests are scheduled (calendar date and milestone). It defines various criteria such
as entry criteria, exit criteria, suspension criteria and resumption criteria at various
stages of testing.
5. Test plan must contain procedures, environment and tools necessary to implement an
orderly, controlled process for test execution, defect tracking, coordination of rework
and configuration, and change control.

Test Management
An important part of software quality is the process of testing and validating the software.
Test management is the practice of organizing and controlling the process and artifacts
required for the testing effort.
Traditional tools used for test management include: Pen and paper, Word processors,
Spreadsheets.
Larger testing efforts usually built on spreadsheets or databases, or commercial test
management applications. Generally, test management allows different teams to plan,
develop, execute, and assess all testing activities.

Choice of Standards
Standards comprise an important part of planning in any organization.
Standards are of two types—external standards and internal standards.

7
Software Testing (22518) CO5I

Internal standards
Naming and storage conventions for test artifacts.
Document standards
Test coding standards
Test reporting standards.

Naming and storage conventions for test artifacts

Every test artifact (test specification, test case, test results and so on) have to be named
appropriately and meaningfully.
It enables Easy identification of the product functionality, Reverse mapping to identify
the functionality corresponding to a given set of tests.
E.g. modules shall be M01, M02. Files types can be .sh, .SQL.

Documentation standards:
Appropriate header level comments at the beginning of a file that outlines the functions
to be served by the test. Sufficient inline comments, spread throughout the file, Up-to-
Date change history information, reading all the changes made to the test file.

Test coding standards:

Enforce right type of initialization, Stipulate ways of naming variables, encourage
reusability of test artifacts and Provide standard interfaces to external entities like
operating system, hardware and so on.

Test reporting standard:

All the stakeholders must get a consistent and timely view of the progress of tests. It
provides guidelines on the level of details that should be present in the test report, their
standard formats and contents.

External Standards:
These are the standards made by an entity external to an organization.
These standards are standards that a product should comply with, are externally visible
and are usually stipulated by external parties.

The three types of external standards are:

1. Customer standard: refer to something defined by the customer as per his/her
business requirement for the given product.
2. National Standard: refer to something defined by the regulatory entities of the
country where the supplier /customer resides.
3. International Standard: are defined at international level and these are applicable
to all customers across the globe.

8
Software Testing (22518) CO5I

Test Infrastructure Management

The testing infrastructure is used to support automated and manual, software testing, It
consists of the testing activities, events, tasks and processes. More stability, continuity
and reliability are provided to the automated testing process using stronger
infrastructure and its good management. Testing requires a robust infrastructure to be
planned upfront.
This infrastructure is made up of three essential elements.
A test case database (TCDB)
A defect repository
Configuration management repository and tool

A test case database captures all the relevant information about the test cases in an
organization.
Entity Purpose Attributes
Test case Records all the “static” information about the Test case ID
tests Test case name (file name)
Test case owner Associated
files for the test case
Test case - Provides a mapping between the tests & the Test case ID Module ID
Product cross- corresponding product features; enables
reference identification of tests for a given feature.
Test case run Gives the history of when a test was run and Test case ID Run date Time
history what was the result; provides inputs on taken Run status
selection of tests for regression runs (success/failure)
Test case— Gives details of test cases introduced to test Test case ID
Defect cross- certain specific defects detected in the Defect reference (points to a
reference product; provides inputs on the selection of record in the defect repository)
tests for regression runs

A defect repository captures all the relevant details of defects reported for a product.
Entity Purpose Attributes

Defect details Records all the Defect ID

“static” information Defect priority/severity Defect description
about the tests Affected product(s)
Any relevant version information
Environmental information (for example, OS
version)
Customers who encountered the problem
Date and time of defect occurrence
Defect test Provides details of Defect ID
details test cases for a given Test case ID
defect. Cross-
references the TCDB

9
Software Testing (22518) CO5I

A defect repository captures all the relevant details of defects reported for a product.
The defect repository is an important vehicle of communication that influences the work
flow within a software organization.

Entity Purpose Attributes

Fix details Provides details of fixes for a given defect; Defect ID

cross-references the configuration Fix details
management repository (file changed, fix release
information)
Communication Captures all the details of the communication Test case ID
that transpired for this defect among the Defect reference #
various stakeholders. These could include Details of communication
communication between the testing team and
development team, customer communication,
and so on. Provides insights into effectiveness
of communication

Software configuration management (SCM) repository

An SCM repository also known as (CM repository) ,Software Configuration Management
is defined as a process to systematically manage, organize, and control the changes in the
documents, codes, and other entities during the Software Development Life Cycle, keeps
track of change control and version control of all the files/entities that make up a software
product, A particular case of the files/entities is test files.
Change control ensures that
Changes to test files are made in a controlled fashion and only with proper approvals.
Change are made by one test engineer are not accidently lost or overwritten by other
changes. Each change produces distinct version of the file that is re-creatable at any
point of time. Everyone gets access to only the most recent version of the test files.

10
Software Testing (22518) CO5I

TCDB, defect repository, and SCM repository should complement each other and work
together in an integrated fashion. For example, the defect repository links the defects,
fixes, and tests. The files for all these will be in the SCM. The meta data about the modified
test files will be in the TCDB.

Test People Management

An application or product success is high just because of use of efficient and effective
testing techniques. Bugs/defects are found based on the skills and knowledge of a tester
and the dedication of the test teams. A test team is formed of people with varying skill
and experience levels.
Team members come up with different expertise levels, Different attitudes, Different
expectations, Interests' levels.
Maximize quality all these factors have to be integrated correctly, Need to work together,
Follow the applied test processes, We liver the work within specified schedule.

TEST PROCESS
Putting Together and Baselining a Test Plan
The test plan is reviewed by a designated set of competent people in the organization.
Approved by a competent authority, who is independent of the project manager directly
responsible for testing. After this, the test plan is baselined into the configuration
management repository. From then on, the baselined test plan becomes the basis for
running the testing project. Baseline is a formal document which acts as a base document
for future work

Test Case Specification

The basis for preparing individual test cases. Test case specification should clearly identify
The purpose of the test: This list what feature or part the test is intended for. The test
case should follow the naming conventions that are consistent with the module being
tested. Items being tested, along with their version/release numbers as appropriate.
Environment that needs to be set up for running the test case:
This can include the hardware environment setup, supporting software environment
setup (for example, setup of the operating system, database, and so on), setup of the
product under test.
Input data to be used for the test case: The choice of input data will be dependent on the
test case itself and the technique followed in the test case.
Steps to be followed to execute the test: If automated testing is used, then, these steps
are translated to the scripting language of the tool. The expected results that are
considered to be “correct results.”
A step to compare the actual results produced with the expected results: This step should
do an “intelligent” comparison of the expected and actual results to highlight any
discrepancies.

11
Software Testing (22518) CO5I

Any relationship between this test and other tests: This can be in the form of
dependencies among the tests or the possibility of reuse across the tests.

Update of Traceability Matrix

Matrix which is associates the requirements to its work product and test cases. Every test
case should associate to a requirement and every requirement should associate with two
or more test cases. Traceability Matrix is a tool to validate that every requirement is
tested or not. The traceability matrix is created during the requirements gathering phase
itself by filling up the unique identifier for each requirement.

Identifying Possible Candidates for Automation

Criteria used in deciding which scripts to automate include Repetitive nature of the test;
Effort involved in automation; Amount of manual intervention required for the test; and
Cost of automation tool.

Developing and Baselining Test Cases

The development of test cases entails translating the test specifications to a form from
which the tests can be executed. If a test case is a candidate for automation, then, this
step requires writing test scripts in the automation language. If the test case is a manual
test case, then test case writing maps to writing detailed step-by-step instructions for
executing the test and validating the results.

Executing Test Cases and Keeping Traceability Matrix Current

During test design and execution, the traceability matrix should be kept current.
As and when tests get designed and executed successfully, the traceability matrix should
be updated.

Collecting and Analyzing Metrics

When tests are executed, information about test execution gets collected in test logs and
other files.

Preparing Test Summary Report

At the completion of a test cycle, a test summary report is produced.
This report gives insights to the senior management about the fitness of the product for
release.

Recommending Product Release Criteria

The job of the testing team is to articulate to the senior management and the product
release team
• What defects the product has;
• What is the impact/severity of each of the defects; and
• What would be the risks of releasing the product with the existing defects.

12
Software Testing (22518) CO5I

• The senior management can then take a meaningful business decision on whether
to release a given version or not.

TEST REPORTING
Test reporting is a means of achieving this communication. There are two types of reports
or communication that are required: test incident reports and test summary reports (also
called test completion reports).

Test incident report

A test incident report is a communication that happens through the testing cycle as and
when defects are encountered. A test incident report is nothing but an entry made in the
defect repository. Each defect has a unique ID and this is used to identify the incident.
The high impact test incidents (defects) are highlighted in the test summary report.

Test summary report

A report that summarizes the results of a test cycle is the test summary report.
There are two types of test summary reports.
Phase-wise test summary, which is produced at the end of every phase
Final test summary reports (which has all the details of all testing done by all phases and
teams, also called as “release test report”)

A Summary report should content:

1. Test Summary report Identifier
2. Description: Identify the test items being reported in this report with test id
3. Variances: Mention any deviation from test plans, test procedures, if any.
4. Summary of results: All the results are mentioned here with the resolved incidents
and their solutions.
5. Comprehensive assessment and recommendation for release should include Fit
for release assessment and recommendation of release

Preparing Test summary report

Step #1: Purpose of the document
<Short description about the objective of preparing the document>
Example: This document explains the various activities performed as part of the Testing
of ‘ABCD transport system’ application.

Step #2: Application Overview

<Brief description of the application tested>
Example: ‘ABCD transport system’ is a web-based Bus ticket booking application. Tickets
for various buses can be booked using the online facilities. There are several modules like
Registration, Booking, Payment and Reports which are integrated to fulfill the purpose.

13
Software Testing (22518) CO5I

Step #3: Testing Scope

In Scope
Out of Scope
Items not tested
<This section explains about the functions/modules in scope & out of scope for testing;
Any items which are not tested due to any constraints/dependencies/restrictions>

a) In Scope: Functional Testing for the following modules are in Scope of Testing
Registration
Booking
Payment
b) Out of Scope: Performance Testing was not done for this application.
c) Items not tested
Verification of connectivity with the third-party system ‘Central repository
system’ was not tested, as the connectivity could not be established due to some
technical limitations.

14
Software Testing (22518) CO5I

Step #4: Metrics

<Metrics will help to understand the test execution results, the status of test cases &
defects etc. >
Required Metrics can be added as necessary.
Example: Defect Summary-Severity wise; Defect Distribution-Function/Module wise;
Charts/Graphs can be attached for better visual representation

Step #5: Types of testing performed

Smoke Testing
System Integration Testing
Regression Testing
<Describe the various types of Testing performed for the Project.
This will make sure the application is being tested properly through testing types agreed
as per Test Strategy.>
Example:
a) Smoke Testing
This testing was done whenever a Build is received (deployed into Test environment) for
Testing to make sure the major functionality is working fine; Build can be accepted and
Testing can start.
b) System Integration Testing
This is the Testing performed on the Application under test, to verify the entire
application works as per the requirements. Critical Business scenarios were tested to
make sure important functionality in the application works as intended without any
errors.
c) Regression Testing
Regression testing was performed each time a new build is deployed for testing which
contains defect fixes and new enhancements if any. Regression Testing is being done on
the entire application and not just the new functionality and Defect fixes.

Step #6: Test Environment & Tools

<Provide details on Test Environment in which the Testing is carried out. Server,
Database, Application URL etc. If any Tools were used like Quality Center (now HP ALM)
for logging defects>

Step #7: Lessons Learned

<This section is used to describe the critical issues faced and their solutions (how they
were solved during the Testing).

15
Software Testing (22518) CO5I

Lessons learned will help to make proactive decisions during the next Testing
engagement, by avoiding these mistakes or finding a suitable workaround>
Example:

Step #8: Recommendations

<Any workaround or suggestions can be mentioned here>

Step #9: Best Practices

<There will be a lot of activities done by the Testing team during the project.
Some of them could have saved time, some proved to be a good & efficient way to work,
etc. These can be documented as a ‘Value Add’ to showcase to the Stakeholders>

Step #10: Exit Criteria

<Exit Criteria is defined as a Completion of Testing by fulfilling certain conditions like
(i) All planned test cases are executed;
(iI) All Critical defects are Closed etc.>
Example:
a) All test cases should be executed – Yes
b) All defects in Critical, Major, Medium severity should be verified and closed – Yes.

Step #11: Conclusion/Sign Off

In this section exit criteria for an application is set and checked to release it. Testing team
verifies the application against set criteria. If it is not met the decision about its release is
taken by management team.

Step #12: Definitions, Acronyms, and Abbreviations

<This section mentions the meanings of Abbreviated terms used in this document and
any other new definitions>

Important Questions: -
1. Enlist any two activities involved in test planning.
2. State the contents of ‘Test Summary Reports’ used in test reporting.
3. Describe the process of preparing summary report in test planning.
4. Design a test plan along with the test cases for edit function in notepad.
5. Describe standards included in Test management.
6. Give any four advantages of test planning.
7. What is Test plan? List test planning activities.
8. Describe test reporting in detail. How to prepare a summary report?

16
Software Testing (22518) CO5I

9. Explain test deliverables in detail.

10. What is test planning and test management?
11. What is test plan? Write its two advantage.
12. How to prepare test plan?
13. Describe test case specification of test process.
14. Describe Test Management Process and give details of following internal
standards for process and method: (i) Naming and storage contention. (ii)
Documentation standard.
15. What are types of test report? Write contents of test summary report.
16. What are the contents of ‘Test Summary Report’ used in test reporting?
17. Explain in detail, how to prepare a test plan with suitable example.

17