Quality Assurance and Testing

Lecture # 7

1
 Test Planning - 1
It is difficult to manage the testing of large
systems, considering the numerous test
cases, problem reporting, and problem fixes
Just as the planning and management of a
project is necessary by estimating resources,
schedule, cost, and so forth, planning for
software testing must also be done

2
 Test Planning - 2
The planning lets you decide what types of
tests should be executed within the
constraints of time and budget
Test planning also includes the estimation
of time for each test case and test scenario

3
 Test Planning - 3
The mistake made by testers is not planning
the schedule upfront
Allocated test time is reduced, when
development is late
If test planning is done right, test plans
show the dependency of receiving code on
time to meet the testing schedule

4
 Test Planning - 4
A test plan acts as a service level agreement
between the test department and other
significant departments
A test plan identifies the test deliverables,
functions to be tested, and risks

5
 Test Planning - 5
A test plan communicates to the audience
The testing methodology
The resources
Hardware
Software
Manpower
The testing schedule
The process used to manage the testing project
6
 Test Planning Pre-Requisites
Obtain the names of all test analysts
Select the automated test tools
Decide on the environment for testing
Obtain sign-offs on service-level
agreements with other departments

7
 Characteristics of a Test Plan
Keep it simple and complete
Keep it current
Keep it accessible
Test plan should be frequently routed to
appropriate people for feedback and sign-
offs
Test plan can be written by test leader, test
coordinator, or test manager
8
Steps in Developing a Test Plan -
1
Depending on the features of a system,
determine which tests you need to perform
Once you have identified the types of tests
needed to test the system, plan how each of
these tests will be performed. Develop test
cases

9
Steps in Developing a Test Plan -
2
Use a Test Plan template
When the contents of the test plan are
finalized, conduct a review for
completeness of the test plan
Incorporate the feedback from the review in
the test plan

10
Steps in Developing a Test Plan -
3
Get approvals on the contents of the test
plan from managers of all the departments
that will be impacted by this new product,
for example, managers of Development,
Customer Support, Marketing, Operations,
Integrated Technology, Product Assurance,
Quality, and User Management

11
Steps in Developing a Test Plan -
4
Follow the test plan to ensure that everyone
on the test team uses the process outlined in
the test plan and if there are any exceptions
to this, the deviations are logged as
addenda to the test plan. For consistency,
these addenda should be approved by the
same group of individuals who approved
the original test plan
12
 Contents of a Test Plan - 1
Introduction
Provides a brief description of the system being
tested
Scope
Describes at a high level what will be tested
and what will not be tested in order to avoid
ambiguity

13
 Contents of a Test Plan - 2
Test Plan Strategy
Outlines the objectives for developing the test
plan and establishes the procedures for
conducting the tests
Identifies all the tests that will be performed to
check the robustness of the product
Contains brief description of each test

14
 Contents of a Test Plan - 3
Test Environment
Lists the hardware and software configurations required
to perform tests
Schedule
Contains detailed test schedule, identifying the start
dates and end dates for
Development of the test plan
Designing test cases
Executing test cases
Problem reporting
Developing the test summary report
15
 Contents of a Test Plan - 4
Control Procedures
Contains detailed instructions for problem
reporting
Whether an automated on-line system will be used
to record and report problems
Whether the problem fixes will be accepted daily or
once a week
How the problems should be prioritized, for
example 1, 2, or 3. (severity)

16
 Contents of a Test Plan - 5
Control Activities
Indicates when walkthroughs or reviews will be
conducted and list list the individuals who
would participate in these activities
Functions to Be Tested
Lists all the functions to be tested

17
 Contents of a Test Plan - 6
Functions Not to Be Tested
Lists all functions not to be tested, along with
the reason for not testing, for example, low
risk, code reused and previously tested with
100% satisfaction

18
 Contents of a Test Plan - 7
Risks and Assumptions
Lists assumptions taken into consideration
when developing the test schedule
Unit testing will be done prior to receiving code for
system testing
All testing personnel will have the knowledge of the
product and will be experienced testers
The computers will be available during all working
hours

19
 Contents of a Test Plan - 8
Deliverable
Lists what will be delivered at the end of test
phase
Test plan
Test cases
Test incidence report
Test summary report

20
 Contents of a Test Plan - 9
Test Tool
Identifies all test tools needed during testing.
For example, test coverage analyzers can be
used to provide information on the
thoroughness of one or more test runs of the
program. At the end of analysis process, the
analyzer produces a visual listing of program
with all unexecuted code flagged

21
 Contents of a Test Plan - 10
Approvals
Ensures that the plan is reviewed and approved by the
affected individuals
Exit Criteria
Identifies the exit criteria for the test phase. For
example:
No priority-one open problems
All functions identified in the requirement document are
present and working
No more than three priority-two problems open

22
 Test Cases
A test case describes how each test is to be
conducted and input/output details
Development of test cases assist in keeping track
of what is tested, when it is tested, and the
outcome of the test
If a defect is found in testing, a documented test
case makes it easy for the developer to re-create
the problem so that proper analysis can be done to
fix it
23
 Design of Test Cases - 1
There are no standards or simple rules for
designing test cases
Test cases should be designed for testing
each function, with expected results
In order to determine what will break the
system, the tester has to be creative and
want to break the system

24
 Design of Test Cases - 2
The test cases should be repeatable, I.e.,
when a test case is rerun, it should give the
same results each time, except in real-time
environment
Test cases should be organized and
developed around system requirements and
for key functions of the system

25
 Design of Test Cases - 3
The test design emphasizes
The type of test needed to test the requirements
How a test will run
How all requirements will be addressed by the
tests
Good test cases catch errors
Develop test cases that cover as many
related functions as possible
26
 Design of Test Cases - 4
A test case contains
Test case identifier
The name of the person responsible for
executing the test
The date the test was executed and the version
of the system tested
Description of the functions to be tested and a
list of actions to be carried out
27
 Design of Test Cases - 5
Input values
Output values (expected results). In most
cases, the success of a test depends on
obtaining the predefined expected results
Actual results (output received when the test
was executed). Indicate whether or not errors
were detected, and if so, complete a problem
report

28
 Characteristics of a Test Case

Repeatable Consistent

Destructive Detects Discrepancy

Manageable Results Can Be Traced

29
 Types of Test Cases
Functional Test Cases
Computation and Boundary Analysis Test
Cases
Cause-Effect and Logic Cases
Security Test Cases

30
 Test Case Development
Considerations
All valid system input must be accepted
All invalid system input must be rejected with
clear error messages
All functions including generation of reports
should be tested
Any reports generated by the system should be
tested and verified
Any interfacing systems must be invoked and
verified that the date is passed to and from the
interfacing system
31
 Automated Test Tools - 1
Usually support automated execution of
black-box tests
An automated test tool reads a test script,
plays the script, and compare the actual
results to the original or expected responses
They enable repeatability and consistency
of test execution as well as productivity

32
 Automated Test Tools - 2
Some tool suites also provide the ability to
distribute or simulate the execution of tests
over multiple processors, thus providing
performance and stress tests to measure the
systems ability to handle high volumes of
transactions and users

33
 Evaluation Criteria - 1
Compatibility
Does the tool support the platform(s) and
environment(s) of the system under test?
Usability
Does it provide the necessary functionality, and
are the testers able to make effective use of it
without a prohibitive learning curve?

34
 Evaluation Criteria - 2
Maintainability
Can changes to the system under test be
accommodated in the automated tests without
undue effort?
Manageability
Does the tool provide sufficient and meaningful
information to enable management of the test
library and measurement of the test results?

35
 Compatibility
Client-Level Compatibility
Communications-Level Compatibility
Host-Level Compatibility
Portability

36
 Usability
Functionality
Extensibility
Learning Curve

37
Maintainability

38
 Manageability
Test Library Management

39
 Test Results
Test results should be meaningful to inform
management of the status of the system under test
and its quality relative to expectations
Test tools should be reviewed to determine the
amount and quality of test result information;
analysis and reporting capabilities are either
included or can be made available

40
 Test Case Generation - 1
Programmatic creation of test case data, as
opposed to sampling or other techniques
Advantage: High volumes of test conditions
can be created
Disadvantage: Volume alone is not
guarantor of coverage or efficacy

41
 Test Case Generation - 2
Equivalence classes
Large number of test cases
More than one test case that exercises the same
path, or too few to cover all the paths
Cause-effect node graphing

42
 Vendor Selection - 1
Experience
What level of experience does the vendor offer
in the implementation of automated test tools?
Service
What services does the vendor offer for training
as well as implementation to assure that the tool
is properly deployed?

43
 Vendor Selection - 2
Support
How readily available and responsive is the
vendor to technical questions and issues?
Commitment
Is the vendor committed to the automated test
tools market or is it only one of several
offerings?

44
 Conclusion
Plan and manage test planning and
document them using the Test Plan
Keep the Test Plan a living document
Develop test cases for maximum coverage
Use automated test tools

45
 Reference
Inroads to Software Quality by Alka Jarvis
and Vern Crandall

46