Software Testing

UNIT V: Managing the Testing Process

Chapter Objectives
 Testing process management activities
 Project planning, budgeting and scheduling
 Testing team formation
 Reviews and monitoring of projects
 Risk management
 Metrics in testing phase
 Defect tracking
 Configuration management
 Testing process improvement
 Testing standards and information security testing
5.1 Management Commitment
Quality assurance program in an organization will succeed only when the top management
is committed to quality. Surprisingly, there are many organizations in which there is no
separate quality assurance team and testing is done very informally in these organizations.
The top management must provide the necessary support for the quality assurance activities
by issuing the necessary policy guidelines, and allocating the budget and manpower.
To start with, the top management needs to measure the current levels of performance,
identify the areas of improvement and establish quality goals.
The current levels of performance can be obtained by using the following mechanisms:
Customer Surveys: if a product has been supplied to a number of customers, the feedback
from the customers on the product quality can be obtained.
Historical Records: For the projects executed/products supplied earlier, the records or
bugs reported by customers after the software is delivered can be analyzed to calculate the
effort/money spent on maintenance and rework. This data reveals the quality of the
products supplied earlier.
The Quality Approaches being undertaken by Competitors: In a competitive
environment, if an organization has to survive and grow, it has to follow the market trends
by observing the quality approaches being undertaken by competitors. If all the competitors
are obtaining ISO 9000 or CMMI or SPICE certification, the management has to also
consider obtaining the certification, otherwise, the organization is no longer competing.
Industry Standards Benchmark: Organizations commit themselves to 'six sigma' quality,
'total quality management etc. These benchmarks can be also be considered for
implementation.
Own Management Criteria: Sometimes, the management may formulate its own criteria
for measuring the quality which can be reflected in:
 Large customer base
 Repeat orders from the same customer
 Productivity of employees which is reflected by the turnover per employee or profit
per employee etc.
After obtaining the current level of performance, the management has to identify the
drawbacks of the present process, set quality goals and commit itself to implement a quality
management system based on international standards.
B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R INAMDAR) Page 1
 Software Testing

A number of models have been developed by international bodies which are based on
industry best practices. The popular models are:
 Capability Maturity Model (CMM)/Capability Maturity Model Integration
(CMMI)---staged or continuous representations
 ISO 9001:2000
 ISO/IEC 12207
 ISO/IEC 15504
Only when the management is committed to quality, the employees can also be made to
commit themselves to quality. The management's commitment to quality is reflected by
creating a quality assurance/testing environment.
Specifically, the management has to:
 Issue a policy statement that indicates the management's commitment.
 Ensure that the necessary processes have been defined and followed.
 Provide the necessary infrastructure, manpower and budget for these activities.
 Provide an environment where there can be a free exchange of ideas and opinions-
good communication is a must for quality to succeed.
 Conduct periodic reviews to ensure that the quality processes are being followed
and improved continuously.
5.1.1 Organization Structure
For any software project to succeed, the project manager has to bring synergy amongst
various groups such as domain experts, development experts, marketing experts and quality
experts, as shown in Figure 5.1.

Figure 5.1: Synergy of expertise

In a traditional organization, a hierarchical structure is followed, where the project manager

will be in-charge of a project and he/she in turn will control the development and quality
teams. This hierarchical structure is not well suited in organizations in which the focus is
quality. A 'flat' structure is recommended where all individuals work together as a team
without any strict hierarchy.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R INAMDAR) Page 2

 Software Testing

For Example, in ISO 9001 quality management system, there should be a separate quality
assurance team and a Management Representative (MR) will be in-charge of the quality
assurance team. The MR will report directly to the top management/CEO. This
organization structure indicates that thrust is indeed being given to quality.
5.2 Testing Process Management
To deliver quality software, a process-oriented approach is a must. The philosophy behind
this approach is very simple: plan your work, track the work and after the work is
completed, do a postmortem analysis to find out what went right and what went wrong.
Based on the analysis, improve the process.
This is shown in Figure 5.2. If you follow this approach, you can deliver better product,
you can plan better next time and you will be better person. This approach is applicable to
everyone---engineers and managers. In fact, we can use this approach even in our daily
lives.
PLAN TRACK POST MORTEM

Better Plan, Better Product and Better Person

Figure 5.2 Quality improvement process

The same philosophy is reflected in PDCA (Plan Do Check Act) model. Here, there are
four steps to be followed:
 Plan: to carry out a detailed planning of your activity.
 Do: to carry out the activity.
 Check: to check whether the activity is going as per plan, also take measurements.
 Act: to take any mid course corrections, and also to improve the process.
5.2.1 Options for Managers
One of the important activities of a manager is to study the possible alternatives. If your
organization wants specific software, there are two alternatives: buy or develop. Buying a
Commercial off the Shelf (COTS) software product is a better option, if the software meets
your requirements is readily available from a commercial vendor. Otherwise, the software
has to be developed.
COTS Software
The advantage of choosing the COTS software option is that the software is readily
available and hence it can be bought and can be used without losing time. However, the
following, aspects need to be considered
 Selection of the software: This is based on the requirements. A comparison of the
requirements and the available features of the COTS software have to be done.
 Vendor selection: The vendor selection is based on a number of factors such as:
 reputation of the vendor
 support to be provided for the software
 whether local presence in your country or city is available
 licensing terms and conditions
 training to be provided
 Cost of the software, etc.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R INAMDAR) Page 3

 Software Testing

 Lack of possibility to make modifications: If you want some special features now
or at a later date, the vendor may not be ready to make the modifications to Suit
your needs. You need to accept the software as it is.
 Need for customization: Some software products such as ERP packages need lot
of customization after installation at your site. Sometimes, the customization cost
will be much more than the actual cost of the software. You need to probe into the
customization cost before taking a decision.
 Testing the COTS software: The software has to be tested to ensure that it meets
all your functional and non-functional needs. The detailed testing process has to be
worked out.
Suppose the top management has given you the responsibility to make a payroll software
package in your organization. If you have to buy COTS software, you have to get the
details of the payroll software packages available in the market and compare their
specifications with your requirements. You have to shortlist a few packages that meet your
requirements and carry out a detailed study of each package the detailed specifications,
reputation of the vendor, support provided by the vendor, the training to be given by the
vendor and the cost.
After selecting a particular vendor's package, you need to test the software. This testing
basically involves checking whether it meets your needs or not and whether it is reliable or
not. You cannot ask for any modifications to be done by the vendor as it is a generic
package being sold by the vendor.
Software Development
If COTS software is not available, you need to consider the option of development. The
development can be done by using in-house development team or development can be sub-
contracted. If the development is sub-contracted, you need to carry out the testing to ensure
that the software is as per your requirements. If the development is done in-house, you need
to have a separate testing team to test the software. You also have an option to sub-contract
the testing activity.
Suppose the top management has given you the responsibility to make a payroll software
package in your organization. If you decide to get it developed, you can either get it
developed by the in-house development team or subcontract the development work. In
either case, testing is your responsibility. Here, you need to test the software, and then give
the feedback to the development team so that the bugs can be rectified and finally suitable
software is delivered to you.
5.2.2 Testing Process Management Activities
A well-defined testing process is fundamental to develop the quality software. Generally, in
ISO 9000/CMM/CMMI certified organizations, an organization-wide testing process is
developed. The process should define the objectives of testing, methodologies to be used,
infrastructure and tools required, method of testing, work products of the process and the
reviews to be conducted during the testing phase. The process should also identify the
training needs of the people.
The manager in-charge of the testing process has to carry out the following activities:
 Planning the testing process, budgeting and scheduling.
 Aligning organization-wide testing process to the project.
 Forming the testing team.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R INAMDAR) Page 4

 Software Testing

 Providing the necessary infrastructure.

 Executing the test plan and regularly reviewing the work and monitoring the
progress.
 Managing the risks.
 Obtaining the test metrics.
 Tracking the defects.
 Carrying out configuration management.
 Doing a postmortem analysis, after the project is completed.
 Based on the postmortem analysis, improving the testing process.
5.3 Planning, Budgeting and Scheduling the Testing Phase
Figure 5.3 shows the activities in the planning phase. The testing requirements need to be
studied in detail and the objectives of testing, criteria for completion of testing need to be
worked out. Then effort/budget/time required for the testing phase can be estimated.
Testing requirements and test design activities should be carried out, in parallel, while the
development work is going on. The effort, time and budget required for testing are worked
out and based on the perceived risks (project risks and product risks), the effort estimation
is fine-tuned.

SRS Document Study Testing

Requirement

Test
Design

Estimation Of
1. Effort
2. Time
3. Budget

Risk Analysis
1. Process Risk
2. Product Risk

Prepare Test Plan

Document

Figure 5.3: Test Planning Process

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R INAMDAR) Page 5

 Software Testing

Test planning involves identification of the following:

 What are the testing objectives and the scope of testing?
 How to align the overall process to the project?
 What are the criteria for completion of testing?
 What types of testing have to be carried out?
 Which testing tools have to be used?
 What is the infrastructure required for testing?
 What are the work products to be generated during the testing phase?
 How much effort is involved in testing?
 How much money and time are required?
 What are the skills set required for the testing professionals?
 What is the team structure?
 What are the risks associated with the project and how to manage risk?
Testing Objectives: For every project, the test manager has to clearly identify the
objectives. The objectives can be:
 To find the defects and report to the QA/development team.
 To fix the defects as well as suggest how to improve the software
 To carry out a review of the source code for completeness, adherence to coding
standards/guidelines.
 To carry out non-functional testing such as for portability, security, safety,
conformance to international standards, accessibility, internationalization etc.
 To check whether the software meets the functional and interface requirements in
case of COTS software.
 If acceptance testing by the customer is assigned as a task to the testing team
acceptance testing is also an objective.
Criteria for completion of testing: It is very important to identify the criteria for
completion of testing at early stages of development. Remember that software testing is
never complete -- at some point, we need to say, enough testing, let's deliver. But this
decision has to be taken based on some criteria which need to be identified during the
planning stage.
Based on the testing objectives and criteria for completion of testing, you will be able to
approximately estimate the effort/cost required for testing in person months. This effort
estimate needs to be adjusted keeping in view the risks associated with the project. For
example, if you perceive that some people are likely to leave in the middle of the project,
then you would like to put additional people from the beginning of the project. In such a
case, you need to add the additional manpower cost to the previously estimated cost. If you
perceive a technology risk and want to appoint a consultant to provide guidance, you need
to add the consultant's fee to the estimates.
5.3.1 Test Plan
The test plan, containing the details of the testing process, has to be prepared during the
planning stage itself. This plan should contain all the details of required resources, the
testing approaches to be followed, the testing methodologies, the test cases etc. This section
will address these aspects of the test plan.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R INAMDAR) Page 6

 Software Testing

A template for the test plan is given in Form 5.1

Test Plan
Project name: _____________________________________________________
Estimated start date for testing: _______________________________________________
Estimated end date for testing: _______________________________________________
Actual start date for testing: _________________________________________________
Actual end date for testing: ___________________________________________________
Estimated effort in person hours/person months for testing: _________________________
Estimated budget for testing: _________________________________________________
Work products based on which tests
are designed (SRS document, design document etc.): ______________________________
Test set-up (including the hardware, software
environment), other peripherals required,
any special tools required and any test
equipment required (such as oscilloscopes etc.): __________________________________
Lest personnel and their responsibility
(Test manager, test leaders and test engineers):___________________________________
Training to be provided to the test engineers: ____________________________________
Types of testing to be carried out (functional,
Structural, alpha, beta, Gorilla, usability, field trial,
Performance, stress, accessibility, conformance etc.): ______________________________
Checklists to be used for static testing and dynamic testing:_________________________
Testing tools to be used: _____________________________________________________
Work products to be generated during the
testing phase (test case design document,
system test results document, code coverage
test results, acceptance test results document,
testing completion document): ________________________________________________
For each type of testing, the test cases have to be specified:_________________________
Defect report format and defect tracking mechanism: ______________________________
Format for defect reporting (incident logging)
S. No. ____________________________________________________________________
Defect found: ______________________________________________________________
Type of defect Classification of the defect (critical, major, minor): ____________________
Whether the defect is removed: ________________________________________________
Time for removal of defect: ___________________________________________________
Stage when the defect was injected: ____________________________________________
Stage when the defect was removed: ___________________________________________
Details of configuration management:
work products to be kept under configuration
control, the configuration management team: ____________________________________
Metrics to be collected: ______________________________________________________
Form 5.1 Test plan format

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R INAMDAR) Page 7

 Software Testing

5.4 Alignment of the Process to the Project

In organizations with ISO 9001/CMM/CMMI certification, organization-wide processes
will be defined. These are generic process definitions applicable for all the projects.
Alignment of the process to the project involves fine-tuning the testing process for the
project based on the overall testing process definitions. This alignment has to be done
keeping in view the following aspects:
 Whether the software is COTS software or custom-built.
 Whether the software is developed in-house or through a subcontractor.
 The software development life cycle chosen.
 Whether procedural languages (like C) have been used or object-oriented languages
have been used.
 The application domain (ERP, CRM, telecom software, networking software,
embedded software etc.).
 Whether development/testing models such as extreme programming and buddy
testing are to be used.
 Whether testing tools have to be used.
If organization-wide processes are not defined, a process specific to the project on hand can
be developed.
5.5 Team Formation
The quality of testing and as a result the quality of the product depends solely on the people
employed for testing. The manager may take excellent individuals with strong technical
background, but still the team may not perform well. Even if, all the players in a team are
excellent, they may not be match winners—be it cricket or football or testing. For the team
to succeed, each individual in the team has to play the different 'roles' needed to achieve the
goal. The Team Roles Concept was formulated by Dr. Meredith Belben can be applied
effectively to build good teams that achieve the goals. According to this concept, people
that can play different roles are required in the team. These roles can be divided into three
categories:
 Action-oriented roles: Shaper, implementer, completer/finisher.
 People-oriented roles: Coordinator, team worker, resource investigator
 Cerebral roles: Monitor, evaluator, specialist.
This simple concept needs to be kept in mind by the manager while forming a team. If the
team has all thinkers' and no 'implementers, then the work will never be completed.
Another model that is widely used for forming teams is based on Margerison-McCann
Team Management System (www.tms.com.au). There are 8 types of profiles defined.
Every individual will belong to one of the following profiles:
 Reporter-Adviser
 Creator-Innovator
 Explorer-Promoter
 Assessor-Developer
 Thruster-Organizer

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R INAMDAR) Page 8

 Software Testing

 Concluder-Producer
 Controller-Inspector
 Upholder-Maintainer
Based on a simple set of questionnaire, it is easy to assess and find out to which category a
person belongs. Team formation can be done by assessing the individuals' preferences and
matching the team requirements with these preferences.
For a testing team to succeed, the test engineers need to have the following skills:
 Good understanding of the application domain.
 Excellent knowledge of the software quality assurance principles.
 Thorough knowledge of software testing.
 Exposure to testing tools.
 Experience in designing test cases.
 Good oral and written communication skills.
 Good listening skills.
 Good inter-personal skills.
 An eye for detail.
 Excellent analytical abilities.
The skills required for test leaders/test managers are:
 Ability to form high-performance teams.
 Ability to identify strengths of the individuals.
 Good written and oral communication skills.
 Ability to resolve conflicts (that arise between testing team and development team;
development team and client).
 Mentoring of the test engineers.
 Motivate and inspire the test engineers to work with strict deadlines.
 Leadership qualities.
 Facilitate a group to achieve the goals.
The test manager's responsibilities are:
 Plan the testing process.
 Form the team and provide the training to the team members.
 Ensure that the necessary infrastructure and tools are available.
 Conduct periodic reviews and manage risk.
 Monitor the progress and communicate the status to all persons concerned.
 Coordinate with different groups.
 Resolve any conflicts that arise between individuals and groups.
 Identify and collect metrics data.
 Configuration management.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R INAMDAR) Page 9

 Software Testing

 Ensure that all the testing objectives are met.

 Keep improving the testing process.
 Ensure that each test engineer fulfills his/her responsibilities.
The test engineers' responsibilities are:
 Study the work products and work out the test specifications including the types of
tests to be done.
 Carry out test design.
 Test case design and generate test scripts.
 Document test cases.
 Test case execution and incident reporting.
 Report the progress to the manager/leader.
5.6 Infrastructure
The test manager has to ensure that the necessary infrastructure is available to the testing
team. The infrastructure includes:
 The working area with provision for refreshments and some recreation.
 Computers and networking.
 Communication facilities (telephone, Internet connectivity).
 Software testing tools.
 All process related documents.
 Work products of the project.
 Special testing equipment such as protocol analyzers, oscilloscopes, spectrum
analyzers etc. (for embedded software testing).
5.6.1 Testing Tools
Test engineers' productivity can be increased if testing tools are used. In most of the
projects, enough time is not allocated to the testing phase. As a result, testing is done in a
hurry and buggy software is delivered to the customer. If testing tools are used, the testing
time and budget can be reduced considerably and quality of the product can be improved.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R INAMDAR) Page 10

 Software Testing

A number of testing tools are available—both open source tools and commercial tools. The
classification of the tools is shown in Figure 7.4.
5.7 Reviewing, Monitoring and Risk Management
The manager expects everything to go as per the test plan. Invariably nothing will go as per
the plan. The manager has to constantly review the work of the individual test engineers,
monitor the overall progress and take corrective action. It is likely that there is a threat to
the project execution due to the risks. Risk control has to be done during the testing phase.
A very simple and effective tool for reviewing the progress of the project is through
Gantt chart, also known as bar chart shown in below Figure. In this chart, the important
activities are shown by bars. Each activity is represented by a bar along with the start date
and completion date. This bar chart is prepared at the time of planning and periodically, it
is checked whether the activities are going on as per the plan. This chart is generally used
by the top management for reviewing the activities.

Testing Tools Training Test Execution Acceptance

Testing

Test Case Design

Test Spec. & Design

T0 T0+1m T0+2m T0+3m T0+4m T0+5m

If the project manager also wants to keep track of the estimated effort for each activity
along with the time frame, the cost-schedule-milestone graph shown in below Figure is
used.
In this figure, X-axis represents the time and Y-axis represents effort in person months or
cost in dollars. On the graph, the various milestones are indicated such as test
specifications, test case design, test execution etc. At the time of planning, the estimated
values are plotted on the graph and while executing the testing phase, the actual values are
plotted. So, periodically the manager will know whether he is 'earning' or 'losing - in terms
of cost and time.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R INAMDAR) Page 11

 Software Testing

5.7.1 Risk Management

We need to take risks if we want to do business. But if we take blind risks then the project
will fail. Fundamentally, the project manager has to assess the risk associated with
releasing untested software to the client. During the planning stage, the various product
risks and project risks are identified. Suppose 'persons leaving in the middle of the project'
is identified as a risk item. If really a few people leave the project, the manager has two
options: (a) transfer some test engineers from other divisions or (b) take test engineers on a
temporary basis. If risk management is done well, the manager will not worry, because he
would have already put some additional people on the project anticipating the risk.
5.7.2 Test Reports
The manager has to release periodically status reports indicating the following:
 Types of tests carried out and pending tests.
 Percentage of requirements covered by testing so far.
 Code coverage achieved by testing so far.
 How much testing is still left out.
 Total effort, time and budget already spent.
 Whether testing will be completed as per schedule or likely to be delayed.
 Any issues to be resolved.
At the end of the testing phase, test report has to be generated by giving the following
details:
 Results of various tests carried out.
 What criteria have been used for declaring that testing is complete?
 Whether the software ready for delivery or not?
 Suggestion for improving the product quality.
 Actual time effort and money spent on testing.
 Various metrics collected.
 Statistical analysis of the defects (defect density defects removed per hour) etc.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R INAMDAR) Page 12

 Software Testing

5.8 Metrics
To effectively manage the testing process, the manager has to identify the metrics and
obtain the measurements. Some important metrics are as follows:
 Effort (in person months), time and budget estimated and the actual values after
completion of the testing phase. The reasons for the difference between estimates
and actual values need to be studied-- is it a problem with estimation or execution?
 The total number of lines of code, number of functions, number of classes in object
oriented design, number of tables and forms in a database, number of web pages in
a web site development etc (also called Project complexity metrics).
 Total number of defects found and defects found in each category (for example,
critical/major/minor defects).
 Root cause of defects (how many defects were due to problems in requirements,
problems in design, problems in coding etc.).
 Severity of defects and percentage of defects in each severity level.
 Defect removal efficiency (how many defects were removed per hour). This gives
an indication of the productivity of the test engineers
 Total number of test cases executed and the percentage of test cases that detected
defects.
 Reliability metrics (MTBF, MTTR and MTTF).
 Testing effort/time/budget as a percentage of total project effort/time/budget.
 Cost of quality (failure cost, appraisal cost, and preventive cost).
5.8.1 Software Reliability
A number of reliability analysis models are available to predict the reliability of the
software, in terms of the number of defects still remaining in the software. This metric is
very useful to decide when to deliver the product or when to release the product into the
market, particularly when the software is highly complex. The software reliability models
generally follow one of the trends as shown in Figure 5.7. Fig 5.7(a) shows the concave
model' and Figure 5.7(b) shows 'S-shaped model'. The assumption made in arriving at these
models is that during initial stages, testing is not as efficient as at later stages and hence
there is an increase in the rate of defect detection. In Figure 5.7(c), the two-stage model
curve is shown; this model is applicable when significant functionality is added to the
software during the testing process (which is normally the case).

Test Time Test Time Test Time

(a) Concave (b) S-Shape (c) 2-Stage
Figure 5.7 Software reliability curves

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R INAMDAR) Page 13

 Software Testing

The software reliability models are still at a theoretical level and studies are being carried
out at academic institutions. Very few organizations use them in practice for obtaining the
reliability figures. The most important reliability figures are MTBF (Mean Time Between
Failures), MTTR (Mean Time To Repair) and MTTF (Mean Time To Fail). The relation
among these three values is shown in Figure 7.8.

Figure 7.8 Relations among MTBF, MTTR & MTTF

While carrying out the testing process, the values of MTBF, MITR and MTTF need be
recorded and if these values are within the limits only, the product can be considered as
'ready for release".
Based on these models, the reliability estimation models can be evolved in the
organization. Organizations, which would like to improve their process based on
quantitative measures, need to apply the reliability models as a part of their quality
improvement process.
5.9 Defect Tracking
While executing the testing phase, every defect has to be recorded in a defect log (also
known as incident report). The test manger has to define a process for defect notification to
disposal of the defect. This process consists of the following steps:
Recognition of a defect: For a particular test case, if there is a difference between the
expected result and actual result, then it is an anomaly or defect and it has to be recorded.
This defect has to be recorded in a defect log. The test engineers can only enter the defect
into the log; they cannot remove data from the log.
Investigation as to why the defect occurred: The root cause of the defect has to be
identified in association with the developer, if required. The test manager has to study the
defect log, find out the severity of the defect, assign priority to its removal (high, low, can
be ignored for now) and identify the person who will remove the defect.
Defect removal: The defect has to be removed by the developer. When the defect is
removed, the test manager again needs to ensure that confirmation testing and regression
testing are done and confirm that the defect has been really removed and that defect
removal did not introduce any other new defect. Then the test manager can declare that the
particular defect is disposed off.
5.9.1 Classification of Defects
All defects are not of the same type. Some defects may result in system crash. Some
defects may be very minor (such as a spelling mistake in an error message). So, the defects
need to be classified based on its impact on the functionality of the software. While

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R INAMDAR) Page 14

 Software Testing

recording the defects, the severity of the defects also needs to be noted down. If a critical
defect is present in the code, then the software is not ready for delivery. The defects can be
classified as:
 Critical
 Major
 Minor
Critical defects result in a system crash, or the application may not be useable at all if this
type of defect occurs. In a time critical application, if the timing is not within the limits or
if the response time is not within limits; then also the defect can be termed as critical.
Inconsistent performance of the application is also a critical defect.
Major defects will not lead to a system crash, but may result in some portions of the
application difficult to use, such as difficult navigation of the menus.
Minor defects can be tolerated, such as lack of help for some functionality, a spelling
mistake in an error message, incorrect ordering of control buttons in a GUI etc.

Another way of classification of defects is based on functional/non-functional

requirements. Here, the categories can be:
 Functional defect
 Non-functional defect
 Security-related
 Safety-related
 External interface related
 Conformance related
 Portability related
 Usability related
 Testability related
For each category, severity can be assigned (the severity levels can be 3 or 5).
After analyzing the test report, the project manager needs to decide whether the software is
ready for delivery or not.
5.10 Configuration Management
Configuration management is an activity that spans across all phases of development.
However, it is during the testing phase, many defects are detected and as a result, the
request for change is initiated. In general, as shown in Figure 5.9, when a product is request
or released or when a mature product is given to the testing team for testing, a change
request may be initiated due to:
 Preventive maintenance: Anticipating a problem, change is being made so that the
software will not fail in future.
 Adaptive maintenance: Due to change in requirements, or due to the need to
enhance the features of the software, a change is being made.
 Corrective maintenance: A problem has been encountered in the software and a
change is being made to correct it.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R INAMDAR) Page 15

 Software Testing

Product
Preventive
Release
Maintenance

Preventive Change
Maintenance Request

Preventive
Maintenance Change Request analysis
[Impact Analysis]
Work
Release Product
Planning Change
Implementation
Change
Release
Work
Product
Figure 5.9 Change management

Whenever a change request is received, a detailed impact analysis is made and then it is
decided whether the proposed change should be implemented or not. If it is decided to
implement the change, the necessary changes are made not only in the code, but in all the
work products and the software is thoroughly tested and then released.

Before accepting a change, a thorough analysis of the impact of the change has to be done,
as shown in Figure 7.10. The impact of the change on both functional requirements and
non-functional requirements has to be studied by the technical team and the impact and the
budget, time and effort have to be studied by the management. If the impact is not very
high, or if the impact is high but the client is ready to foot the bill, then the proposed
change can be accepted.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R INAMDAR) Page 16

 Software Testing

In many projects, the managers innocently accept a change proposed by the customer,
when the customer tells that it is a 'minor change. Without studying the impact (or the
damage to be caused by the change), if the proposed change is accepted, and later on if the
change turns out to be major---it is going to affect the profits of the organization.
Particularly, when changes in performance parameters are requested by the client, the
development team needs to be extremely cautious.
5.11Test Closure and Process Improvement
When the testing phase is completed, the manager and all the team members would gain
lot of experience from the project. The experience helps in improving the testing process
further.
As shown in Figure 7.11 for every project, we define a process and set quality goals. We
carry out the testing activities as per the plan using the process definitions. Then we need to
compare the quality goals with the actual results. If there is no match, we need to study
where there is a scope for improvement. This postmortem analysis helps in finding our
weaknesses so that we can work on how to overcome those weaknesses.

Define a Set Quality Plan Testing Track the Compare Goals

Process Goals Activities Testing activities & results

Process Refinement

Figure 7.11 improving the testing process

Using the metrics, we can find out the deficiencies in our processes and people, and try to
rectify these deficiencies.
 If the testing phase could not be completed as per the estimated time frame or
budget or effort, the manager has to find out the reasons. Is it due to the problems in
estimation? If so, the estimation process has to be improved. Otherwise, if there are
problems in execution, then the execution process needs improvement.
 If it is realized that more defects are due to improper specifications/design, the
requirements engineering/design process has to be improved.
 If more defects were introduced in coding stage, the programmers need to be
trained on programming.
 If there are too many defects being reported by the customer, then the testing phase
is not executed well and hence the test manager needs to be trained.
 If the defect removal density is low, test engineers need training or more
sophisticated tools need to be introduced.
Continuous improvement of the testing process should be the aim of the management. We
need to appreciate that our success today is based on our past experience. Our experience is
based on our past failures. And, we need to learn from failures--in fact, we can learn more
from failures than successes.
5.11.1 Software Testing Maturity Model (SW-TMM)
Software Testing Maturity Model (SW-TMM) was developed Dr. IIene Burnstein of
Illinois Institute of Technology and her associates. This model is similar to CMM/CMMI
and can be used effectively by software testing organizations to improve their testing
processes systematically, stage by stage. In SW-TMM, there are 5 levels
B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R INAMDAR) Page 17
 Software Testing

Level 1: Initial
Level 2: Phase definition
Level 3: Integration
Level 4: Management and measurement
Level 5: Optimization/defect prevention and quality control.
Level 1 Organizations are characterized by:
 Chaotic testing activities.
 Testing is done on ad hoc basis.
 Testing is done just to prove that software works.
 Testing and debugging are not differentiated.
 Staff not trained in intricacies of testing.
Level 2 organizations are characterized by:
 Testing is recognized as a separate phase after construction /implementation
 Testing is considered as separate from debugging.
 Basic testing methods are followed.
 Testing is done to show that software meets the requirements.
Level 3 organizations are characterized by:
 Management considers testing as a professional activity and gives the necessary
support and thrust to testing phase.
 Staff is trained thoroughly on testing process.
 Formal testing processes for planning, reviewing and monitoring are in place.
 Tools are used for testing
 Testing objectives are defined on the basis of software requirements.
Level 4 organizations are characterized by:
 Metrics are used for managing the testing process (testing process is measured and
quantified).
 Quality parameters such as reliability, usability, security etc. are tested.
 Test cases are designed, documented and reused for regression testing.
 Defects are tracked.
Level 5: organizations are characterized by:
 Well defined and managed testing process.
 Testing effort and cost are planned and monitored.
 Testing tools are evaluated, selected and introduced based on a defined process.
 Testing process is continuously improved.
SW-TMM is a simple model that can be followed by every organization even if it is not
following CMM/CMMI. To start with, an organization has to assess itself at which level it
is at present based on the current process definitions. Then, it has to implement the various
processes to achieve Level 5.
B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R INAMDAR) Page 18
 Software Testing

5.12 Information Security

Nowadays, test engineers and test managers have to give lot of thrust to information
security. Using the software products and services, lot of information is stored in the
computers and this information has to be kept secure.
Security of information stored in the computers has become a major concern. The statistics
on cyber crime is very unpleasant:
 Almost every business or government organization is affected by attacks on
information systems one time or the other.
 More than 80% of the people who shop online are concerned about the credit card
fraud.
 Employees and ex-employees cause nearly 50% of the security breaches.
 Viruses such as Melissa, I Love You cause millions of dollars of information loss
across the world every year.
Keeping in view the importance of providing security to the information systems, a number
of international standardization bodies and government organizations took initiatives to
develop standards for developing secure information systems. All these initiatives aim to
provide a framework for 'security assurance’ similar to 'software quality assurance'.
Security assurance involves:
 To find out the flaws in the present security system by carrying out a rigorous
vulnerability analysis, also known as penetration testing.
 To define a security policy that indicates the management’s commitment to the
security aspects of information system, and ensures that each and every employee is
also committed to the security policy.
 To define security procedures and guidelines that needs to be followed across the
organization. These procedures include user management and password protection,
anti-virus software usage and updating, backup and restoration procedure, user
training on security aspects, auditing of the security system, and disaster recovery
and business continuity mechanism.
To provide security controls to information systems, we need to put in place Information
Security Management Systems (ISMS). The ISMS can be developed based on international
standards such as:
 In Common Criteria (ISO 15408) standard, a product is evaluated based on seven
evaluation levels. These levels are:
o EAL1: The product is tested for it functionality,
o EAL2. The product is tested structurally.
o EAL3. The product is methodically tested and checked.
o EAL4: The product is methodically designed, tested and reviewed.
o EAL5: The product is semi-formally designed and tested.
o EAL6: The product is based on semi-formally verified design and tested.
o EAL7: The product is based on formally verified design and tested.

B.L.D.E.A’S SSM POLYTECHNIC, CSE (S.R INAMDAR) Page 19