Relieving the Testing Bottle Neck in Your Projects | cPrime + QASymphony

September 25, 2015
Introduction to
QASymphony for [INSERT
COMPANY NAME]
Relieving theTesting Bottle
Neck inYour Projects

AGENDA
• HowTesting Becomes a Bottleneck
• Removing the Bottleneck with a New Process
• Increasing ManualTesting Efficiency
• Improving AutomatedTesting Coverage & Reliability
• BuildTestability into the Application
• Managing anTesting Strategy
• Measuring a DevelopmentTeam on Quality
• Q&A (based on interest)

September 25, 2015
Introduction to
COMPANY NAME]
Why doesTesting
become a
Bottleneck?

Past development cycles often modeled the Rational Unified Process:
Source: http://www.psa-software.com/_img/_knowledge_center/rup.jpg
WHEREWE CAME FROM

If testers cannot test efficiently with predictable estimates of effort,
testing can become the bottleneck for development (less features
deployed):
TESTING CAN BECOME A BOTTLENECK
Design Code Test Deploy
Features that couldn’t be
tested, so not deployed

Given a fixed level of depth in the testing , we can test and deploy more
features but it will be less coverage
WHAT IFWETEST LESS?
Design Code Test Deploy
Reduced depth of testing
on features

If given the choice, we would have moved testing earlier in the cycle, but
we were constrained by many environmental factors:
WHY DOESTESTING NEEDTO BE LAST?
Environment
Creation
Code Merges
On-Premise
Prevalence
Desktop
Focus
Lack of
Collaboration
Off-Shore
Development

Many of our prior limitations have been replaced, based on macro trends
around technology and industry:
WE CAN MOVETHETESTING UP FRONT
• Containers have simplified the process dramaticallyEnvironment Creation
• Git has replaced Subversion as the industry standardCode Merges
• Cloud adoption is at an all time high, increased securityOn-Premise Prevalence
• Prevalence of Web, Mobile, Internet of thingsDesktop Focus
• Increase in teamwork, chat and collaboration technologyLack of Collaboration
• Shifts towards rural sourcing, onshoring of laborOff-Shore Development

Now that we have freed ourselves of past limitations, the process can be
shifted to one that aligns more with our needs:
Traditional Approach
Test-First Approach
HOWTHE PROCESS CAN LOOK NOW
Design Requirements Code Test Deploy
Design
(Automated)
Test
Code Refactor Deploy

Creating tests up front means code can be deployed almost right upon
completion, removing the testing bottleneck:
HOWTHE FUNNEL LOOKS NOW?
Create
Tests
Code Deploy

Moving testing to the front of the process is desirable, but many realities
of software teams restrict them from doing so:
• Lack of automation coverage for legacy features
• Lack of testability (API’s, UI frameworks) in legacy applications
• Lack of automation talent within the organization
• Finite project length (i.e. package software implementation)
• Regulatory compliance (need to document requirements)
For these organizations – we will also cover ways to make a traditional
requirements driven approach more efficient
FLIPPINGTHE FUNNEL MAY NOT BE POSSIBLE

September 25, 2015
Introduction to
COMPANY NAME]
For ManualTesters

Manual test execution can be optimized by implementing exploratory testing. A 2007
controlled study found that:
– Testing with test cases vs. exploratory testing take almost 7 times longer, due to the
amount of time needed to write the tests and report on them – TEST FASTER
– Testing with test cases vs. exploratory testing doesn’t find any more defects, and
does not miss many (if any) critical or severe defects in comparison to test case
testing – TEST BETTER
– Testing with test cases causes more false defect reports vs. exploratory testing –
TEST SMARTER
Study link: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.167.3696&rep=rep1&type=pdf
HOWTO OPTIMIZE MANUALTESTING

1. Parallel test planning, test design, and test execution
2. Specific yet flexible
3. Aligned towards investigation of potential opportunities
4. Values depth and attention to detail during testing
5. Fosters knowledge sharing and awareness
WHAT IS EXPLORATORYTESTING?

Unlike traditional testing techniques, planning, design, and execution happen concurrently,
allowing efficiencies of time as well as flexibility in approach
PARALLEL PLANNING, DESIGN & EXECUTION
Plan Design Execute Report
Plan
DesignExecute
Report

Exploratory testing provides a specific lens through which to perform testing – whether
that be a user persona, functionality, criteria (i.e. localization), etc.
However, it allows testers to use the tool as an end user would, not necessarily as the
product owner envisioned it
SPECIFIC,YET FLEXIBLE
Manual Scripted Testing
I tested the application as the script
prescribed
Exploratory Testing
I tested the application as the end user
would

Exploratory testing rewards testers who identify unknown areas of “opportunity” within
the application, as they are essential in maintaining a backlog of future test charters
INVESTIGATING OPPORTUNITIES
Manual Scripted Testing Exploratory Testing

Exploratory testing relies on knowledge sharing to reach full potential – developing testers
who understand the impact of more areas of the application allows them to identify more
areas of risk and opportunity
KNOWLEDGE SHARING
Plan
DesignExecute
Report
Transfer
Learning
Example Questions to Ask
• Have you seen this before?
• What am I not considering?
• Why would someone do this?
• How would you have tested this?

September 25, 2015
Introduction to
COMPANY NAME]
For Automated
Testers

Moving towardsTDD will also demand a more complete automated
testing strategy focused on more than just UI testing:
BUILDINGA COMPLETETESTING STRATEGY
UI Tests
Integration
Tests
Unit
Tests
UI Tests
Integration
Tests
Unit Tests

UI automation should be only be used for key features and functionality,
given the following limitations:
• SPEED – tests take much longer to run when waiting for pages to load,
objects to interact, etc.
• RELIABILITY – tests break frequently for reasons besides failing
functionality (like renamed or moved objects)
• MAINTENANCE – tests require frequent updating as UI elements are
changed, requiring more maintenance
• COST – many UI automation tools are expensive and based on
perpetual licensing
WHY UI AUTOMATION IS LIMITING

The ideal testing pyramid would look like the following:
THE IDEAL RATIOS
UI Tests
Integration
Tests
Unit Tests 70% of coverage, effort, maintenance
20% of coverage, effort, maintenance
10% of coverage, effort, maintenance

Focus on the missing box (es) in the AgileTesting Matrix - typically the
performance and load tests:
Source: http://image.slidesharecdn.com/agiletestingandtheroleoftheagiletester2008-09-26rev4-090615163750-phpapp01/95/agile-
testing-the-role-of-the-agile-tester-40-728.jpg?cb=1245084455
ADDING OTHER MISSINGTESTS

September 25, 2015
Introduction to
COMPANY NAME]
For Developers

Moving toTDD process moves testing activity from solely UI based to
focusing more testing at the services/API layer:
BUILDINGTESTABLE SOFTWARE
Need for Testability

Testing in the Unit Layer will require more mock objects to “unhinge”
features that are being tested, removing them from dependencies:
Source: http://zeroturnaround.com/wp-content/uploads/2015/12/PUZZLE-1-min.png
BUILDINGTESTABLE SOFTWARE

Many teams struggle to pick an automation framework, but the reality is
most frameworks are similar and differentiate in a few key areas:
PICKING AN AUTOMATION FRAMEWORK
•Does it support dev
centric languages
(i.e. Java) or easier
to use languages
(i.e. Ruby)
•Does it support the
concept of
features, or
stories?
•Does it have
features to support
large enterprises,
as well as small
teams?
•Is it more focused
on BDD, TDD, or
ATDD?
Dev Style Scalability
LanguageWorkflow

September 25, 2015
Introduction to
COMPANY NAME]
ForTest Managers

There is a myth that EVERY test should be automated immediately for a
team to be as agile as possible
It is impossible that creating an automated test would the most efficient
approach in all scenarios
Some examples:
• Tests being used one time
• Tests on difficult parts of an application
• Tests for parts of the system that change frequently
THE AUTOMATE EVERYTHING MYTH

A more balanced approach is to estimate the total cost of creating AND
maintaining a test over a course of its life vs the cost of manual testing:
AN ROI BASED APPROACH
Automation Manual
Script Creation Cost $500 $200
Script Maintenance Cost $200/year $50/year
Script Execution Cost $120/year $480/year
Cost after Year 1 $820 $730
Manual
Even
Automate

For teams that are looking to build automation coverage on legacy
applications, another approach might be to automate based on time saved
A SAVINGS BASED APPROACH
Test Case A Test Case B Test Case C
Manual Execution $500/year $300/year $400/year
Manual Maint. $200/year $100/year $200/year
Auto. Creation $300 $150 $480
Automation Maint. $100/year $50/year $100/year
Year 1 Savings $400 $250 $120

September 25, 2015
Introduction to
COMPANY NAME]
For Dev Managers

Agile promotes delivering software quickly, so development managers
often incentivize developers based on features, story points, or lines of
code delivered
However, this approach can often incentivize bad behavior – pushing
incomplete or unstable code over to testers, who are then responsible to
identify the issues developers missed
Development managers should incentivize their teams for building good
code quickly, which is understandable and easy to test and debug
MANAGING ENGINEERSTO QUALITY METRICS

Coupling Between Objects (CBO) – A measure of how closely or loosely
coupled methods are. 1-4 is good, higher than that may mean risk in
trying to refactor or resolve issues
Comment Ratio – Ratio of comments to total lines of code, indicating how
much the developer has documented the code for future debugging or
refactoring
Associated Critical Defects Ratio – Ratio of critical defects opened on
specific features divided by the effort (story points) for that feature. Often
normalized for feature complexity
SOME GOOD QUALITY METRICSTOTRACK

Kevin Dunne
kevindunne@qasymphony.com
Twitter: @KevinDunneQA
QUESTIONS?

Relieving the Testing Bottle Neck in Your Projects | cPrime + QASymphony

More Related Content

Relieving the Testing Bottle Neck in Your Projects | cPrime + QASymphony