OmniTestingConf: Taking Test Automation to the Next Level

OmniTesting
Sérgio Freire
Solution Architect
& Testing Advocate
@darktelecom
www.sergiofreire.com

Thanks dad…
You gave me
life, which
gives me
hope

@darktelecom
Agenda
3
Cutting-edge testing
1. Agile testing
2. Exploratory testing
3. Holistic testing
4. OmniTesting
Test automation
1. “Continuous testing tools”
2. Quality, testing and test automation
3. Experiences and lessons
01 02

@darktelecom
Testing, my perspective (2020)
Testing is a targeted process of challenging something (e.g. a product) and
our understanding of it, using a set of activities leveraged by specific
human skills.
Sergio Freire et al
4

@darktelecom
Testing, my perspective (today, i.e. 2021)
Testing is a targeted yet exploratory process of challenging something (e.g.
an idea, a product, its assets and its usage) and our understanding of it,
using a set of activities leveraged by specific human skills, augmented by
tools.
Testing is amplified with knowledge, collaboration and tooling to enable
consistent delivery of value, so teams can embrace change with lower risk.
Sergio Freire et al
5

@darktelecom
Agile Testing
Collaborative testing practices that occur continuously, from inception to
delivery and beyond, supporting frequent delivery of value for our
customers. Testing activities focus on building quality into the product,
using fast feedback loops to validate our understanding. The practices
strengthen and support the idea of whole team responsibility for quality.
Lisa Crispin & Janet Gregory
6

@darktelecom
Holistic Testing
When we test, we need to consider all types of testing, not only the ones we think a tester is responsible for. It includes
automation, exploratory testing, or any other type of human-centric testing. It involves the whole team, the product
organization, and even the customer. We need to consider testing from a holistic point of view.
Janet Gregory
8

@darktelecom
OmniTesting
• Intrinsic
• Team
• Throughout / “Unshiftable”
• Complete and unbound
• Multiple-approaches
• Multiple-techniques
• Multiple-levels
• Multiple-types
• Tooling
• Dynamic
9

Test Automation
Image from Unsplash

Continuous Testing
Tools?
Image from Unsplash

@darktelecom
Value
the importance, worth, or usefulness
of something
principles or standards of behaviour;
one's judgement of what is important
in life
Feeling
an emotional state or reaction
an attitude or opinion
VS
14

@darktelecom
Quality in simple words
What makes people ”happy,” in the team and outside of it
Who?
• End users
• Team
• Organization
Image from Unsplash

@darktelecom
Quality is multidimensional
• Multiple quality criteria
• Time dependent
Image from Unsplash

@darktelecom
Software delivery performance
17
Lead Time — the average amount of time it takes from the
time code is checked in to the version control system, to the
point in time where it is deployed to production.
Deployment Frequency — the number of times deploys to
production occur in a period of time.
Tempo
Mean Time to Restore (MTTR) — how long it takes to resolve, or
rollback, an error in production.
Change Fail Percentage — what percentage of changes to
production (software releases and configuration changes) fail.
Stability, resilience, recoverability, upgradability
https://www.holistics.io/blog/accelerate-measure-software-development/

@darktelecom
“Quality Ice Cream Truck”
19
testing
coding
quality
value
risks
shapes
informs on
informs on
threaten
increases
informs on
affects
informs on
defines
observ
ability
operat
ion
highlights
informs on
gaps affects
informs on,
affects
influences
informs on
opport
unities
shapes
gives clues
be confirmed? inspects
become
affects
confirms
Based on previous work from Rich Rogers and David Greenlees

Test automation is
like taking a train
Image from Unsplash

Orange vs grapefruit
Image from Unsplash

“The rotten apple”
DON’T start by implementing a ton of UI/Selenium+Gherkin based tests…
• Costly to run and feedback takes a while
• Do not help in refactoring, do not help the team
• Flaky
Image from Unsplash

@darktelecom
This requirement for the sensor is UNCOVERED
• This requirement has no linked Tests
... is OK on v1.0
• All Tests related with that requirement are PASSING for v1.0
... is NOK on v2.0
• Some Test(s) related with that requirement are FAILING for v2.0
... is NOK on v2.0 in environment X (e.g. Chrome)
• Some Test(s) related with that requirement are FAILING for v2.0 in environment X
... is OK on v2.0 in environment Y (e.g. Safari)
• Some Test(s) related with that requirement are PASSING for v2.0 in environment Y
Example: Coverage calculation in Xray
Coverage as a
heuristic that
depends on
version and/or
environment
23

@darktelecom
Test Run
step statuses
Status of
Test Run
Coverage
status* of
requirement
Test status*
(*) depends on scope/context
Example: Coverage calculation in Xray
24

@darktelecom
Another example: my personal blog
• Performance
• Accessibility
• SEO
• Mobile friendly
• Security
25

@darktelecom 26
Embedding static
analysis on the PRs (and
even on the IDEs) is a
simple yet effective way
of having more quality in
the code in terms of
reliability, security,
maintainability, …

@darktelecom
Test automation as a silver bullet
• Complete test automation doesn’t exist
• Perfection is impossible and an enemy of improvement
• Feel the pains and use TA as a “vaccine”
28

@darktelecom
The house
Image from Unsplash

@darktelecom
Can I enter the house?
Can I open/close the door?
Image from Unsplash

@darktelecom
Does it support load?
Image from Unsplash

@darktelecom
How secure is it?
Image from Unsplash

@darktelecom
What can and should we
automate? And how?
Image from Unsplash

Wrapping it up
Image from Unsplash

Testing is not automatable
You need test automation
Test automation needs you

Thank you!
getxray.app | mail@getxray.app | @XrayApp
@darktelecom

OmniTestingConf: Taking Test Automation to the Next Level

More Related Content

OmniTestingConf: Taking Test Automation to the Next Level