The promise of DevOps is that we can push new ideas out to market faster while avoiding delivering serious defects into production. Andreas Grabner explains that testers are no longer measured by the number of defect reports they enter, nor are developers measured by the lines of code they write. As a team, you are measured by how fast you can deploy high quality functionality to the end user. Achieving this goal requires testers to increase their skills. It’s all about finding solutions—not just problems. Testers must transition from reporting “app crashes” to providing details such as “memory leak caused by bad cache implementation.” Instead of reporting “it’s slow,” testers must discover “wrong hibernate configuration causes too much traffic from the database.” Using three real-life examples, Andreas illustrates what it takes for testing teams to become part of the DevOps transformation—bringing more value to the entire organization.
1 of 90
Download to read offline
More Related Content
DevOps: Find Solutions, Not More Defects
1. @Dynatrace
- More on http://blog.dynatrace.com
- Dynatrace Free Trial: http://bit.ly/dttrial
Dev(T)Ops
Find Solutions, Not
more Defects
Hosted by: Andreas Grabner - @grabnerandi
7. Waterfall -> Agile: 3 Years
220 Apps: 1 Deployment per Month
“EVERYONE can do Continuous Delivery”
“EVERY MANUAL TESTER DOES AUTOMATION!”
16. High performers are more agile
30x 8,000x
more frequent
deployments
faster lead times
than their peers
Source: Puppet Labs 2013 State Of DevOps: http://puppetlabs.com/2013-state-of-devops-infographic
17. High performers are more reliable
Source: Puppet Labs 2013 State Of DevOps: http://puppetlabs.com/2013-state-of-devops-infographic
2x 12x
the change
success rate
faster mean time
to recover (MTTR)
54. Distance calculation issues
480km biking
in 1 hour!
Solution: Unit Test in
Live App reports Geo
Calc Problems
Finding: Only
happens on certain
Android versions
59. 26.7s
Execution Time 33! Calls to the
same Web
Service
171! SQL Queries through LINQ
by this Web Service – request
similar data for each call
Architecture Violation: Direct access
to DB instead from frontend logic
60. 21671! Calls to Oracle
3136! Calls to H2
mostly executed on
async background
threads
33! Different
connections used
DB Exceptions on both
Databases
DB Exceptions on both
Databases
40! internal Web
Service Calls that
do all these DB
Updates
61. #2: Thread Column Tells me how many and which
physical Thread is involved in every step along the
execution flow
#3: Breakdown Column Tells
me where Wait or Sync is
happening
#1: Link Nodes Tell
me whether the
service/thread call is
synchronous or
asynchronous
62. Key Metrics
6. # Service Calls
7. # of Threads
8. Sync and Wait Times
9. # SQL executions
10. # of SAME SQL’s
75. Putting it into Continuous Deployment
12 0 120ms
3 1 68ms
Build 20 testPurchase OK
testSearch OK
Build 17 testPurchase OK
testSearch OK
Build 18 testPurchase FAILED
testSearch OK
Build 19 testPurchase OK
testSearch OK
Build # Test Case Status # SQL # Excep CPU
12 0 120ms
3 1 68ms
12 5 60ms
3 1 68ms
75 0 230ms
3 1 68ms
Test & Monitoring Framework Results Architectural Data
We identified a regresesion
Problem solved
Exceptions probably reason for
failed tests
Problem fixed but now we have an
architectural regression
Problem fixed but now we have an
architectural regressionNow we have the functional and
architectural confidence
Let’s look behind the
scenes
76. #1: Analyzing each Test
#2: Metrics for each Test
#3: Detecting Regression
based on Measure
83. #1: Define your Metrics
# of Service Calls Bytes Sent & Received
# of Worker
Threads
# of Worker
Threads
# of SQL Calls, # of
Same SQLs # of DB
Connections
# of SQL Calls, # of
Same SQLs # of DB
Connections