Testing Does Not Equal Quality
- 1. Testing != QualityRethinking quality and how to achieve it in an agile worldBob HartmanRichard Lawrence
- 2. “How does QA fit into all of this?”
- 3. Speed/valueand quality are conflicting goals
- 4. Or are they?
- 5. What is quality?How do we ensure our software has it?How can quality enable speed and value?
- 7. The goal of a software org:Produce valuable software now and in the future.
- 8. A feature can be modeled with a cumulative cash flow chart
- 9. Or, we can remove the investment and just look at value flow
- 10. Value flow for a feature that never works right
- 11. Value flow for a feature that fails for some users some of the time
- 12. Value flow for a feature with downtimes
- 13. Value flow for a feature that doesn’t scale
- 14. Quality is the foundation of value
- 15. A team’s productivity can be measured with a velocity chart
- 16. Velocity of a team with a well-designed, high quality system
- 17. Velocity of a team with a poorly-designed, low quality system
- 18. Quality is the foundation of speed
- 19. Photo credit: woodsy on sxc.huQuality is what ensures: Value now and in the future
- 20. How do we ensure our software has it?
- 21. Testing at the end doesn’t workIteration StartIteration EndPlanDevTestThe PlanThe Reality
- 22. What’s really going on?DevTestThe PlanDev to test is really a loopDevTestThe RealityDev expands to fill the available time
- 23. Build quality in vs. inspecting for it at endDevTestPlanInspect for qualityBuild quality in
- 24. Manual testing doesn’t scale(or happens in overtime)Testing that doesn’t happen, but shouldRegression testingTesting capacityNew feature testingSprint 1Sprint 2Sprint 3Sprint 4Sprint 5
- 25. Automate and get regression tests for freeAutomated tests that are now regression testsTesting capacityNew feature testingSprint 1Sprint 2Sprint 3Sprint 4Sprint 5
- 26. What is really being tested?IterationsFeatures
- 27. Which features need to work best?Usage of features in a typical software product
- 28. But without regression it is differentIterationsFeatures
- 29. Minimal regression is a little betterIterationsFeatures
- 30. But what will happen where we don’t have regression tests running?IterationsFeatures = Missed tests, which Murphy’s Law says will lead to defects being missed!
- 31. Build and deploy processescan kill speedDONEThe PlanDefine acceptance criteria and automate testsDevelopPackageDeploy to test envTestLoops, waits, errors…more timeThe Reality
- 32. To ensure quality in an agile world:Move testing to the front of the processCollaborateAutomate tests (using agile-friendly tools)Automate/accelerate build and deployWork on user stories one at a time, all the way to DONE
Editor's Notes
- Photo: Tug o’ war between speed/value and quality
- This is the structure/agenda for the rest of the presentation
- Briefly dig into “valuable” and “now and in the future.” Big idea: quality is about ensuring business value and speed/sustainability.
- Explain the cumulative cash flow chart, what it means. Wave at how quality might change its shape.
- The impact of quality on value is clearer if we just look at value over time non-cumulatively. Here’s an example of a value flow for a feature that works as intended, the counterpart to the cumulatively cash flow in the previous slide.
- straight line at 0 – there’s never any value
- Less value than the original value flow, and goes up and down over time.
- Value flow with gaps
- Increasing value flow with a cliff when the system gets overloaded.
- We’ve been looking at the value over time of a feature. We can also look at the value produced over time by a team. We normally call this “velocity.”
- Slightly increasing, pretty stable velocity
- Decreasing velocity, as tech debt increases the cost of new features over time.
- Quality is not something you add on top of value. It only makes sense in terms of value now and in the future, as revealed by feature value flows and team value flows. (Flip side of this: Anything we would do in the name of quality that doesn’t ensure value now and in the future is probably waste.)
- Inspect for quality: PO asks for something. Dev builds what they think PO wants. Test inspects for what they think PO wants. Loops ensue. Sometimes it works. More often, it doesn’t. Build quality in: Use tests as specs. Engage QA/testers with PO and dev at the start of a feature instead of just at the end.
- Moving testing to the front is good. But if our testing is all or mostly manual, we have a problem after just a few iterations.
- If you automate your tests as you build them, this iteration’s new tests become next iteration’s regression tests, but you only need to build them once. Also, you give your devs something they can run to ensure that their code passes all the tests before they call their work done. This reduces the loops we mentioned a moment ago.Note, however, automation can’t be much more expensive than manual testing or the whole thing falls apart. This drives tool choice. So does the need to write automated tests against not-yet-existent features. No time to talk about it here, but there’s a reason certain test tools are more popular in the agile community and others are more popular in the waterfall world: your tool can make or break your success with agile testing.
- Remember, features are in priority order, so the most valuable feature is the most tested feature.
- We’ve moved testing to the front and started automating our tests. But if it still takes a day or two to get code built and into a test environment, completing a story can take twice as long. If the process is manual and error-prone or if you regularly find defects post-deployment, now you have loops. Add in multitasked RM people who aren’t available when you need them, and your 2 day story easily takes 6-8. In a two week sprint, you’re not getting much done.