Data-Driven Software Mastery @Open Mastery Austin
•
1 like•597 views
What if we could measure the indirect costs of pain building up on a software project? What if we could measure the loss of productivity, the escalating costs and risks, and could steer our projects with a data-driven feedback loop? By measuring the friction in “Idea Flow”, the flow of ideas between the developer and the software, we can create a data-driven feedback loop for learning what works. Rather than making decisions based on anecdote and gut feel, we can start driving our improvement decisions with real data. Data-Driven Software Mastery is about learning and improving faster than ever. Find out how you can: • Identify the biggest causes of productivity loss on your software project. • Avoid spending tons of time solving the wrong problems • Collaborate with other industry professionals in the art of data-driven software mastery Idea Flow gives us a universal language for describing our experience, so we can share the patterns and principles of what works. With a feedback loop, we can run real experiments! Idea Flow turns the development community into a scientific community.
Report
Share
Data-Driven Software Mastery @Open Mastery Austin
- 1. Janelle Klein openmastery.org @janellekz Tour Speaker Data-Driven Software Mastery newiron.com
- 2. , Developer, Consultant, CTO @New Iron Specialized in Statistical Process Control (SPC) and Supply Chain Optimization from Lean Manufacturing (I <3 cool data) Continuous Delivery infrastructure, automation strategy & technical mentorship Janelle Klein Who Am I? I’m also a hobbyist Cognitive Scientist How to Measure the PAIN in Software Development Janelle Klein Author of “Idea Flow” leanpub.com/ideaflow
- 3. LEARN YOUR WAY TO AWESOME. Community of Practice for Data-Driven Software Mastery Why Are We Here?
- 4. Why Should You Care? Problem #1: Organizational Dysfunction
- 5. Why Should You Care? Problem #2: Solving the Wrong Problem
- 6. Why Should You Care? This talk is about a REALISTIC STRATEGY to solve these problems.
- 7. I’m going to show you: POINT A (I’ve been working on this for 5 years)
- 8. I Need Your Help to Get to: POINT AWESOME Point A Point AWESOME? Iterate!
- 9. LEARN YOUR WAY TO AWESOME. Why Are We Here? Learning Takes Work.
- 10. Let’s Make the PAIN Visible! #OpenDX Our Campaign:
- 11. This isn’t About Me. This is About ALL OF US.
- 12. Five Years Ago I Read This Book: How to Build a Learning Organization
- 13. Five Disciplines of a Learning Organization Personal Mastery Mental Models Shared Vision Team Learning Systems Thinking These disciplines were emergent practice on our team.
- 14. Five Disciplines of a Learning Organization Personal Mastery Mental Models Shared Vision Team Learning Systems Thinking These disciplines are emergent practice in software development
- 15. What’s the Gap? What was different about our team?
- 16. My Day Job is Technical Assessment GeorgeMe What does “better quality” mean to you?
- 17. What does “better” really mean?
- 18. What does “better” really mean? Where does my “better” come from?
- 19. “Better” Fifth Discipline: “Shared Vision” = “Shared Better” The Key to Team Learning: “Better”
- 20. Five Disciplines of a Learning Organization Personal Mastery Mental Models Team Learning Systems Thinking This book is about the art of group problem-solving. Shared LearningShared Better
- 21. Five Years to Put “Better” into Words How to Measure the PAIN in Software Development Janelle Klein This is the my story.
- 22. We were trying to do all the “right” things. About 8 Years Ago…
- 23. We were building a factory automation system...
- 24. We shipped to production...
- 25. We shipped to production... (again)
- 26. We couldn’t reproduce the problem.
- 27. We shipped to production... (AGAIN) The rollback failed!
- 28. Project FAILURE
- 29. The Retrospective “What are we going to do?!” Our biggest problem “Well, we know we’ve got a quality problem right?”
- 30. The Retrospective “What are we going to do?!” Our biggest problem “The problem is we don’t have enough test automation!”
- 31. So the Test Automation Began… Our regression testing took 3-4 weeks… Let’s automate the tests!
- 32. 20% Percent Automated: So the Test Automation Began…
- 33. 50% Maintenance started getting PAINFUL… Let’s refactor the tests! Percent Automated: So the Test Automation Began…
- 34. 80% It was still really PAINFUL… “Well, at least our regression cycle is faster, right?” Our regression cycle still took 3-4 weeks! Percent Automated: So the Test Automation Began…
- 36. ApplicationTests Our Test Suite Any ideas?
- 37. The Retrospective “What are we supposed to do?!” Our biggest problem “Well, if we don’t understand a problem, we should collect data.”
- 38. The Retrospective Our biggest problem “What data would help us understand the problem?” “What are we supposed to do?!”
- 39. Technical Debt Mistakes I thought the main obstacle was Technical Debt
- 40. ? Most of our mistakes were in the most well-written parts of the code. Mistakes
- 41. We made significantly more mistakes in code that we didn’t write ourselves. Lower Familiarity More Mistakes= There had to be more to the story...
- 42. Complex( So*ware( PAIN This is what I knew... What made development feel painful?
- 43. Unexpected Behavior Problem Resolved Tracking Painful Interaction with the Code (Friction) Troubleshooting Progress 5 hours and 18 minutes of troubleshooting... PAINFUL
- 44. The amount of PAIN was caused by… Likeliness(of(( Unexpected( Behavior( Cost(to(Troubleshoot(and(Repair( High(Frequency( Low(Impact( Low(Frequency( Low(Impact( Low(Frequency( High(Impact( PAIN(
- 45. What Causes Unexpected Behavior (likeliness)? What Makes Troubleshooting Time-Consuming (impact)? Semantic Mistakes Stale Memory Mistakes Association Mistakes Bad Input Assumption Tedious Change Mistakes Copy-Edit Mistakes Transposition Mistakes Failed Refactor Mistakes False Alarm Non-Deterministic Behavior Ambiguous Clues Lots of Code Changes Noisy Output Cryptic Output Long Execution Time Environment Cleanup Test Data Creation Using Debugger Most of the pain was caused by human factors. What causes PAIN?
- 46. What Causes Unexpected Behavior (likeliness)? What Makes Troubleshooting Time-Consuming (impact)? Non-Deterministic Behavior Ambiguous Clues Lots of Code Changes Noisy Output Cryptic Output Long Execution Time Environment Cleanup Test Data Creation Using Debugger What causes PAIN? Most of the pain was caused by human factors. Semantic Mistakes Stale Memory Mistakes Association Mistakes Bad Input Assumption Tedious Change Mistakes Copy-Edit Mistakes Transposition Mistakes Failed Refactor Mistakes False Alarm
- 47. What Causes Unexpected Behavior (likeliness)? What Makes Troubleshooting Time-Consuming (impact)? Non-Deterministic Behavior Ambiguous Clues Lots of Code Changes Noisy Output Cryptic Output Long Execution Time Environment Cleanup Test Data Creation Using Debugger What causes PAIN? Semantic Mistakes Stale Memory Mistakes Association Mistakes Bad Input Assumption Tedious Change Mistakes Copy-Edit Mistakes Transposition Mistakes Failed Refactor Mistakes False Alarm Most of the pain was caused by human factors.
- 48. What Causes Unexpected Behavior (likeliness)? What Makes Troubleshooting Time-Consuming (impact)? Non-Deterministic Behavior Ambiguous Clues Lots of Code Changes Noisy Output Cryptic Output Long Execution Time Environment Cleanup Test Data Creation Using Debugger What causes PAIN? PAIN is a consequence of how we interact with the code. Semantic Mistakes Stale Memory Mistakes Association Mistakes Bad Input Assumption Tedious Change Mistakes Copy-Edit Mistakes Transposition Mistakes Failed Refactor Mistakes False Alarm
- 49. PAIN occurs during the process of understanding and extending the software Complex( So*ware( PAIN Not the Code. Optimize “Idea Flow”
- 50. Analyzing the Types of Mistakes #1 Cause of Mistakes: Misunderstandings of how the system worked
- 51. ApplicationTests The Bugs were in Our Conceptual Models Fix: We built a Data Dictionary
- 52. The “Coat of Armor” Principle Software Coat of Armor
- 53. “Alarm Pain” Six hours of troubleshooting and repairing tests when nothing is broken.
- 54. The Waxy Coating Principle Optimize the signal to noise ratio. Software (before) Software (after) Tests are like a waxy coating poured over the code.
- 55. We deleted everything… The automation wasn’t solving the problem!
- 56. What are the specific risks in this release? How can we mitigate the risks? Handful of manual tests Custom Tools +
- 57. Release Size (weeks) Time Defect Rates We Could See the Effects… Effective Risk Management Looks Like This
- 58. My team spent tons of time working on improvements that didn’t make much difference. We had tons of automation, but the automation didn’t catch our bugs.
- 59. My team spent tons of time working on improvements that didn’t make much difference. We had well-modularized code, but it was still extremely time-consuming to troubleshoot defects.
- 60. The hard part isn’t solving the problems it’s identifying the right problems to solve. “What are the specific problems that are causing the team’s pain?”
- 61. “Better” Fifth Discipline: “Shared Vision” = “Shared Better” The Key to Team Learning: “Better”
- 62. “Better” following best practices (solution-focused) (problem-focused) “Better” solving the problems What does “better” really mean? How do we identify the right problems to solve?
- 63. Cost & Risk are a Function of Increased Difficulty Cost & Risk Difficulty of Work The Difficulty of Doing Our Jobs Human Limitations
- 64. Likelihood)of)) Unexpected) Behavior) Cost)to)Troubleshoot)and)Repair) High)Frequency) Low)Impact) Low)Frequency) Low)Impact) Low)Frequency) High)Impact) PAIN) “Better” = Reduce the Risk Reduce Likeliness = Mistake Proofing Reduce Recovery Cost = Mistake Tolerance
- 65. What Exactly does Idea Flow Measure?
- 66. Complex( So*ware( PAIN Not the Code. Optimize “Idea Flow”
- 67. FRICTION is a Function of Increased Difficulty Cost & Risk Difficulty of Work The Difficulty of Doing Our Jobs Human Limitations
- 68. The Rhythm of “Idea Flow” Write a little code. Work out the kinks. Write a little code. Work out the kinks. Write a little code. Work out the kinks.
- 69. ConflictConfirm Rework' Learn' Validate( Modify' Progress Loop Conflict Loop Troubleshoot' Write a little code. Work out the kinks. The Rhythm of “Idea Flow”
- 71. Confirmed Expectations Our brain works like a giant prediction engine…
- 72. Conflicting Expectations Our brain works like a giant prediction engine…
- 73. ConflictConfirm Rework' Learn' Validate( Modify' Progress Loop Conflict Loop Troubleshoot' The Rhythm of “Idea Flow”
- 74. “Friction” in Idea Flow ConflictConfirm Rework' Learn' Validate( Modify' Progress Loop Conflict Loop Troubleshoot'
- 75. “Idea Flow Map” “Friction” A “conflict” starts with an unexpected observation and ends when understanding is resolved.
- 76. Idea Flow Mapping Tools (Open Source, Supported GA ~June 2016) github.com/ideaflow/tools
- 77. Typical Idea Flow Maps Single Problem Multi-Problem
- 78. Reading Visual Indicators in Idea Flow Maps Le#$Atrium$ Le#$Ventricle$ Right$Ventricle$ Right$Atrium$ What’s$causing$this$pa7ern?$ Similar to how an EKG helps doctors diagnose heart problems...
- 79. ...Idea Flow Maps help developers diagnose software problems. Problem-Solving Machine Reading Visual Indicators in Idea Flow Maps
- 80. = Solution Strategy The “Heart” of Software Development (the problem-solving machine)
- 81. = Solution Strategy The “Heart” of Software Development (the problem-solving machine) Choose a general strategy
- 82. = Solution Strategy The “Heart” of Software Development (the problem-solving machine) Understand the system
- 83. = Solution Strategy The “Heart” of Software Development (the problem-solving machine) Code & work out the kinks
- 84. = Solution Strategy The “Heart” of Software Development (the problem-solving machine) Back to the drawing board
- 85. The symptoms we experience depend on where the disruptions are in the problem-solving process
- 86. Trial and Error The Friction is Here Troubleshooting Progress Learning Rework
- 88. Struggling with Understanding the Code The Friction is Here Troubleshooting Progress Learning Rework
- 90. Struggling with Feedback The Friction is Here Troubleshooting Progress Learning Rework
- 92. measures the time spent on: Idea Flow x Troubleshooting x Learning x Rework Quality Risk Familiarity Risk Assumption Risk
- 93. What percentage of time do you think your team spends on friction?
- 94. Troubleshooting Progress Learning 7:070:00 0:00 19:52 12 year old project after all original developers left. Case Study: Huge Mess with Great Team 70-90% of dev capacity on “friction”
- 95. The Team’s Improvement Focus: Increasing unit test coverage by 5% Case Study: Huge Mess with Great Team 1. Test Data Generation 2. Merging Problems 3. Repairing Tests 1000 hours/month The Biggest Problem: ~700 hours/month generating test data
- 96. The Retrospective “What’s the biggest opportunity for improvement?” Our biggest problem
- 97. “What’s the biggest opportunity for improvement?” “The awful email template engine code!” Our biggest problem The Retrospective
- 98. “What’s the biggest opportunity for improvement?” “Fill in missing unit tests!” Our biggest problem The Retrospective
- 99. “What’s the biggest opportunity for improvement?” “I know how to improve database performance!” Our biggest problem The Retrospective
- 100. “What’s the biggest opportunity for improvement?” “Let’s improve maintainability of our test framework!” Our biggest problem The Retrospective
- 101. “What’s the biggest opportunity for improvement?” Just because a problem comes to mind, doesn’t mean it’s an important problem to solve. Our biggest problem The Retrospective
- 102. “What’s the biggest opportunity for improvement?” Our biggest problem What do I feel the most intensely about? Daniel Kahneman Thinking Fast and Slow The Retrospective
- 103. “What’s the biggest opportunity for improvement?” “The awful email template engine code!” Recency Bias Our biggest problem The Retrospective
- 104. “What’s the biggest opportunity for improvement?” Guilt Bias “Fill in missing unit tests!” Our biggest problem The Retrospective
- 105. “What’s the biggest opportunity for improvement?” “I know how to improve database performance!” Known Solution Bias Our biggest problem The Retrospective
- 106. “What’s the biggest opportunity for improvement?” Sunk Cost Bias “Let’s improve maintainability of our test framework!” Our biggest problem The Retrospective
- 107. Relying on Gut Feel:
- 108. 18 months after a Micro-Services/Continuous Delivery rewrite. Troubleshooting Progress Learning 40-60% of dev capacity on “friction” 0:00 28:15 12:230:00 Case Study: From Monolith to Microservices
- 109. The Architecture Looked Good on Paper Team A Team B Team C Complexity Moved Here
- 110. We Don’t Have TIME To Fix It!
- 111. PAIN The Classic Story of Project Failure Problems get deferred Builds start breaking Releases get chaotic Productivity slows to a crawl Developers begging for time It’s never enough Project Meltdown
- 112. The Cost of Escalating Risk 0% 100% Release 1 Release 2 Release 3 Troubleshooting Progress Learning Percentage Capacity spent on Troubleshooting (red) and Learning (blue) (extrapolated from samples)
- 113. 0% 100% Release 1 Release 2 Release 3 Percentage Capacity spent on Troubleshooting (red) and Learning (blue) Figure out what to do Learning is front-loaded Troubleshooting Progress Learning The Cost of Escalating Risk
- 114. 0% 100% Release 1 Release 2 Release 3 Percentage Capacity spent on Troubleshooting (red) and Learning (blue) Rush Before the Deadline Validation is Deferred Troubleshooting Progress Learning The Cost of Escalating Risk
- 115. 0% 100% Release 1 Release 2 Release 3 Percentage Capacity spent on Troubleshooting (red) and Learning (blue) Pain Builds Baseline friction keeps rising Troubleshooting Progress Learning The Cost of Escalating Risk
- 116. 0% 100% Release 1 Release 2 Release 3 Percentage Capacity spent on Troubleshooting (red) and Learning (blue) Chaos Reigns Unpredictable work stops fitting in the timebox Troubleshooting Progress Learning The Cost of Escalating Risk
- 117. This is the Biggest Cause of FAILURE In Our Industry: Have you struggled with these problems in your organization?
- 118. “Better” “Better” Fifth Discipline: “Shared Vision” = “Shared Better” Developer Developer Manager Manager The Key to Organizational Learning:
- 119. Lessons Learned
- 120. How can we reduce the impact of bad architecture assumptions? (Assumption Risk) The cost of bad architecture decisions in the microservices world is EXTREMELY HIGH.
- 121. If you’ve got beautiful code Because you pushed all the PAIN to your clients… Your code SUCKS. How do we know what the client’s experience will be like?
- 122. Lots of Friction in BRAND NEW code: 27:15 “WHY?”
- 123. Application 3rd Party Dependencies Local Components Usability problems cascade through dependencies Usability Problems This is Painful!
- 124. Application 3rd Party Dependencies Local Components Usability Problems This is Painful! 90% of our Software is HERE
- 125. “The Idea Flow Factory” (supply chain model) Optimize the Rate of Idea Flow Across ANY Software Supply Chain
- 126. What if we learned our way to “better”, together? Company Company Company Company Company Company Company
- 127. LEARN YOUR WAY TO AWESOME.
- 128. Start with Why Simon Sinek
- 129. Why - Your Purpose How - Your Process What - Your Solution Humans Align Around Shared Purpose
- 130. Let’s Make the PAIN Visible! #OpenDX
- 131. Why? #OpenDX
- 132. Why? Because This Really SUCKS.
- 133. Why? And So Does This.
- 134. #OpenDX Measure the PAIN that you CARE About
- 135. LEARN YOUR WAY TO AWESOME. We BELIEVE That: Even though we may not have all the answers, what matters is that we know how to learn.
- 136. How Do We Learn Our Way There? Point A Point AWESOME? Iterate!
- 137. Because we’re part of a community with other smart people… We BELIEVE That: We will get better, faster, by working together.
- 138. #OpenDX How?
- 139. Community Analytics Platform (Long-Term Roadmap)
- 140. Community Problem #1: Solve This Pain:
- 141. Team Visibility Platform IFM Data Collection Team Visibility Raw Data Timeline Abstraction 1. Record an Idea Flow Map 2. Review/Annotate an Idea Flow Map 3. Identify the Biggest Pains Project Abstraction #Private Taxonomy
- 142. Community Analytics Platform Anonymizer Anonymizer Anonymizer Anonymizer #Private Taxonomy Team Visibility #Private Taxonomy Team Visibility #Private Taxonomy Team Visibility #Private Taxonomy Team Visibility Community Analytics #Shared Taxonomy
- 143. Community Analytics Platform Community Analytics Shared Taxonomy Tiger Bear Requirement: Analyze Patterns & Codify Lessons Learned Community Focus Community Analytics #Shared Taxonomy
- 144. Community Problem #2: Solve This Pain:
- 145. Team Server Team Server “Idea Flow Factory” Supply Chain Cost Model Supply Chain Optimization Team Server “Customer Factory” Revenue Model (Ash Maurya) Throughput Accounting (Effort vs Revenue) Organizational Mastery Platform
- 146. Community Problem #3: Solve This Pain with Lack of Skills: Company Nope.Nope.Nope.Nope.Nope. Nope.Nope.Nope.Nope. Nope.Nope. Nope.Nope.Nope. Nope.Nope.Nope.Nope.Nope.Nope.Nope.Nope. Nope. Yay! We need a Sr Developer!
- 147. Open Certification Platform (Crowd-Sourced Peer Mentorship) Decision Assessment Archetype Examples & Decision-Making Tests Mentorship Training M1 M2 M3 C1 C2 C3 Crowd-Sourced Peer Video Assessment #Shared Taxonomy
- 148. Community Problem #4: Solve This Pain with Upstream Usability:
- 149. #OpenDX Bring visibility up to the highest levels of our organizations and across the industry
- 150. The “Open Source” Roadmap Point A Point AWESOME? Iterate!
- 151. Get together over lunch and share lessons learned. Leaders Circle Developers Circle
- 152. Open Mastery Circle Meetings Circle Leader Circle Member Focus: What’s the problem to solve? What: Ask questions about the facts Why: Breakdown the causes How: Strategies to reduce the pain Codify: Lessons Learned into #Patterns. Observation Questions The Explicit Mastery Loop
- 153. The Explicit Mastery Loop Input: Constraints Target: Optimize the Rate of Idea Flow short-term looplong-term loop 1. What2. Why3. How F ocus! Output: Pain Signal Improve Quality of Decisions
- 154. Input: Constraints Target: Optimize the Rate of Idea Flow short-term looplong-term loop 1. What2. Why3. How F ocus! Output: Pain Signal Target - The direction of “better” Target: Optimize the Rate of Idea Flow The Explicit Mastery Loop
- 155. Input: Constraints Target: Optimize the Rate of Idea Flow short-term looplong-term loop 1. What2. Why3. How F ocus! Output: Pain Signal Input - The constraints that limit our short-term choices… The Explicit Mastery Loop
- 156. Input: Constraints Target: Optimize the Rate of Idea Flow short-term looplong-term loop 1. What2. Why3. How F ocus! Output: Pain Signal Output - The pain signal we’re trying to improve The Explicit Mastery Loop
- 157. Input: Constraints Target: Optimize the Rate of Idea Flow short-term looplong-term loop 2. What3. Why4. How F ocus! Output: Pain Signal 1. ! 1. Focus on the biggest pain F ocus! The Explicit Mastery Loop
- 158. Input: Constraints Target: Optimize the Rate of Idea Flow short-term looplong-term loop 2. What3. Why4. How F ocus! Output: Pain Signal 1. ! 2. What - Identify the specific symptoms 2. What The Explicit Mastery Loop
- 159. Input: Constraints Target: Optimize the Rate of Idea Flow short-term looplong-term loop 2. What3. Why4. How F ocus! Output: Pain Signal 1. ! 3. Why - Breakdown cause and effect 3. Why The Explicit Mastery Loop
- 160. Input: Constraints Target: Optimize the Rate of Idea Flow short-term looplong-term loop 2. What3. Why4. How F ocus! Output: Pain Signal 1. ! 4. How 4. How - Run Experiments to learn what works The Explicit Mastery Loop
- 161. Input: Constraints Target: Optimize the Rate of Idea Flow short-term looplong-term loop 2. What3. Why4. How F ocus! Output: Pain Signal 1. ! 5. Codify Lessons Learned and Modify Behavior The Explicit Mastery Loop
- 162. #OpenDX What?
- 163. Companies Community Groups HQ in Austin Open Mastery Austin WE ARE a “Pay it Forward” Industry Peer Mentorship Network
- 164. LEARN YOUR WAY TO AWESOME.
- 165. Five Years Ago I Read This Book: How to Build a Learning Organization
- 166. —> 2.0 Visibility Changes Everything How to Measure the PAIN in Software Development Janelle Klein Visibility + Lean Manufacturing
- 167. —>+ Everyday Science (Lean Startup) We get a design template for how to build a learning organization. Visibility Changes Everything 2.0
- 168. Focus: Quality of Execution Decisions Focus Traction $$$ $$ Focus: Quality of Investment Decisions Risk Translator (interface) Management (Coordination) Engineering (Execution) Open Mastery Learning Framework Idea Flow Pain Sensor
- 169. is a Learning Organization
- 170. Visibility Starts a Chain Reaction #OpenDX