Why we don’t use the Term DevOps: the Journey to a Product Mindset - Destination AARhus
•
3 likes•538 views
While the adoption of DevOps makes teams move faster with reduced dependency on central operations, it can constrain teams who lack the skills to self-manage the full application and infrastructure stack. The way to overcome this challenge is creating an internal platform and treating it as a world-class product offering. “Applying product management to internal platforms means establishing empathy with internal consumers (read: developers) and collaborating with them on the design. Platform product managers establish roadmaps and ensure the platform delivers value to the business and enhances the developer experience”, via ThoughtWorks Technology Radar. In this talk, Henning Jacobs will walk you through how Zalando adopted a customer-first mindset with regards to its developer tooling. He will show the effect on developer satisfaction when internal platforms are given the same respect as external product offerings. Henning will furthermore tell his story about how Zalando moved from a classical infrastructure team to a product mindset with strong focus on building a world-class developer experience. Henning shares both their learnings and challenges going through this transition, and the impact it has on the daily life of Zalando’s customers (developers). This talk was given in Aarhus on 4th of June 2019.
Report
Share
Why we don’t use the Term DevOps: the Journey to a Product Mindset - Destination AARhus
- 1. Why we don't use the term DevOps DESTINATION AARHUS 2019-06-04 HENNING JACOBS @try_except_
- 2. 2 EUROPE’S LEADING ONLINE FASHION PLATFORM
- 3. 3 ZALANDO AT A GLANCE ~ 5.4billion EUR revenue 2018 > 250 million visits per month > 15.000 employees in Europe > 79% of visits via mobile devices > 26 million active customers > 300.000 product choices ~ 2.000 brands 17 countries
- 4. 4 THE BIRTH OF DEVOPS - 2009 “10+ Deploys per Day: Dev and Ops Cooperation at Flickr.” DevOps Days Belgium https://devops.com/the-origins-of-devops-whats-in-a-name/
- 8. 8 THE PHOENIX PROJECT - 2013 "The Three Ways" 1. Systems Thinking 2. Amplify Feedback Loops 3. Culture of Continual Experimentation And Learning
- 10. 10 CARGO CULTING
- 11. 11 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
- 13. 13
- 16. 16 DEV + OPS ⇒ DEVOPS Devs "DevOps" Team Ops
- 17. 17 DEV + OPS ⇒ ENGINEERING Devs "DevOps" Team Ops Engineering Team
- 18. 18 PRODUCT DEVELOPMENT TEAMS "Great products emerge from empowered, fully cross-functional teams." - John Cutler, 2019 https://amplitude.com/blog/on-being-product-led
- 19. 19 DEVOPS IS It is a culture, in which people work together to improve the product delivery cycle.
- 20. 20 DEVOPS ANTI-PATTERNS Dev and Ops Silos DevOps Team Silo Dev Don't Need Ops DevOps as Tools Team Rebranded SysAdmin Ops Embedded in Dev Team web.devopstopologies.com
- 21. 21
- 22. 22 TEAM INTERACTION MODES • Collaboration: 2 teams working together • X-as-a-Service: 1 provides, 1 consumes • Facilitating: 1 team helps another Product teams need a family too - DevOpsDays Portugal
- 23. 23 A BRIEF HISTORY OF ZALANDO TECH
- 25. 25 DEVOPS IS ABOUT SKIN IN THE GAME Absence of skin in the game: People who are isolated from the impacts of their decisions do not learn. They remain captive to their erroneous ideas about how the world works.
- 27. 27 2015: RADICAL AGILITY AWS STUPS DOCKER DEPLOY SSH ACCESS AUDIT REPORTS FULL AWS ACCESS Teams have admin access & full responsibility
- 28. 28 2015: ISOLATED AWS ACCOUNTS Internet *.abc.example.org *.xyz.example.org Team ABC Team XYZ EC2EC2 ELBELB EC2
- 29. 29 Platform > 1100 developers > 200 development teams 2019
- 30. 30 YOU BUILD IT, YOU RUN IT The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it. This brings developers into contact with the day-to-day operation of their software. It also brings them into day-to-day contact with the customer. - A Conversation with Werner Vogels, ACM Queue, 2006
- 31. 31 ON-CALL: YOU OWN IT, YOU RUN IT When things are broken, we want people with the best context trying to fix things. - Blake Scrivener, Netflix SRE Manager
- 32. 32 "SKIN IN THE GAME"
- 33. 33 DEVELOPER JOURNEY Consistent story that models all aspects of SW dev
- 35. 35 Developer Journey Correctness Compliance GDPR Security Cost Efficiency 24x7 On Call Governance Resilience Capacity ...
- 36. 36 DEVELOPER PRODUCTIVITY Code Build Test Deploy OperateSetup Cloud Native Application Runtime
- 37. 37 CLOUD NATIVE .. uses an open source software stack to deploy applications as microservices, packaging each part into its own container, and dynamically orchestrating those containers to optimize resource utilization. Cloud native technologies enable software developers to build great products faster. - https://www.cncf.io/
- 38. 38 CONTAINERS END-TO-END Code Build Test Deploy OperateSetup Cloud Native Application Runtime
- 39. 39 CONTAINERS
- 40. 40 CONTAINERS
- 41. 41 PLATFORM AS A PRODUCT "A good platform is treated as a product (reliable, usable, fit for purpose)." - Manuel Pais, 2019 Product teams need a family too - DevOpsDays Portugal
- 43. 43 PLAN & SETUP
- 44. 44 Plan Stories Rules of Play Tech Radar
- 49. 49 BUILD & TEST
- 52. 52 DEPLOY
- 54. 54 DEPLOYMENT CONFIGURATION ├── deploy/apply │ ├── deployment.yaml │ ├── credentials.yaml # Zalando IAM │ ├── ingress.yaml │ └── service.yaml └── delivery.yaml # Zalando CI/CD
- 55. 55 INGRESS.YAML kind: Ingress metadata: name: "..." spec: rules: # DNS name your application should be exposed on - host: "myapp.foo.example.org" http: paths: - backend: serviceName: "myapp" servicePort: 80
- 56. 56 TEMPLATING: MUSTACHE kind: Ingress metadata: name: "..." spec: rules: # DNS name your application should be exposed on - host: "{{{APPLICATION}}}.example.org" http: paths: - backend: serviceName: "{{{APPLICATION}}}" servicePort: 80
- 58. 58 CDP: DEPLOY "glorified kubectl apply"
- 61. 61 STACKSET CRD kind: StackSet ... spec: ingress: hosts: ["foo.example.org"] backendPort: 8080 stackLifecycle: scaledownTTLSeconds: 1800 limit: 5 stackTemplate: spec: podTemplate: ... github.com/zalando-incubator/stackset-controller
- 62. 62 TRAFFIC SWITCHING STEPS IN CDP github.com/zalando-incubator/stackset-controller
- 63. 63 EMERGENCY ACCESS SERVICE Emergency access by referencing Incident zkubectl cluster-access request --emergency -i INC REASON Privileged production access via 4-eyes zkubectl cluster-access request REASON zkubectl cluster-access approve USERNAME
- 64. 64 INTEGRATIONS
- 65. 65 CLOUD FORMATION VIA CI/CD ├── deploy/apply │ ├── deployment.yaml # Kubernetes │ ├── cf-iam-role.yaml # AWS IAM Role │ ├── cf-rds.yaml # AWS RDS Database │ ├── kube-ingress.yaml │ ├── kube-secret.yaml │ └── kube-service.yaml └── delivery.yaml # CI/CD config "Infrastructure as Code"
- 66. 66 ZALANDO IAM/OAUTH VIA CRD kind: PlatformCredentialsSet .. spec: application: my-app tokens: read-only: privileges: - com.zalando::foobar.read clients: employee: grant: authorization-code realm: users redirectUri: https://example.org/auth/callback Extensible Kubernetes API
- 67. 67 POSTGRES OPERATOR Application to manage PostgreSQL clusters on Kubernetes >700 clusters running on Kubernetes github.com/zalando/postgres-operator
- 68. Elasticsearch in Kubernetes Elasticsearch 2.500 vCPUs 1 TB RAM github.com/zalando-incubator/es-operator/
- 69. 69 SUMMARY • Application Bootstrapping • Git as source of truth and UI • 4-eyes principle for master/production • Extensible Kubernetes API as primary interface • OAuth/IAM credentials • PostgreSQL • CloudFormation for proprietary AWS services
- 70. 70 DELIVERY PERFORMANCE METRICS • Lead Time • Release Frequency • Time to Restore Service • Change Fail Rate https://srcco.de/posts/accelerate-software-delivery-performance.html
- 71. 71 CONTAINERS From "Accelerate: The Science of Lean Software and DevOps"
- 72. 72 DELIVERY PERFORMANCE METRICS • Lead Time • Release Frequency • Time to Restore Service • Change Fail Rate ≙ Commit to Prod ≙ Deploys/week/dev ≙ MTRS from incidents ≙ n/a
- 73. “.. means establishing empathy with internal consumers (read: developers) and collaborating with them on the design. Platform product managers establish roadmaps and ensure the platform delivers value to the business and enhances the developer experience.” - ThoughtWorks Technology Radar
- 74. 74 PRODUCT MINDSET ... means we are developing all our products for the best customer experience and biggest business impact while all actions are aligned with our strategy.
- 77. 77 DOCUMENTATION "Documentation is hard to find" "Documentation is not comprehensive enough" "Remove unnecessary complexity and obstacles." "Get the documentation up to date and prepare use cases" "More and more clear documentation" "More detailed docs, example repos with more complicated deployments."
- 78. 78 DOCUMENTATION • Restructure following www.divio.com/en/blog/documentation/ • Concepts • How Tos • Tutorials • Reference • Global Search • Weekly Health Check: Support → Documentation
- 80. 80 NEWSLETTER "You can now.." • You can now benefit from the most recent Kubernetes 1.12 features, e.g. .. • You can now analyse your Kotlin project with SonarQube and upload your Scala code coverage report to SonarQube
- 82. 82 TESTIMONIALS “Useful information, good level of details and pleasant to read. It's one of the few newsletters that I took time to read entirely :)” - a reader, July 2018
- 83. 83 TESTIMONIALS “So, thank you, Team Automata, for listening to our community, taking our upvotes in consideration when developing new solutions and building every day 'the first CI that doesn't suck'.” - a user, October 2018
- 84. 84 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
- 85. 85 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
- 86. 86 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
- 87. 87 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
- 88. 88 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
- 89. 89 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
- 92. 92 You build it, you run it is normal
- 93. 93 WAIT A SECOND.. What about my job title?
- 94. 94 "DEVOPS ENGINEER" AS A JOB TITLE
- 95. 95 T-SHAPED ENGINEER "Generalizing specialist" Breadth of Knowledge TechnicalDepth
- 96. 96 DevOps Engineer Software Engineer / Software Developer (T-Shaped)
- 97. 97
- 98. 98 OPEN SOURCE & MORE Kubernetes on AWS github.com/zalando-incubator/kubernetes-on-aws Skipper HTTP Router & Ingress controller github.com/zalando/skipper External DNS github.com/kubernetes-incubator/external-dns Postgres Operator github.com/zalando-incubator/postgres-operator More Zalando Tech Talks github.com/zalando/public-presentations
- 99. QUESTIONS? HENNING JACOBS HEAD OF DEVELOPER PRODUCTIVITY henning@zalando.de @try_except_ Illustrations by @01k