Devops at Netflix (re:Invent)

Editor's Notes

1. My friends Joe and Carl already told you about Nac and our build system. This allows the devs to take control of their deployment. Each team is responsible for their own deployments and uptime. When something breaks, we have a system that lets us page a team who then gets on and fixes their stuff. Each team is responsible for their own destiny. So how do we stay reliable when we have no control? Information.
2. Automate as much as you can
3. The more automated things are, the easier it is to be a sysadmin. Application startup Configuration Code deployment Full system deployment The more automated things are, the easier it is to scale especially in a virtualized environment with auto-scaling And virtualized computing added the last bit, the ability to automate system deployment. (Ok, that ’ s not entirely true, but watch me wave my hands and say it is)
4. In most places, you have this. Standard image with tools to manage the systems and the deployment.
5. By building for three, you can reasonably lose one of your instances and still be stable.
6. replication factor quorum reads / writes
7. In most systems, you worry about the software and installing it on an OS. At Netflix, the smallest thing we worry about is the instance image, which lives in a cluster. We ’ ve essentially built a platform for doing automated deployment of Java code (and some Python too!)
8. So, why do we bake custom images instead of just using Puppet or Chef to deploy packages dynamically to launched generic machines? We like to front-load the full machine assembly to build time, instead of waiting until deployment time. We do this because: • More Reliable: less systems that can fail at deploy time right when we need them most. • Faster Launch: means quicker reaction to load increases, e.g. autoscaling up can be more precise. • Single image: produces exactly homogeneous clusters. No file/package version skew across machines in a cluster
9. OK, back to the build pipeline again. We have a vague “ app bundles ” output in this diagram. Let ’ s delve into how we manage the application bundle artifacts in more detail.
10. The first step of the baking process is to create the “ base ” image that we will use for baking all app images. This is done once every week or two. We start with a standard Linux distro as a foundation (CentOS now, Ubuntu on the way), and add in our favorite, our custom and customized packages: • Apache, Java (JDK 6 and 7), Tomcat, Perl, Python, provisioning and startup scripts, log management tools, monitoring agents, etc. The end result is a beefed-up OS image that is ready to go, and just needs an app added.
11. The first step of the baking process is to create the “ base ” image that we will use for baking all app images. This is done once every week or two. We start with a standard Linux distro as a foundation (CentOS now, Ubuntu on the way), and add in our favorite, our custom and customized packages: • Apache, Java (JDK 6 and 7), Tomcat, Perl, Python, provisioning and startup scripts, log management tools, monitoring agents, etc. The end result is a beefed-up OS image that is ready to go, and just needs an app added.
12. Gateway classifies and routes events based on severity and the systems involved. The gateway currently processes around 48K events a day
13. (step through) here are some best practices we ’ ve learned over the last year with EC2. Some of these we follow well, and some we need to follow better.
14. At Netflix we use autoscaling the help manage reliability and cost. Here is one of our clusters scaling up and down. We are tuning for the holidays, so you can see parts where we are doing squeeze tests and adjusting the scaling speed and values.
15. Amazon will help you as well. One way they do this is by providing zones. Each zone is like an island that is loosely connected to the other zones, but mostly distinct.
16. So how do you get better than 99.95% uptime? Multiple zones! By spreading your systems out across multiple zones, you should be able to withstand the failure of one zone. In a little bit, I ’ ll go over how reddit and Netflix used a multizone strategy to survive outages.
17. Amazon, as well as other providers, offer multiple regions as well. Regions are essentially like separate providers with the same featureset. Your data does not get shared across regions
18. You can contact me in one of these ways, or ask your question now. thank you.