Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

Jörg Schad - Hybrid Cloud (Kubernetes, Spark, HDFS, …)-as-a-Service - Codemotion Berlin 2018

•
•
Codemotion
Codemotion

The overall evolution towards microservices has caused a lot of IT leaders to radically rethink architectures and platforms. One can hardly keep up with the rapid onslaught on new distributed technologies. The same people who just asked yesterday "how can we deploy Docker containers?", are now asking "how can we operate Kubernetes-as-a-Service on-premise?", and are about to start asking "how can we operate the open source frameworks of our choice, such as Spark, TensorFlow, HDFS, and more, as a service across hybrid clouds?”. This session will discuss: Challenges of orchestrating and operating.

1 of 40
Download to read offline
Hybrid Cloud (Kubernetes, Spark, HDFS)-as-a-Service @joerg_schad Berlin | November 20 - 21, 2018
Jörg Schad Technical Lead Data Science ● Core Mesos developer at Mesosphere ● Twitter: @joerg_schad
Hybrid Cloud (Kubernetes, Spark, HDFS, …)-as-a-Service @joerg_schad @mesosphere
Yesterday: Individual Containers 4
Present: On-Prem Container Orchestration 5
So you’ve decided you want to run Kubernetes ... 6
ETCD 01 MASTER 01 etcd LoadBalancer WORKER 01 API Server Scheduler Controller Manager Kubectl Kubelet Kube Proxy Container Engine (Docker, CRI-O) Pod Pod Pod HTTPS HTTPS HTTPS HTTPS
Deploying Kubernetes 1. Prerequisites 2. Installing the Client Tools 3. Provisioning Compute Resources 4. Provisioning the CA and Generating TLS Certificates 5. Generating Kubernetes Configuration Files for Authentication 6. Generating the Data Encryption Config and Key 7. Bootstrapping the etcd Cluster… 3x for HA 10. Bootstrapping the Kubernetes Control Plane… 3x for HA 13. Bootstrapping the Kubernetes Worker Nodes 14. Configuring kubectl for Remote Access 15. Provisioning Pod Network Routes 16. Deploying the DNS Cluster Add-on… Deploying other Add-ons 20. Smoke Test 21. Cleaning Up Running on your own
© Gerard Julien/AFP 3 AM
Kubernetes Lifecycle Management 10 Deploy Scale Upgrade Recover ... 1.9.3 1.9.4 1.9.6 1.9.7 1.10.3 1.10.4 ...
Manual - Exceptionally difficult - DIY PRs dormant for years - Multiple steps - Zero day-two DIY Configuration Management - Easy “hello world” - No commercial support - DIY security - DIY PRs dormant for years - Lacks auxiliary components Vendor Configuration Management - Easy provisioning of clusters - Hard coded infrastructure - Can include auxiliary components Cloud Provider - Easy provisioning multi-clusters - Standard centralized management - Includes auxiliary components KOPS Kubespray Kubeadm Public Anywhere (Bare Metal, Cloud, Virtualization) Passive Intelligent Google Container Engine (GKE) How to manage Kubernetes
Future: Hybrid Cloud Service Orchestration 12 Kafka Cassandra Spark HDFSCassandra On-Prem AWS
Why Services? 13 Microservice 1. Microservice records running activity
Why Services? 14 Microservice Display 1. Microservice records running activity 2. Data written to Kafka topic 3. Data analyzed Flink 4. Results stored in Cassandra 7. Reminder displayed using Microservice
Why Services? Source Code Control Build and Test Release Deploy, Monitor and Log Automatically trigger CI/CD pipeline based on code check-in. Start automated build and test, including functional, security and performance tests. Update artifact repository with latest successful code artifacts and pull newest images Deploy applications to container orchestration and watch with monitoring and logging CONTINUOUS INTEGRATION (CI) CONTINUOUS DEPLOYMENT (CD) Marathon
Why Services? Continuous Integration Monitoring & Operations Distributed Data Storage and Streaming Data Preparation and Analysis Storage of trained Models and Metadata Use trained Model for Inference Distributed Training using Machine Learning Frameworks Data & Streaming Model Engineering Model Management Model Serving Model Training Management
© Gerard Julien/AFP Service Orchestration Cassandra
18 Distributed Systems Are hard...
Container Orchestration Service Orchestration Container: Deploy/Remove E.g., ReplicaSet Controller 19 Replicas = 4 DESIRED Replicas = 0 CURRENT #Namenodes = 2 DESIRED #Namenodes = 0 CURRENT HDFS Namenode 1. Recover State from Journal 2. Launch Namenode 1 3. Launch Namenode 2
State Machines
Challenge 1: Service Lifecycle Management 21 Deploy Scale Configure Recover ... Typical Datacenter siloed, over-provisioned servers, low utilization Kubernetes Jenkins Kafka Kubernetes TensorFlow
Deploying Kubernetes 1. Prerequisites 2. Installing the Client Tools 3. Provisioning Compute Resources 4. Provisioning the CA and Generating TLS Certificates 5. Generating Kubernetes Configuration Files for Authentication 6. Generating the Data Encryption Config and Key 7. Bootstrapping the etcd Cluster… 3x for HA 10. Bootstrapping the Kubernetes Control Plane… 3x for HA 13. Bootstrapping the Kubernetes Worker Nodes 14. Configuring kubectl for Remote Access 15. Provisioning Pod Network Routes 16. Deploying the DNS Cluster Add-on… Deploying other Add-ons 20. Smoke Test 21. Cleaning Up Running on your own $ dcos package install kubernetes on
Encrypted Communication 1. Generate certificates and keys using DC/OS PKI 2. Store certificates in DC/OS Secret Store 3. Distribute certificates and keys to data services 4. Apply security configuration 5. Perform rolling restart of the service Each data service has its own distinct security procedure, which sometimes change per version Kafka Cassandra Elastic Spark HDFS
Complete Lifecycle $ dcos hdfs plan list [ "deploy", "recovery", "replace", "update" ] dcos hdfs plan show deploy deploy (serial strategy) (IN_PROGRESS) ├─ journal (parallel strategy) (IN_PROGRESS) │ ├─ journal-0:[node] (COMPLETE) │ ├─ journal-1:[node] (COMPLETE) │ └─ journal-2:[node] (PREPARED) ├─ name (serial strategy) (PENDING) │ ├─ name-0:[format] (PENDING) │ ├─ name-0:[node] (PENDING) │ ├─ name-1:[bootstrap] (PENDING) │ └─ name-1:[node] (PENDING) ├─ zkfc (serial strategy) (PENDING) │ ├─ name-0:[zkfc-format] (PENDING) │ ├─ name-0:[zkfc] (PENDING) │ └─ name-1:[zkfc] (PENDING) └─ data (parallel strategy) (PENDING) ├─ data-0:[node] (PENDING) ├─ data-1:[node] (PENDING) └─ data-2:[node] (PENDING)
25
Challenge 2: Cluster Management Typical Datacenter siloed, over-provisioned servers, low utilization Kubernetes Jenkins Kafka Kubernetes TensorFlow
Challenge 2: Cluster Management Typical Datacenter siloed, over-provisioned servers, low utilization Mesos/ DC/OS automated schedulers, workload multiplexing onto the same machines Kubernetes Jenkins Kafka Kubernetes TensorFlow
© 2018 Mesosphere, Inc. All Rights Reserved. Powered by Apache Mesos
Stateful Services Kelsey Hightower Kubernetes Thought Leader Ranked #1 K8s Influencer Staff Developer Advocate PM & Chief Advocate Top Kubernetes Advocate
CRDs
CRDs https://book.kubebuilder.io/
Challenge 3: Multi-tenancy 32 Powered by Apache Mesos
Multiple Kubernetes Cluster 33 Powered by Apache Mesos Line of Business 1 Line of Business 2 Developer Cluster Shared Production Skaffold
Multiple Kubernetes Cluster 34 Powered by Apache Mesos Line of Business 1 Line of Business 2 Developer Cluster Shared Production Skaffold
Resource Quota 35 • Share resources between multiple services • No static partitioning • One role per job/entity • Use quota per role • Min and Max resource allocation
Challenge 4: Hybrid Cloud Powered by Apache Mesos
Why Hybrid 37
© 2018 Mesosphere, Inc. All Rights Reserved. 38 THANK YOU! ANY QUESTIONS? @mesosphere users@dcos.io /groups/8295652 /dcos /dcos/examples /dcos/demos chat.dcos.io
CONFIDENTIAL Make it insanely easy to build and scale world-changing technology
Questions? @joerg_schad @dcos @dcos users@dcos.io /groups/8295652 /dcos /dcos/examples /dcos/demos chat.dcos.io

More Related Content

Jörg Schad - Hybrid Cloud (Kubernetes, Spark, HDFS, …)-as-a-Service - Codemotion Berlin 2018

  • 1. Hybrid Cloud (Kubernetes, Spark, HDFS)-as-a-Service @joerg_schad Berlin | November 20 - 21, 2018
  • 2. Jörg Schad Technical Lead Data Science ● Core Mesos developer at Mesosphere ● Twitter: @joerg_schad
  • 3. Hybrid Cloud (Kubernetes, Spark, HDFS, …)-as-a-Service @joerg_schad @mesosphere
  • 5. Present: On-Prem Container Orchestration 5
  • 6. So you’ve decided you want to run Kubernetes ... 6
  • 7. ETCD 01 MASTER 01 etcd LoadBalancer WORKER 01 API Server Scheduler Controller Manager Kubectl Kubelet Kube Proxy Container Engine (Docker, CRI-O) Pod Pod Pod HTTPS HTTPS HTTPS HTTPS
  • 8. Deploying Kubernetes 1. Prerequisites 2. Installing the Client Tools 3. Provisioning Compute Resources 4. Provisioning the CA and Generating TLS Certificates 5. Generating Kubernetes Configuration Files for Authentication 6. Generating the Data Encryption Config and Key 7. Bootstrapping the etcd Cluster… 3x for HA 10. Bootstrapping the Kubernetes Control Plane… 3x for HA 13. Bootstrapping the Kubernetes Worker Nodes 14. Configuring kubectl for Remote Access 15. Provisioning Pod Network Routes 16. Deploying the DNS Cluster Add-on… Deploying other Add-ons 20. Smoke Test 21. Cleaning Up Running on your own
  • 11. Manual - Exceptionally difficult - DIY PRs dormant for years - Multiple steps - Zero day-two DIY Configuration Management - Easy “hello world” - No commercial support - DIY security - DIY PRs dormant for years - Lacks auxiliary components Vendor Configuration Management - Easy provisioning of clusters - Hard coded infrastructure - Can include auxiliary components Cloud Provider - Easy provisioning multi-clusters - Standard centralized management - Includes auxiliary components KOPS Kubespray Kubeadm Public Anywhere (Bare Metal, Cloud, Virtualization) Passive Intelligent Google Container Engine (GKE) How to manage Kubernetes
  • 12. Future: Hybrid Cloud Service Orchestration 12 Kafka Cassandra Spark HDFSCassandra On-Prem AWS
  • 14. Why Services? 14 Microservice Display 1. Microservice records running activity 2. Data written to Kafka topic 3. Data analyzed Flink 4. Results stored in Cassandra 7. Reminder displayed using Microservice
  • 15. Why Services? Source Code Control Build and Test Release Deploy, Monitor and Log Automatically trigger CI/CD pipeline based on code check-in. Start automated build and test, including functional, security and performance tests. Update artifact repository with latest successful code artifacts and pull newest images Deploy applications to container orchestration and watch with monitoring and logging CONTINUOUS INTEGRATION (CI) CONTINUOUS DEPLOYMENT (CD) Marathon
  • 16. Why Services? Continuous Integration Monitoring & Operations Distributed Data Storage and Streaming Data Preparation and Analysis Storage of trained Models and Metadata Use trained Model for Inference Distributed Training using Machine Learning Frameworks Data & Streaming Model Engineering Model Management Model Serving Model Training Management
  • 17. © Gerard Julien/AFP Service Orchestration Cassandra
  • 19. Container Orchestration Service Orchestration Container: Deploy/Remove E.g., ReplicaSet Controller 19 Replicas = 4 DESIRED Replicas = 0 CURRENT #Namenodes = 2 DESIRED #Namenodes = 0 CURRENT HDFS Namenode 1. Recover State from Journal 2. Launch Namenode 1 3. Launch Namenode 2
  • 21. Challenge 1: Service Lifecycle Management 21 Deploy Scale Configure Recover ... Typical Datacenter siloed, over-provisioned servers, low utilization Kubernetes Jenkins Kafka Kubernetes TensorFlow
  • 22. Deploying Kubernetes 1. Prerequisites 2. Installing the Client Tools 3. Provisioning Compute Resources 4. Provisioning the CA and Generating TLS Certificates 5. Generating Kubernetes Configuration Files for Authentication 6. Generating the Data Encryption Config and Key 7. Bootstrapping the etcd Cluster… 3x for HA 10. Bootstrapping the Kubernetes Control Plane… 3x for HA 13. Bootstrapping the Kubernetes Worker Nodes 14. Configuring kubectl for Remote Access 15. Provisioning Pod Network Routes 16. Deploying the DNS Cluster Add-on… Deploying other Add-ons 20. Smoke Test 21. Cleaning Up Running on your own $ dcos package install kubernetes on
  • 23. Encrypted Communication 1. Generate certificates and keys using DC/OS PKI 2. Store certificates in DC/OS Secret Store 3. Distribute certificates and keys to data services 4. Apply security configuration 5. Perform rolling restart of the service Each data service has its own distinct security procedure, which sometimes change per version Kafka Cassandra Elastic Spark HDFS
  • 24. Complete Lifecycle $ dcos hdfs plan list [ "deploy", "recovery", "replace", "update" ] dcos hdfs plan show deploy deploy (serial strategy) (IN_PROGRESS) ├─ journal (parallel strategy) (IN_PROGRESS) │ ├─ journal-0:[node] (COMPLETE) │ ├─ journal-1:[node] (COMPLETE) │ └─ journal-2:[node] (PREPARED) ├─ name (serial strategy) (PENDING) │ ├─ name-0:[format] (PENDING) │ ├─ name-0:[node] (PENDING) │ ├─ name-1:[bootstrap] (PENDING) │ └─ name-1:[node] (PENDING) ├─ zkfc (serial strategy) (PENDING) │ ├─ name-0:[zkfc-format] (PENDING) │ ├─ name-0:[zkfc] (PENDING) │ └─ name-1:[zkfc] (PENDING) └─ data (parallel strategy) (PENDING) ├─ data-0:[node] (PENDING) ├─ data-1:[node] (PENDING) └─ data-2:[node] (PENDING)
  • 25. 25
  • 26. Challenge 2: Cluster Management Typical Datacenter siloed, over-provisioned servers, low utilization Kubernetes Jenkins Kafka Kubernetes TensorFlow
  • 27. Challenge 2: Cluster Management Typical Datacenter siloed, over-provisioned servers, low utilization Mesos/ DC/OS automated schedulers, workload multiplexing onto the same machines Kubernetes Jenkins Kafka Kubernetes TensorFlow
  • 28. © 2018 Mesosphere, Inc. All Rights Reserved. Powered by Apache Mesos
  • 29. Stateful Services Kelsey Hightower Kubernetes Thought Leader Ranked #1 K8s Influencer Staff Developer Advocate PM & Chief Advocate Top Kubernetes Advocate
  • 30. CRDs
  • 33. Multiple Kubernetes Cluster 33 Powered by Apache Mesos Line of Business 1 Line of Business 2 Developer Cluster Shared Production Skaffold
  • 34. Multiple Kubernetes Cluster 34 Powered by Apache Mesos Line of Business 1 Line of Business 2 Developer Cluster Shared Production Skaffold
  • 35. Resource Quota 35 • Share resources between multiple services • No static partitioning • One role per job/entity • Use quota per role • Min and Max resource allocation
  • 36. Challenge 4: Hybrid Cloud Powered by Apache Mesos
  • 38. © 2018 Mesosphere, Inc. All Rights Reserved. 38 THANK YOU! ANY QUESTIONS? @mesosphere users@dcos.io /groups/8295652 /dcos /dcos/examples /dcos/demos chat.dcos.io
  • 39. CONFIDENTIAL Make it insanely easy to build and scale world-changing technology