Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

K8S in prod

•Download as PPTX, PDF•
•
M
Mageshwaran Rajendran

This document discusses Kubernetes usage at VMware SAAS. It covers dynamic provisioning of applications on Kubernetes, monitoring tools used like DataDog and Log Insight, and best practices for upgrading Kubernetes clusters. Key points include using stateless applications where possible, service discovery using Kubernetes services, dynamic provisioning using an onboarding service, and performing rolling upgrades for stateful applications to minimize downtime.

Related slideshows

Kubernetes Introduction & Whats new in Kubernetes 1.6
Kubernetes Introduction & Whats new in Kubernetes 1.6Kubernetes Introduction & Whats new in Kubernetes 1.6
Kubernetes Introduction & Whats new in Kubernetes 1.6
 
Kubernetes 101
Kubernetes 101Kubernetes 101
Kubernetes 101
 
Kubernetes One-Click Deployment: Hands-on Workshop (Munich)
Kubernetes One-Click Deployment: Hands-on Workshop (Munich)Kubernetes One-Click Deployment: Hands-on Workshop (Munich)
Kubernetes One-Click Deployment: Hands-on Workshop (Munich)
 
1 of 15
© 2016 VMware Inc. All rights reserved.© 2016 VMware Inc. All rights reserved. Kubernetes in PROD VMware SAAS (Cloud Management BU) Mageshwaran R Kumar Gaurav
VMware SAAS 2 1 K8S Usage 2 Dynamic Provisioning 3 Monitoring & Upgrade VMware SAAS in the current context refers to CMBU initiative alone
VMware SAAS: Overview • MCM solution combining the capabilities of Application orchestration, monitoring, management, costing etc • SaaS first, platform not a product • Enables DevOps engineers and Developers to be more agile and leverage resources from any cloud. • Built using variety of tech Stack – Distributed Micro-services – Java, Scala, Spring, Guice, Xenon etc., – Relational databases, Key value stores, Document stores etc., VMware SAAS in the current context refers to CMBU initiative alone 3
Types of Application • Green Field (New) Apps – Xenon based Micro-services – Distributed – Soft states often categorized to Stateless apps • Brown Field (Existing) Apps – Existing applications follows n-tier architecture – Containerization – Leverages K8S Pod – Separate Stateful & Stateless apps 4
Deployments • Core business logic are modelled as stateless containers – Java & Scala applications – Load based replica sets, auto scale • Soft state applications – Xenon services & distributed task processing – Distributed applications uses remote nodes to store the data than disk – Spread pod across nodes (anti-affinity) • Configure Liveness Probe – Health Checks API 5
Stateful Sets • Persistence layer & distributed applications • Mongo DB – Containerized, replicated – Entry point Initializes and configures replica set – Customized Storage class : aws-ebs (type, iops, zone) • Postgres DB – Containerized, Active-stand by – Streaming replication – Pgpool 6
Best Practices • Cloud agnostic – Avoid AWS specific services • Prefer stateless • Service Discovery – Internal communication • Uses FQDN of K8S service name • HTTP, TCP – External Service communication • NGINX/ HA proxy • CI/CD Pipeline using vRealize code stream 7
K8S in PROD 8 1 K8S Usage 2 Dynamic Provisioning 3 Monitoring & Upgrade
Dynamic Provisioning • Use cases – Single instance of N-tier application/distributed application supports “X” tenants – Use case demands due to compliance/ data isolation requirements etc., – Horizontal scaling of application (collection of heterogeneous K8S resources) • Onboarding Service – Runs as a deployment inside cluster – Use JSON templates to create K8S resources – Lazy while provisioning and avoid shuffling – Orchestrates updates 9
Dynamic Provisioning • Control plane – Build using Xenon – K8S REST spec client in Java – Business logic to scale based on the usage & load – Wraps bootstrap logic using containers • Challenges – K8S Version upgrade – Scale down – Environment abstractions 10
K8S in PROD 11 1 K8S Usage 2 Dynamic Provisioning 3 Monitoring & Upgrade
Monitoring • Tools Used – VMWare vRealize Log Insight for log monitoring (Kibana is good too) – Data dog (Grafana is good too) • Support for all standard processes like JVM, RDBMS, Mongo etc., • Docker & Kubernetes • Support for posting custom metrics – Pingdom • Users perspective – Pager duty – Status.io • Communicate health to stakeholders 12
Monitoring • Application – Pod contains Data dog agent & Log insight agent – Captures application metrics • Dropwizard metrics & Xenon Stats • Publishes to data dog – Custom monitoring dashboards in Data dog • Dev Ops UI – Uses Kubernetes auth – Separate from application authn & authz • Fault Injection – Simulates Pod failures – Simulates CPU Usage & network delays 13
Upgrade • Side by side Upgrade – Preferred option for soft state (deployment) applications – Pauses user request momentarily to minimize downtime • Rolling upgrade – Preferred option for Stateful sets – Avoid transformations and provide backward compatibility • Backup & Restore – Soft states are backed up to S3. – EBS are periodically snapshotted to S3 – Periodic restore to Staging – DR scenarios 14
Questions ??? 15

More Related Content

K8S in prod

  • 1. © 2016 VMware Inc. All rights reserved.© 2016 VMware Inc. All rights reserved. Kubernetes in PROD VMware SAAS (Cloud Management BU) Mageshwaran R Kumar Gaurav
  • 2. VMware SAAS 2 1 K8S Usage 2 Dynamic Provisioning 3 Monitoring & Upgrade VMware SAAS in the current context refers to CMBU initiative alone
  • 3. VMware SAAS: Overview • MCM solution combining the capabilities of Application orchestration, monitoring, management, costing etc • SaaS first, platform not a product • Enables DevOps engineers and Developers to be more agile and leverage resources from any cloud. • Built using variety of tech Stack – Distributed Micro-services – Java, Scala, Spring, Guice, Xenon etc., – Relational databases, Key value stores, Document stores etc., VMware SAAS in the current context refers to CMBU initiative alone 3
  • 4. Types of Application • Green Field (New) Apps – Xenon based Micro-services – Distributed – Soft states often categorized to Stateless apps • Brown Field (Existing) Apps – Existing applications follows n-tier architecture – Containerization – Leverages K8S Pod – Separate Stateful & Stateless apps 4
  • 5. Deployments • Core business logic are modelled as stateless containers – Java & Scala applications – Load based replica sets, auto scale • Soft state applications – Xenon services & distributed task processing – Distributed applications uses remote nodes to store the data than disk – Spread pod across nodes (anti-affinity) • Configure Liveness Probe – Health Checks API 5
  • 6. Stateful Sets • Persistence layer & distributed applications • Mongo DB – Containerized, replicated – Entry point Initializes and configures replica set – Customized Storage class : aws-ebs (type, iops, zone) • Postgres DB – Containerized, Active-stand by – Streaming replication – Pgpool 6
  • 7. Best Practices • Cloud agnostic – Avoid AWS specific services • Prefer stateless • Service Discovery – Internal communication • Uses FQDN of K8S service name • HTTP, TCP – External Service communication • NGINX/ HA proxy • CI/CD Pipeline using vRealize code stream 7
  • 8. K8S in PROD 8 1 K8S Usage 2 Dynamic Provisioning 3 Monitoring & Upgrade
  • 9. Dynamic Provisioning • Use cases – Single instance of N-tier application/distributed application supports “X” tenants – Use case demands due to compliance/ data isolation requirements etc., – Horizontal scaling of application (collection of heterogeneous K8S resources) • Onboarding Service – Runs as a deployment inside cluster – Use JSON templates to create K8S resources – Lazy while provisioning and avoid shuffling – Orchestrates updates 9
  • 10. Dynamic Provisioning • Control plane – Build using Xenon – K8S REST spec client in Java – Business logic to scale based on the usage & load – Wraps bootstrap logic using containers • Challenges – K8S Version upgrade – Scale down – Environment abstractions 10
  • 11. K8S in PROD 11 1 K8S Usage 2 Dynamic Provisioning 3 Monitoring & Upgrade
  • 12. Monitoring • Tools Used – VMWare vRealize Log Insight for log monitoring (Kibana is good too) – Data dog (Grafana is good too) • Support for all standard processes like JVM, RDBMS, Mongo etc., • Docker & Kubernetes • Support for posting custom metrics – Pingdom • Users perspective – Pager duty – Status.io • Communicate health to stakeholders 12
  • 13. Monitoring • Application – Pod contains Data dog agent & Log insight agent – Captures application metrics • Dropwizard metrics & Xenon Stats • Publishes to data dog – Custom monitoring dashboards in Data dog • Dev Ops UI – Uses Kubernetes auth – Separate from application authn & authz • Fault Injection – Simulates Pod failures – Simulates CPU Usage & network delays 13
  • 14. Upgrade • Side by side Upgrade – Preferred option for soft state (deployment) applications – Pauses user request momentarily to minimize downtime • Rolling upgrade – Preferred option for Stateful sets – Avoid transformations and provide backward compatibility • Backup & Restore – Soft states are backed up to S3. – EBS are periodically snapshotted to S3 – Periodic restore to Staging – DR scenarios 14