Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo
State of the Container Ecosystem
prepared for Huawei
Vinay Rao
Former Chief Architect
Formation Data Systems Inc
Static website
Web frontend
User DB
Queue Analytics DB
Background workers
API endpoint
nginx 1.5 + modsecurity + openssl +
bootstrap 2
postgresql + pgv8 + v8
hadoop + hive + thrift + OpenJDK
Ruby + Rails + sass + Unicorn
Redis + redis-sentinel
Python 3.0 + celery + pyredis + libcurl + ffmpeg +
libopencv + nodejs + phantomjs
Python 2.7 + Flask + pyredis + celery + psycopg +
postgresql-client
Development VM
QA server
Public Cloud
Disaster recovery
Contributor’s laptop
Production Servers
Docker Basics: The Challenge
Multiplicityof
Stacks
Multiplicityof
hardware
environments
Production Cluster
Customer Data Center
Doservicesand
appsinteract
appropriately?
CanImigrate
smoothlyand
quickly?
Docker Basics: The Matrix From Hell
Static website
Web frontend
Background workers
User DB
Analytics DB
Queue
Developm
ent VM
QA Server
Single
Prod
Server
Onsite
Cluster
Public
Cloud
Contribut
or’s
laptop
Customer
Servers
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
Static website Web frontendUser DB Queue Analytics DB
Developm
ent VM
QA server Public Cloud Contributor’s
laptop
Docker is a shipping container system for code
MultiplicityofStacks
Multiplicityof
hardware
environments
Production
Cluster
Customer Data
Center
Doservicesandapps
interact
appropriately?
CanImigrate
smoothlyandquickly
…that can be manipulated using
standard operations and run
consistently on virtually any
hardware platform
An engine that enables
any payload to be
encapsulated as a
lightweight, portable, self-
sufficient container…
Static website
Web frontend
Background workers
User DB
Analytics DB
Queue
Developm
ent VM
QA Server
Single
Prod
Server
Onsite
Cluster
Public
Cloud
Contribut
or’s
laptop
Customer
Servers
Docker eliminates the matrix from Hell
Why Developers Care
• Build once…(finally) run anywhere
• A clean, safe, hygienic and portable runtime environment for your app.
• No worries about missing dependencies, packages and other pain points
during subsequent deployments.
• Run each app in its own isolated container, so you can run various
versions of libraries and other dependencies for each app without worrying
• Automate testing, integration, packaging…anything you can script
• Reduce/eliminate concerns about compatibility on different platforms,
either your own or your customers.
• Cheap, zero-penalty containers to deploy services? A VM without the
overhead of a VM? Instant replay and reset of image snapshots? That’s
the power of Docker
Why it works—separation of concerns
• Dan the Developer
• Worries about what’s “inside” the
container
• His code
• His Libraries
• His Package Manager
• His Apps
• His Data
• All Linux servers look the same
• Oscar the Ops Guy
• Worries about what’s “outside” the
container
• Logging
• Remote access
• Monitoring
• Network config
• All containers start, stop, copy,
attach, migrate, etc. the same way
App
A
Containers vs. VMs
Hypervisor (Type 2)
Host OS
Server
Gues
t
OS
Bins/
Libs
App
A’
Gues
t
OS
Bins/
Libs
App
B
Gues
t
OS
Bins/
Libs
AppA’
Docker
Host OS
Server
Bins/Libs
AppA
Bins/Libs
AppB
AppB’
AppB’
AppB’
VM
Container
Containers are isolated,
but share OS and, where
appropriate, bins/libraries
Gues
t
OS
Gues
t
OS
…result is significantly faster
deployment, much less overhead,
easier migration, faster restart
What makes containers lightweight?
App
A
Original App
(No OS to take
up space, resources,
or require restart)
AppΔ
Bins
/
App
A
App
A’
Modified App
Copy on write
capabilities allow
us to only save the
diffs
Between container A
and container
A’
VMs
Every app, every copy of an
app, and every slight modification
of the app requires a new virtual server
Copy of
App
No OS. Can
Share bins/libs
App
A’
Guest
OS
Guest
OS
VMs Containers
Bins/
Libs
Bins/
Libs
Bins/
Libs
App
A
Guest
OS
Bins/
Libs
Basics of the Docker system
Source Code
Repository
Dockerfile
For
A
Docker Engine
Docker
Containe
r
Image
Registry
Build
Docker
Host 2 OS (Linux)
ContainerA
ContainerB
ContainerC
ContainerA
Push
Search Pull
Run
Host 1 OS (Linux)
Docker Basics: The Internals
Docker Daemon
(dockerd)
libcontainer
cgroups namespace
AUFS
BTRFS
VFS
User
Space
Kernel
Docker Basics: The Internals
• Namespaces provides the isolated workspace of a container.
• The pid namespace: Process isolation (PID: Process ID).
• The net namespace: Managing network interfaces (NET: Networking).
• The ipc namespace: Managing access to IPC resources (IPC: InterProcess Communication).
• The mnt namespace: Managing filesystem mount points (MNT: Mount).
• The uts namespace: Isolating kernel and version identifiers. (UTS: Unix Timesharing System)
• Docker uses control groups or cgroups to limit an application to a specific set of
resources. For example, you can limit the memory available to a specific
container.
• Docker Engine uses UnionFS to provide the building blocks for containers.
Docker Engine can use AUFS, btrfs, vfs, and DeviceMapper.
Docker Basics: Filesystems
Union File System
Snapshot
FileSystem
Copy-on-write Block
Dev
Provisioning
Superfast
supercheap
fast/cheap fast/cheap
Small Files
Superfast
supercheap
fast/cheap fast/costly
Large Files
slow(first-time)
Inefficient
fast/cheap fast/cheap
Diffing superfast superfast slow
Memory usage Efficient Efficient Inefficient
Implementation AUFS BTRFS ZFS
Container ecosystem components
• Container Orchestration
• Kubernetes
• Docker Swarm
• Apache Mesos
• Networking
• Flannel
• Weave
• Storage
• Native filesystems.
• Volume Plugins for NAS/SAN, DAS
• Storage virtualization platforms - portworx, quobyte
• Container security
• Application Lifecycle Management
• Rancher Labs
• Robin Systems
Kubernetes Container Orchestration
Key Concepts of Kubernetes
● Pod - A group of Containers
● Labels - Labels for identifying pods
● Kubelet - Container Agent
● Proxy - A load balancer for Pods
● etcd - A metadata service for service discovery
● cAdvisor - Container Advisor provides resource
usage/performance statistics
● Replication Controller - Manages replication
of pods
● Scheduler - Schedules pods in worker nodes
● API Server - Kubernetes API server
Docker Swarm Architecture
Native Docker Networking - Not cloud ready!!
• Docker Containers on different machines cannot reach each
other
• Docker on different machines can assign conflicting network
ranges and IP addresses.
• Solution is to allocate unique ports on the machine’s own IP
address, which are then forwarded or proxied to the containers.
• Flannel and Weave are the real cloud-ready solutions using
overlay networking architecture.
Container Networking with Flannel
• Flannel is a basic overlay network that works by assigning a
range of subnet addresses (usually IPv4 with a /24 or /16
subnet mask).
• Flannel gives each host an IP subnet (/24 by default) from
which the Docker daemon is able to allocate IPs to the
individual containers.
• Allows each container to have its own IP host address allowing
true distributed container clustering on different hosts.
Container Networking with Flannel
•
Container Networking using Weave
Container Networking - Weave
• Weave Net use standard port numbers, (for example MySQL’s default is port 3306),
• Weave Net uses Encapsulation based overlay. It has a Weave router and bridge modules
in each host.
• Weave Net implements service discovery by providing a fast “micro DNS” server at each
node. You simply name containers and everything ‘just works’, including load balancing
across multiple containers with the same name.
• Weave Net is bundled with a Docker Network plugin that doesn’t require an external cluster
store. You can get started right away and you can start and stop containers even when
there are network connectivity problems.
• Operates in Partially Connected Networks - works well in a mix of legacy systems and
containerized apps environment
Storage for Containers
• A data volume is a specially-designated directory within one or
more containers that bypasses the Union File System.
• Volumes are initialized when a container is created. If the
container’s base image contains data at the specified mount point,
that existing data is copied into the new volume upon volume
initialization.
• Data volumes can be shared and reused among containers.
• Changes to a data volume are made directly.
• Changes to a data volume will not be included when you update an
image.
• Data volumes persist even if the container itself is deleted.
Persistence
IOPs
Connectors
SSD
HDD SAN
EBS
S3
Admins Realize that Apps Don’t Want Volumes
1. Don’t make data protection a science project
2. Stop trying to connect legacy SAN/NAS/SDS to containers – They are not
container or application aware
3. Your applications want on-demand services delivered programmatically via
the scheduler
4. Storage services need to be “app-aware”, for example Cassandra-aware –
Traditional storage is not.
Storage for Containers - Missing pieces
• What is really required is:
• A storage virtualization layer that encapsulates SAN, NAS, DAS or public-cloud
storage elements such as S3/EBS
• Specify application level QoS as IOPS/throughput and enforcement down the
stack.
• Control snapshot/cloning schedules per data volume based on application
characteristics such as data churn, RPO/RTO, business criticality.
• Control tiering between various storage media in the backend to match the
application performance requirements.
• Storage container motion capability for migration of storage containers to a
different pod or across data centers.
• Seamless integration with application management layers.
• Integration with container orchestration layers
PortWorx (PX): Data services management
WordPres
s
WordPres
s
PX
MySQL
MySQL
persistence
encryption
CoS
namespace
Auth/ACLs
HA
Operational Experience
Delivered as a lightweight
container with support for
volumes and graph. Runs on
any server, any cloud.
Converged Performance
Data collocated with
container compute
Container Granular Controls
Encryption, CoS, replication,
snapshots are done at
container granularity
Hybrid Cloud Ops
Cloud agnostic deployment
and data accessibility across
multiple clouds
S3 Object
Portworx vs ScaleIO or CEPH : Unique Architectural Tenet
x86 Servers x86 Servers
rack
…
x86 Servers x86 Servers
rack
Portworx Places an Application’ss Data
Local to a Few Topology Constrained
Servers. Cassandra runs converged.
x86 Servers x86 Servers
rack
…
x86 Servers x86 Servers
rack
x86 Servers
ScaleIO and CEPH shard an application’s
data across many servers to get capacity.
This breaks convergence. You also don’t
have application granular settings on the
volumes.
: Complete Container Management Platforms
RUN CONTAINERS
• with the most complete set of container and infrastructure
management capabilities
MANAGE APPLICATIONS
• by simplifying day to day application lifecycle management
INNOVATE WITH CONTAINERS
• without compromising flexibility by empowering developers with
fast access to the latest tools
Production ready
✔ 20 million+
downloads
✔ Open platform for
innovating
✔ Easy to use interface
✔ Multi-tenant
✔ Role based access
✔ 24X7 support
RancherOS: Complete Container Management Platform
Application Catalog
Container Orch. & SchedulingUser Mgmt
RBAC
AD/LDAP
SAML
Ops Mgmt
CI/CD
Registries
Monitoring
Networking
Multi-tenant Environments
Environment 1 Environment N
Infrastructure Services
Storage
……. ..Environment 2
Security DNS/LB
RancherOS: Volume Plugin Manager
• Rancher is your Docker volume plugin manager and
provides the following:
• Distribution – automatically deploys volume plugins to desired hosts
managed by Rancher so they are made available immediately
• Version Management and Upgrades – provides version
management and allows users to automatically update plugins on
all hosts with a single click (or API call)
• Lifecycle Management – Provides health checks and automatic
redeployment of plugins if things go wrong
• Plugin-aware Container Scheduling – provides container scheduling
based on deployment of plugins so you never schedule containers
on hosts that have no access to storage
Diamanti - Appliances for converged container clouds
• Network and Storage QoS -Enforced performance policies
across container cloud users
• Multi-tenant isolation with role-based access policies.
• Built-in Kubernetes and Docker with zero effort integration
• Add container cloud capacity in seconds, one appliance at a time
• Utilization and capacity monitoring for all containers and
resources
• Selectable Docker network configuration per container
• On-demand low-latency persistent flash storage per Docker
container
Conclusions
• Integrate storage and networking stacks to support
production (persistent) workloads
• Keep a check on Operational complexity
• Application lifecycle management integration is key.
• Container orchestration enables cross-cloud
migration in a hybrid cloud model.

More Related Content

State of the Container Ecosystem

  • 1. State of the Container Ecosystem prepared for Huawei Vinay Rao Former Chief Architect Formation Data Systems Inc
  • 2. Static website Web frontend User DB Queue Analytics DB Background workers API endpoint nginx 1.5 + modsecurity + openssl + bootstrap 2 postgresql + pgv8 + v8 hadoop + hive + thrift + OpenJDK Ruby + Rails + sass + Unicorn Redis + redis-sentinel Python 3.0 + celery + pyredis + libcurl + ffmpeg + libopencv + nodejs + phantomjs Python 2.7 + Flask + pyredis + celery + psycopg + postgresql-client Development VM QA server Public Cloud Disaster recovery Contributor’s laptop Production Servers Docker Basics: The Challenge Multiplicityof Stacks Multiplicityof hardware environments Production Cluster Customer Data Center Doservicesand appsinteract appropriately? CanImigrate smoothlyand quickly?
  • 3. Docker Basics: The Matrix From Hell Static website Web frontend Background workers User DB Analytics DB Queue Developm ent VM QA Server Single Prod Server Onsite Cluster Public Cloud Contribut or’s laptop Customer Servers ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
  • 4. Static website Web frontendUser DB Queue Analytics DB Developm ent VM QA server Public Cloud Contributor’s laptop Docker is a shipping container system for code MultiplicityofStacks Multiplicityof hardware environments Production Cluster Customer Data Center Doservicesandapps interact appropriately? CanImigrate smoothlyandquickly …that can be manipulated using standard operations and run consistently on virtually any hardware platform An engine that enables any payload to be encapsulated as a lightweight, portable, self- sufficient container…
  • 5. Static website Web frontend Background workers User DB Analytics DB Queue Developm ent VM QA Server Single Prod Server Onsite Cluster Public Cloud Contribut or’s laptop Customer Servers Docker eliminates the matrix from Hell
  • 6. Why Developers Care • Build once…(finally) run anywhere • A clean, safe, hygienic and portable runtime environment for your app. • No worries about missing dependencies, packages and other pain points during subsequent deployments. • Run each app in its own isolated container, so you can run various versions of libraries and other dependencies for each app without worrying • Automate testing, integration, packaging…anything you can script • Reduce/eliminate concerns about compatibility on different platforms, either your own or your customers. • Cheap, zero-penalty containers to deploy services? A VM without the overhead of a VM? Instant replay and reset of image snapshots? That’s the power of Docker
  • 7. Why it works—separation of concerns • Dan the Developer • Worries about what’s “inside” the container • His code • His Libraries • His Package Manager • His Apps • His Data • All Linux servers look the same • Oscar the Ops Guy • Worries about what’s “outside” the container • Logging • Remote access • Monitoring • Network config • All containers start, stop, copy, attach, migrate, etc. the same way
  • 8. App A Containers vs. VMs Hypervisor (Type 2) Host OS Server Gues t OS Bins/ Libs App A’ Gues t OS Bins/ Libs App B Gues t OS Bins/ Libs AppA’ Docker Host OS Server Bins/Libs AppA Bins/Libs AppB AppB’ AppB’ AppB’ VM Container Containers are isolated, but share OS and, where appropriate, bins/libraries Gues t OS Gues t OS …result is significantly faster deployment, much less overhead, easier migration, faster restart
  • 9. What makes containers lightweight? App A Original App (No OS to take up space, resources, or require restart) AppΔ Bins / App A App A’ Modified App Copy on write capabilities allow us to only save the diffs Between container A and container A’ VMs Every app, every copy of an app, and every slight modification of the app requires a new virtual server Copy of App No OS. Can Share bins/libs App A’ Guest OS Guest OS VMs Containers Bins/ Libs Bins/ Libs Bins/ Libs App A Guest OS Bins/ Libs
  • 10. Basics of the Docker system Source Code Repository Dockerfile For A Docker Engine Docker Containe r Image Registry Build Docker Host 2 OS (Linux) ContainerA ContainerB ContainerC ContainerA Push Search Pull Run Host 1 OS (Linux)
  • 11. Docker Basics: The Internals Docker Daemon (dockerd) libcontainer cgroups namespace AUFS BTRFS VFS User Space Kernel
  • 12. Docker Basics: The Internals • Namespaces provides the isolated workspace of a container. • The pid namespace: Process isolation (PID: Process ID). • The net namespace: Managing network interfaces (NET: Networking). • The ipc namespace: Managing access to IPC resources (IPC: InterProcess Communication). • The mnt namespace: Managing filesystem mount points (MNT: Mount). • The uts namespace: Isolating kernel and version identifiers. (UTS: Unix Timesharing System) • Docker uses control groups or cgroups to limit an application to a specific set of resources. For example, you can limit the memory available to a specific container. • Docker Engine uses UnionFS to provide the building blocks for containers. Docker Engine can use AUFS, btrfs, vfs, and DeviceMapper.
  • 13. Docker Basics: Filesystems Union File System Snapshot FileSystem Copy-on-write Block Dev Provisioning Superfast supercheap fast/cheap fast/cheap Small Files Superfast supercheap fast/cheap fast/costly Large Files slow(first-time) Inefficient fast/cheap fast/cheap Diffing superfast superfast slow Memory usage Efficient Efficient Inefficient Implementation AUFS BTRFS ZFS
  • 14. Container ecosystem components • Container Orchestration • Kubernetes • Docker Swarm • Apache Mesos • Networking • Flannel • Weave • Storage • Native filesystems. • Volume Plugins for NAS/SAN, DAS • Storage virtualization platforms - portworx, quobyte • Container security • Application Lifecycle Management • Rancher Labs • Robin Systems
  • 16. Key Concepts of Kubernetes ● Pod - A group of Containers ● Labels - Labels for identifying pods ● Kubelet - Container Agent ● Proxy - A load balancer for Pods ● etcd - A metadata service for service discovery ● cAdvisor - Container Advisor provides resource usage/performance statistics ● Replication Controller - Manages replication of pods ● Scheduler - Schedules pods in worker nodes ● API Server - Kubernetes API server
  • 18. Native Docker Networking - Not cloud ready!! • Docker Containers on different machines cannot reach each other • Docker on different machines can assign conflicting network ranges and IP addresses. • Solution is to allocate unique ports on the machine’s own IP address, which are then forwarded or proxied to the containers. • Flannel and Weave are the real cloud-ready solutions using overlay networking architecture.
  • 19. Container Networking with Flannel • Flannel is a basic overlay network that works by assigning a range of subnet addresses (usually IPv4 with a /24 or /16 subnet mask). • Flannel gives each host an IP subnet (/24 by default) from which the Docker daemon is able to allocate IPs to the individual containers. • Allows each container to have its own IP host address allowing true distributed container clustering on different hosts.
  • 22. Container Networking - Weave • Weave Net use standard port numbers, (for example MySQL’s default is port 3306), • Weave Net uses Encapsulation based overlay. It has a Weave router and bridge modules in each host. • Weave Net implements service discovery by providing a fast “micro DNS” server at each node. You simply name containers and everything ‘just works’, including load balancing across multiple containers with the same name. • Weave Net is bundled with a Docker Network plugin that doesn’t require an external cluster store. You can get started right away and you can start and stop containers even when there are network connectivity problems. • Operates in Partially Connected Networks - works well in a mix of legacy systems and containerized apps environment
  • 23. Storage for Containers • A data volume is a specially-designated directory within one or more containers that bypasses the Union File System. • Volumes are initialized when a container is created. If the container’s base image contains data at the specified mount point, that existing data is copied into the new volume upon volume initialization. • Data volumes can be shared and reused among containers. • Changes to a data volume are made directly. • Changes to a data volume will not be included when you update an image. • Data volumes persist even if the container itself is deleted.
  • 24. Persistence IOPs Connectors SSD HDD SAN EBS S3 Admins Realize that Apps Don’t Want Volumes 1. Don’t make data protection a science project 2. Stop trying to connect legacy SAN/NAS/SDS to containers – They are not container or application aware 3. Your applications want on-demand services delivered programmatically via the scheduler 4. Storage services need to be “app-aware”, for example Cassandra-aware – Traditional storage is not.
  • 25. Storage for Containers - Missing pieces • What is really required is: • A storage virtualization layer that encapsulates SAN, NAS, DAS or public-cloud storage elements such as S3/EBS • Specify application level QoS as IOPS/throughput and enforcement down the stack. • Control snapshot/cloning schedules per data volume based on application characteristics such as data churn, RPO/RTO, business criticality. • Control tiering between various storage media in the backend to match the application performance requirements. • Storage container motion capability for migration of storage containers to a different pod or across data centers. • Seamless integration with application management layers. • Integration with container orchestration layers
  • 26. PortWorx (PX): Data services management WordPres s WordPres s PX MySQL MySQL persistence encryption CoS namespace Auth/ACLs HA Operational Experience Delivered as a lightweight container with support for volumes and graph. Runs on any server, any cloud. Converged Performance Data collocated with container compute Container Granular Controls Encryption, CoS, replication, snapshots are done at container granularity Hybrid Cloud Ops Cloud agnostic deployment and data accessibility across multiple clouds S3 Object
  • 27. Portworx vs ScaleIO or CEPH : Unique Architectural Tenet x86 Servers x86 Servers rack … x86 Servers x86 Servers rack Portworx Places an Application’ss Data Local to a Few Topology Constrained Servers. Cassandra runs converged. x86 Servers x86 Servers rack … x86 Servers x86 Servers rack x86 Servers ScaleIO and CEPH shard an application’s data across many servers to get capacity. This breaks convergence. You also don’t have application granular settings on the volumes.
  • 28. : Complete Container Management Platforms RUN CONTAINERS • with the most complete set of container and infrastructure management capabilities MANAGE APPLICATIONS • by simplifying day to day application lifecycle management INNOVATE WITH CONTAINERS • without compromising flexibility by empowering developers with fast access to the latest tools Production ready ✔ 20 million+ downloads ✔ Open platform for innovating ✔ Easy to use interface ✔ Multi-tenant ✔ Role based access ✔ 24X7 support
  • 29. RancherOS: Complete Container Management Platform Application Catalog Container Orch. & SchedulingUser Mgmt RBAC AD/LDAP SAML Ops Mgmt CI/CD Registries Monitoring Networking Multi-tenant Environments Environment 1 Environment N Infrastructure Services Storage ……. ..Environment 2 Security DNS/LB
  • 30. RancherOS: Volume Plugin Manager • Rancher is your Docker volume plugin manager and provides the following: • Distribution – automatically deploys volume plugins to desired hosts managed by Rancher so they are made available immediately • Version Management and Upgrades – provides version management and allows users to automatically update plugins on all hosts with a single click (or API call) • Lifecycle Management – Provides health checks and automatic redeployment of plugins if things go wrong • Plugin-aware Container Scheduling – provides container scheduling based on deployment of plugins so you never schedule containers on hosts that have no access to storage
  • 31. Diamanti - Appliances for converged container clouds • Network and Storage QoS -Enforced performance policies across container cloud users • Multi-tenant isolation with role-based access policies. • Built-in Kubernetes and Docker with zero effort integration • Add container cloud capacity in seconds, one appliance at a time • Utilization and capacity monitoring for all containers and resources • Selectable Docker network configuration per container • On-demand low-latency persistent flash storage per Docker container
  • 32. Conclusions • Integrate storage and networking stacks to support production (persistent) workloads • Keep a check on Operational complexity • Application lifecycle management integration is key. • Container orchestration enables cross-cloud migration in a hybrid cloud model.

Editor's Notes

  1. Specific services provided by PXß
  2. Rancher is a complete container management platform that makes it easy to… Run containers in production with the most complete set of container and infrastructure management capabilities Manage applications by simplifying day to day application lifecycle management Innovate with containers without compromising flexibility by empowering developers with fast access to the latest tools And Rancher is production ready Enterprise-class features such as role-based access control, integration with LDAP and Active Directories, detailed audit logs, high-availability management servers and encrypted networking are available out of the box. With over 2.4 million downloads and optional 24x7x365 support, Rancher has quickly become the platform of choice for devops and IT teams who are serious about running containers in production More than 30 customers are using Rancher in production including large enterprise such as US Bank, IBM, Cisco, Invisalign and more.