How Puppet Enables the Use of Lightweight Virtualized Containers - PuppetConf 2014

2014
Presented by
How Puppet Enables
the Use of Lightweight
Virtualized Containers
Jeff McCune
Software Developer | Puppet Labs
@0xEFF

What we’ll cover
Compare and contrasts a Jenkins+LDAP deployed onto
virtual machines and onto lightweight containers.
Presented by
!
Jenkins and LDAP Puppet Forge Modules
Migrating from a shared VM to containers
Common Problems with Service Management
Summary of Lessons Learned

Presented by
Starting Point
https://github.com/jeffmccune/puppetconf2014
Traditional VM
Puppet deploys Jenkins with LDAP security
http://sharedvm:18080

Presented by
Shared VM
Vagrant VM
Puppet
Jenkins OpenLDAP
Puppet configures Jenkins
and OpenLDAP
Vagrant Puppet
provisioner

Presented by
Migrating to Containers
Vagrant VM
Puppet at runtime
Jenkins OpenLDAP
Docker Host
!
!
OpenLDAP
Container
Puppet at
build !
time
!
Jenkins
Container
Puppet at
build time

Puppet Forge module re-use
It is faster to deploy Jenkins with LDAP using forge modules:
rtyler/jenkins module
90% code re-use: 831 lines from the Forge, 88 lines added
camptocamp/openldap module
85% code re-use: 584 lines from the Forge, 108 lines added
Presented by

Presented by
Site Specific Customizations
Initialize LDAP for dc=jeffmccune,dc=net
Configure LDAP admin account and password
Load initial schema and base OU’s into LDAP
Configure Jenkins to use LDAP security

Moving from a VM to a Container
Challenges in Puppet:
Service resources fail and must be overridden
Downstream resources require the service resource
LDAP initial schema load requires Service[slapd]
Presented by

Why do service resources fail?
Two different models of containers:
Those with and without a service management framework.
In either case there is no service management framework
running during the image build phase.
Presented by

Presented by
Containers
Two Models of Containers:
OS Virtualization Model - Heavy
(Solaris Zones, FreeBSD Jails, IBM WPAR’s)
Microservice Model - Light
(Docker, CoreOS, Kubernetes)

False Dichotomy over “Light”
Weight is a range, not a boolean.
Single-process image
Multi-process image
Full-os image
Runtime manageability decreases as processes decrease
Presented by

Single-process Microservices
In general:
Executes the application process directly, no init system
startup time comparable to normal process startup
Puppet does not operate at runtime
Difficult to manage with volumes and sidekick processes
Presented by

Multi-process Microservices
In general:
Start an init system (SysV init, systemd, upstart)
The service management framework manages services
Runs ~5 or more processes per container
Puppet works at runtime without modification
Presented by

Full OS Virtualization Containers
Run everything a traditional OS would run
Dozens of running processes in the container
Long startup time (> 30 seconds)
Easier to manage at runtime
Better for some situations, e.g. mimicking a full running OS
Presented by

Container Images
Microservice containers have a distinct build phase
The result of the build step is an image, ideally immutable
Instances execute from a built base image
The Dockerfile is an example of a build script
Presented by

Dockerfile
FROM centos:centos7
ADD puppet.tar.gz /
RUN puppet apply -v --modulepath=/puppet/modules
/puppet/manifests/ldap.pp
EXPOSE 389
CMD ["/usr/sbin/init"]
Presented by

Presented by
docker build
$ docker build --tag base_image .
Sending build context to Docker daemon 4.718 MB
Step 0 : FROM centos:centos7
---> 3236320e3562
Step 1 : ADD puppet.tar.gz /
---> bc9aa11de092
Step 2 : RUN puppet apply …
---> Running in efdda7633bfc

docker build
$ docker build --tag base_image .
Sending build context to Docker daemon 4.718 MB
Step 0 : FROM centos:centos7
---> 3236320e3562
Step 1 : ADD puppet.tar.gz /
---> bc9aa11de092
Step 2 : RUN puppet apply …
---> Running in efdda7633bfc
Presented by

Docker Build with Puppet Try #1
Error: /Service[slapd]: Could not evaluate:
Could not find init script for 'slapd'
Warning: Openldap_database[dc=jeffmccune,dc=net]:
Skipping because of failed dependencies
Warning: Exec[inetorgperson schema]:
Skipping because of failed dependencies
And so on for all dependent resources…
Presented by

systemd is not installed
The error, Could not find init script for ‘slapd’ is caused by
a fake systemd in the base centos image
$ docker run -i -t centos rpm -qa | grep systemd
fakesystemd-1-15.el7.centos.noarch
systemd-libs-208-11.el7_0.2.x86_64
Presented by

Fix #1 Install the real systemd
Replace fakesystemd with the real deal
microservice => virtualized os model
Start a minimum number of services with systemd
New base image: centos7vps with systemd
Presented by

Presented by
Fix #1 Dockerfile
FROM centos:centos7
RUN yum -y swap
-- remove fakesystemd
-- install systemd systemd-libs
# RUN rm unit files in /{lib,etc}/systemd/system

Presented by
Build new base image
$ docker build -t centos7vps .
This base image has the real systemd installed.

New Dockerfile
FROM centos7vps # <= was centos:centos7
ADD puppet.tar.gz /
RUN puppet apply -v --modulepath=/puppet/modules
/puppet/manifests/ldap.pp
EXPOSE 389
Presented by

Presented by
Working for docker run…
Puppet works inside a running docker container
systemd is running

Presented by
Start the systemd container
$ CID=$(docker run --privileged -d centos7vps)
Enter the container
$ sudo /usr/local/bin/docker-enter $CID

Presented by
Run Puppet:
bash-4.2# puppet apply /puppet/manifests/ldap.pp
Compiled catalog for ldap in 1.05 seconds
Service[slapd]/ensure: 'stopped' => 'running'
Finished catalog run in 30.15 seconds

Not working for docker build
systemd is present, but not running during the build phase
$ docker build -t ldap:vps .
…
Error: Could not start Service[slapd]:
Execution of '/usr/bin/systemctl start slapd'
returned 1: Failed to get D-Bus connection:
No connection to service manager.
Presented by

Moving ahead
We could stop now and simply run puppet every time the
container starts prior to the application starting.
This would increase service start time.
Application images become mutable, managed by Puppet.
Doesn’t fit the microservice model very well.
Presented by

Goal: Puppet configured images
Puppet configures immutable pre-configured base
application images
Immutable images provide known good state
Closer to the microservice model
Compatible with the os virtualization model
Presented by

Docker build with Puppet Try #2
Override the Service[slapd] to avoid systemd at build time
class ldap_override inherits openldap::server::service {
Service[slapd] {
Presented by
ensure => running,
enable => undef,
start => '/usr/sbin/slapd -u ldap -h "ldapi:/// ldap:///"
}
}

Success!
We’re able to build an immutable application image fully
configured by Puppet.
Presented by

Presented by
LDAP running in systemd
/usr/bin/docker -d --selinux-enabled
_ /usr/sbin/init
_ /usr/lib/systemd/systemd-journald
_ /bin/dbus-daemon --system
_ /usr/sbin/slapd -u ldap -h ldapi:/// ldap:///

Moving further ahead
We could stop here, but there are a number of trade-offs
systemd requires the container to run in privileged mode
More processes: systemd, journald, dbus, application
Slower startup / tear-down time (~5 seconds)
Still faster than a VM (~30+ seconds)
Presented by

Presented by
Final Dockerfile
FROM centos7vps
ADD puppet.tar.gz /
RUN puppet apply -v /puppet/manifests/ldap.pp
EXPOSE 389
CMD ["/usr/sbin/slapd", "-u", "ldap",
"-h", "ldapi:/// ldap:///", "-d", "0"]

docker build ldap:microservice
Build is required for deployment
$ docker build -t ldap:microservice .
Presented by

Presented by
Single-process Microservice
Only one process running in the container:
/usr/bin/docker -d --selinux-enabled
_ /usr/sbin/slapd -u ldap -h ldapi:/// ldap:/// -d 0
Start / Stop time: ~1 second

Puppetized Microservice
The application image is immutable
Changes to the application require a new image build
Centralized reporting via Puppet
version control via Puppet code
Re-usable modules from the Puppet forge
Presented by

Putting it together
Link the jenkins microservice with the ldap microservice
$ docker run -d --name ldap ldap:microservice
$ docker run -d --name jenkins
--link ldap:ldap
--publish 18080:8080
jenkins:microservice
Presented by

Presented by
Results
The application (jenkins) is now isolated
With isolated dependencies included (ldap)
The applications are immutable
Multiple copies can easily be deployed
Difficult to change running instances

Making a change to LDAP
Need to modify the LDAP indexes as an optimization
1: Update Puppet Configuration
2: Rebuild LDAP image
3: Re-deploy LDAP
4: Re-deploy Jenkins (The link to LDAP is static)
Presented by

Advantages and Disadvantages
Immutable known good state
Deployment is highly repeatable and consistent
App and Dependency are tightly coupled
Ambassador Pattern decouples services (with tedium)
Presented by
!

Summary
Forge modules are re-usable in Docker containers
Service resources pose a challenge
Override the service resource to build immutable images
Multi-process images with systemd are an option
Presented by

Presented by
Thank you!
Code:
github.com/jeffmccune/puppetconf2014
David Lutterkort’s Dockercon talk:
http://links.puppetlabs.com/lutter-docker
!

How Puppet Enables the Use of Lightweight Virtualized Containers - PuppetConf 2014

More Related Content

How Puppet Enables the Use of Lightweight Virtualized Containers - PuppetConf 2014