Building a secure image pipeline with Ansible. Generating secure OS images for OpenShift Virtualization. Creating a immutable image pipeline with Ansible, OpenSCAP, Packer, Molecule and Vagrant. Packaging OS images for consumption to OpenShift Virtualization.
1 of 49
More Related Content
AnsibleFest 2021 - DevSecOps with Ansible, OpenShift Virtualization, Packer and OpenSCAP
1. Automate Everything - DevSecOps for OS Images with Ansible
OpenShift Virtualization Image Pipeline
Mihai Criveti, CTO @ IBM Cloud Native and Red Hat Solutions, RHCA
August 19, 2021
1
2. 1) OpenShift Virtualization - OpenShift managed KVM
2) Build KVM images by hand - what are we automating?
3) Automate the OS build steps with Ansible
4) Automate the OS install and trigger Ansible post-install steps with Packer
5) Automate Security and Compliance with OpenSCAP profiles
6) Test your Ansible Playbooks with Molecule
7) Templates with cookiecutter
8) Orchestrate Workflows with Ansible Tower
99) Resources, Questions and Answers
2
3. Introduction
Mihai Criveti, CTO Cloud Native and Red Hat Solutions at IBM
• Cloud Native & Red Hat Solutions Leader
• Red Hat Certified Architect
• Builds multi-cloud environments for large customers
Base OS Image Automation journey
• Started building OS images using automation in 2010, as Architect for IBM’s first Red Hat based cloud.
• Used a horrible combination of Perl and Jenkins for image build process.
• Had to develop base OS images, as well as various middleware images.
• Later on, worked on VMware image builds and had to re-write the pipeline.
• Found the combination of Ansible, Tower and Packer to be an amazing combination to build image
workflow.
3
4. Example Workflow: Build, Secure and Test Images for Multiple Environments
Example automated, layered image build workflow
1. Store all image build scripts, code and artifacts in Git.
2. Code changes or security updates trigger Tower or Jenkins to start a new image build.
3. Ansible will provision and setup a bare metal build node.
4. Parallel builds generate base OS images (RHEL 7, 8, Windows, Fedora, etc) when needed (Packer) - for
KVM, VMware, VirtualBox various cloud providers.
5. Secondary builds based on the base image now trigger (OpenSCAP profiles).
6. Tertiary builds now trigger to install a variety of middleware.
7. Images are packed, signed and uploaded to an image store.
8. Images are provisioned and tested.
9. Dismantle the build infrastructure.
4
6. What is OpenShift Virtualization
OpenShift Virtualization
• A feature of the OpenShift container platform, based on the open-source KubeVirt project.
• Manage Virtual Machines into containerized workloads.
• Develop, manage and deploy VMs side-by-side with containers and serverless on one platform: OpenShift.
What is KubeVirt?
• Open source project that makes it possible to run virtual machines in a Kubernetes-managed container
platform.
• Delivers container-native virtualization by leveraging KVM, the Linux Kernel hypervisor, within a Kubernetes
container.
• Provides services like those associated with traditional virtualization platforms, providing the best of both
mature virtualization management technology and Kubernetes container orchestration.
5
7. How are OpenShift Virtualization images built
What images does OpenShift Virtualization support?
• OpenShift images are the same kind of images used by KVM / qemu / qcow2 - and are build in a similar way
to OpenStack or RHV images.
• Can also import RHV / VMware images, will use qemu-img convert behind the scenes to convert them to
RAW and prepare them for OpenShift Virtualization.
Image Format
• Only RAW and QCOW2 formats are supported disk types for the container image registry. QCOW2 is
recommended for reduced image size.
• virt-sparsify to help ‘compress’ the image (sparsify).
• Images can be compressed with xz or gzip.
Additional image requirements
• The QEMU guest agent is a daemon that runs on the virtual machine and passes information to the host
about the virtual machine, users, file systems, and secondary networks.
• VirtIO drivers are paravirtualized device drivers required for Microsoft Windows virtual machines to run in
OpenShift Virtualization.
• cloud-init lets OpenShift customize the image after provisioning - such as setting the Hostname,
authorized SSH keys or running a custom script.
• Security patches and your configuration.
6
8. Install OpenShift Virtualization using an Operator
Install the OpenShift Virtualization
Operator and create a cluster
OpenShift Virtualization can be installed on
any OpenShift cluster, using the Operator
Hub.
• Install the OpenShift Virtualization
Operator (2.4 or higher)
• Create a OpenShift Virtualization
Operator Deployment:
kubevirt-hyperconverged cluster on
the Bare Metal node(s).
Create a new project namespace
oc new-project virtual-machines
You can now create Virtual Machines by
clicking on Workloads > Virtualization.
OpenShift Operator Hub
Figure 1: OpenShift Virtualization Operator
7
9. Virtual Machine Types
Ephemeral
• When using a ephemeral storage volume type, or containerDisk.
• The ephemeral image is created when the virtual machine starts and stores all writes locally. The
ephemeral image is discarded when the virtual machine is stopped, restarted, or deleted. The backing
volume (PVC) is not mutated in any way.
Persistent (persistentVolumeClaim)
• When using a persistentVolumeClaim
• Attaches an available PV to a virtual machine. Attaching a PV allows for the virtual machine data to persist
between sessions.
• Importing an existing virtual machine disk into a PVC by using CDI and attaching the PVC to a virtual
machine instance is the recommended method for importing existing virtual machines into OpenShift
Container Platform.
Note CDI: Containerized Data Importer.
8
10. CDI: Containerized Data Importer Overview
CDI Function
• persistent storage management add-on for Kubernetes. Uses qemu-img to manipulate disk images.
• provides a declarative way to build Virtual Machine Disks on PVCs for Kubevirt VMs.
• provides a way to populate PVCs with VM images or other data upon creation.
• data can come from different sources: a URL, a container registry, another PVC (clone), or an upload from a
client.
• CDI will automatically decompress and convert the file from qcow2 to raw format if needed. It will also
resize the disk to use all available space.
Import from URL
This method is selected when you create a DataVolume with an http source. Supports basic authentication
(secret) and custom TLS certificates (ConfigMap).
Import from container registry
When a DataVolume has a registry source CDI will populate the volume with a Container Disk downloaded
from the given image URL.
9
12. 2) Build KVM images by hand -
what are we automating?
13. Architectural Decisions and Image Build considerations
AD001: Build from scratch (kickstart / sysprep) every time?
• Do we want to rebuild the golden image every time, or do we create a major release build?
• Post-install steps are based on existing golden image (major release).
• Ansible used for post-install steps and parametrization (update the image, secure it, perform various
post-install steps).
• Additional ‘layers’ for middleware or product installs.
AD002: Clean builds or image conversion?
• When building multi-platform images (ex: VMware, KVM, OpenStack) - do we convert images, or use the
right hypervisor for every build?
AD003: What goes into post-provisioning scripts (ex: cloud-init)
• How much do we want to delegate to cloud-init?
• For example, do we want to bake in the security, patches and compliance in the golden image, or trigger it
post provisioning?
• Triggering it after provisioning can be slow, consume bandwidth and expose the image during the update
process.
11
14. Setting up a bare metal build host
Provision a BM host and install KVM, qemu-img and various build tools
• Consider the number of builds running in parallel when sizing the host.
• Bare Metal is recommended for performance considerations.
• Consider using solid state storage for the build host.
sudo yum install @virt virt-top libguestfs-tools
virt-manager virt-install virt-viewer qemu-img
sudo systemctl enable --now libvirtd
Install virtctl client on RHEL 7
sudo subscription-manager repos --enable rhel-7-server-cnv-2.4-rpms
sudo yum -y install kubevirt-virtctl
Install the virtctl client on RHEL 8
sudo subscription-manager repos --enable cnv-2.4-for-rhel-8-x86_64-rpms
sudo dnf -y install kubevirt-virtctl
12
15. Building a Red Hat OS Image with Kickstart
Create a Kickstart file (response file)
• You can manually install the OS to generate a .ks file.
• Installations from kickstart are automated, and you could use this as part of a CI/CD OS build.
Kickstart install the OS
virt-install
--name guest1-rhel7
--memory 2048 --vcpus 2 --disk size=8
--location http://example.com/path/to/os
--os-variant rhel7
--initrd-inject /path/to/ks.cfg
--extra-args="ks=file:/ks.cfg console=tty0 console=ttyS0,115200n8"
Setup QEMU guest agent on virtual machines
systemctl enable qemu-guest-agent
13
16. Building a Windows Image from ISO
Download container-native-virtualization/virtio-win - Red Hat Container Catalog.
This contains the VirtIO drivers for Windows.
podman login registry.redhat.io
podman pull registry.redhat.io/container-native-virtualization/virtio-win
Create a image disk at least 15GB in size
qemu-img create -f qcow2 w2016.qcow2 15G
Install Windows using virt-install
virt-install --connect qemu:///system
--name ws2016 --ram 4096 --vcpus 2
--network network=default,model=virtio
--disk path=ws2016.qcow2,format=qcow2,device=disk,bus=sata
--cdrom Windows_Server.ISO --disk path=virtio-win-0.1.189.iso,device=cdrom
--vnc --os-type windows --os-variant win2k16
Post-install steps
• Microsoft update, likely a few reboots required.
• Install QEMU guest agent and VirtIO Drivers.
• Configure RDP access, install cloud init and sysprep the image.
14
17. Processing and converting images
Sparsify then compress the image
virt-sparsify --in-place rhel7.qcow2
qemu-img convert -O qcow2 -c w2016.qcow2 windows2016.qcow2
qemu-img convert -O qcow2 -c r7.qcow2 rhel7.qcow2
Create a SHA256 for your images
This is optional, but good practice when uploading your images to a webserver, etc.
sha256sum *qcow2 > SHA256SUMS
Optionally, sign your image with GPG.
gpg --sign myfile
15
18. Creating and pushing images to the container registry
Create a Dockerfile
FROM scratch
ADD windows2016.qcow2 /disk/
Create a container
podman build -t cmihai/windows2016 .
Login to the container registry
REGISTRY="$(oc get route/default-route
-n openshift-image-registry -o=jsonpath='{.spec.host}')"
podman login ${REGISTRY}
Tag and push the image to your desired namespace (ex: virtual-machines)
podman tag localhost/virtual-machine/windows2016
${REGISTRY}/virtual-machines/windows2016
podman push ${REGISTRY}/virtual-machines/windows2016
16
19. Creating container images with Buildah
Create a Dockerfile in /tmp/vmdisk
cat << END > Dockerfile
FROM kubevirt/container-disk-v1alpha
ADD fedora34.qcow2 /disk
END
Build and push to registry
buildah bud -t vmidisk/fedora34:latest /tmp/vmdisk
buildah push --tls-verify=false
vmidisk/fedora34:latest
docker://cdi-docker-registry-host.cdi/fedora34:latest
17
20. Import the registry image into a Data volume
YAML
apiVersion: cdi.kubevirt.io/v1alpha1
kind: DataVolume
metadata:
name: fedora34image
spec:
source:
registry:
url: "docker://image-registry.openshift-image-registry.svc:5000/
virtual-machines/fedora34"
pvc:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 20Gi
Get the image info
oc apply -f datavolume.yaml
oc get pvc, dvs, pods # look for importer-fedora34image
18
21. Uploading local disk images by using the virtctl tool
Creating an upload DataVolume YAML
apiVersion: cdi.kubevirt.io/v1alpha1
kind: DataVolume
metadata:
name: <upload-datavolume>
spec:
source:
upload: {}
pvc:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: <2Gi>
Create the volume
oc create -f <upload-datavolume>.yaml
Upload the image
virtctl image-upload dv <volume_name>
--size=2G
--image-path=</path/to/image>
Verify that a DataVolume was created
View all DataVolume objects
oc get dvs
19
22. Cleanup and deprovisioning
• To reduce costs, you can de-provision the build host after running a build.
• There may be trade-offs here: running costs with elastic provisioning vs. reserved instances.
ibmcloud sl hardware cancel IDENTIFIER --immediate
20
24. Automation process overview
Considerations
• Do we need to have this always available?
• Create, build and tear down the build host on
demand - will also reduce costs.
• Are we build images for other hypervisors? For
example, we can install VMware workstation and
build VMware images as well.
Build Process Overview
Figure 3: Automation Process Overview
21
25. Setting up the build host on demand with Ansible
Using an Ansible collection to install Virtualization Tools
• Created a Ansible collection for various virtualization tools (KVM, VMware, VirtualBox, etc).
• Collections make maintaining, publishing and testing roles easier.
• A simple example on how such a role can be used is listed below.
• This will install KVM, VMware, Packer and Vagrant on our build host.
Install the Ansible collection
pip install --upgrade ansible
ansible-galaxy collection install
crivetimihai.virtualization
Run the playbook
ansible-playbook -i localhost, playbook.yml
-e "vmware_workstation_license_key='...'"
-e "ansible_python_interpreter=/usr/bin/python3"
playbook.yaml
- name: setup a virtualization environment
hosts: all
connection: local
become: yes
gather_facts: yes
roles:
- role: crivetimihai.virtualization.kvm
- role: crivetimihai.virtualization.vmware
- role: crivetimihai.virtualization.packer
- role: crivetimihai.virtualization.vagrant
22
40. Ansible Molecule
Creating a vagrant or docker machine and trigger goss tests:
molecule create -s vagrant-rhel-8
molecule converge -s vagrant-rhel-8
molecule login
In one step
molecule test
Another OS:
molecule create -s docker-rhel-7
34
43. Molecule Cookie Cutter Templates
Cookiecutter: Better Project Templates
• Cookiecutter creates projects from project templates, e.g. Ansible role structure, with molecule tests.
• Molecule provides a native cookiecutter interface, so developers can provide their own templates.
Create a new role from a template, with molecule tests included
molecule init template
--url https://github.com/crivetimihai/ansible_cookiecutter.git
--role-name httpd
36