How to Monitor Microservices

•

4 likes•3,466 views

How to Monitor Microservices running in Docker, Kubernetes, Openshift, DC/OS Mesos or any other container orchestration platform.

Jorge Salamero Sanz
@bencerillo @sysdig
How to
Monitor Microservices

How to Monitor Microservices?
Apps
Infra
Health
Checks
JVM/JMX
Custom
metrics
Metrics Processing Unicorns, rainbows
And cute dashboards

% whoami
Jorge Salamero Sanz
<jorge.salamero@sysdig.com>
• Working on OSS last 12 years
• Working on monitoring last 3 years
• Containers gamer @sysdig
@bencerillo
@sysdig

Agenda
• Challenges of container infrastructures
• Traditional monitoring limitations
• Best practices monitoring Microservices
• Sysdig, container native monitoring & troubleshooting

Breaking traditional model
Microservices and containers break traditional monitoring and
troubleshooting models

Traditional deployment
Full host OS
kernel
systemd
syslogd
App services
MySQL
Nginx
OpenSSL
Java
App A
App B
App C
Ops Devs

Application workflow
Database App Cache
backend middleware frontend

The Docker revolution
Servers Virtual Machines Containers
Unit: machine
Orchestration: no
Architecture:
monolithic
Unit: machine
Orchestration:
external
Architecture:
monolithic
Unit: (micro)service
Orchestration:
native
Architecture:
distributed

Containerized deployment
Full host OS
kernel
+
Docker
MySQL App A
Ops DevOps
Nginx + OpenSSL App B
Java 8.0 build XXX App C

… but in reality
Database App Cache/Frontend
Computing node
Computing node Computing node
Computing node Computing node
Computing node

Container monitoring
New challenges:
1. How do we get the metrics?
2. How do we shape all this amount of metrics?
3. Analysis and troubleshooting
4. Teams on Microservices infrastructure

1. Metric collection
• We containers, because:
– are simple
– are small
– are isolated
– less dependencies
• … but they are an opaque blackbox

“Workarounds”
Agent in the
Docker container
Agent in the
Kubernetes pod
Export metrics through
an external agent
App Agent App Agent
App
Agent
App
App
App
1. Complex instrumentation (x2 because just the monitoring) plus
service monitoring configuration
2. Limited and pre-established metric collection (Docker API, etc)

Kernel instrumentation
Kernel
Docker
Container
1
Container
2
Container
3
App App
rkt LXC
Sysdig
Docker

Why this is cool?
• Just one instrumentation per host:
– spawning or destroying a container is instrumentation-less
• Full visibility: all the system calls:
– automatic service discovery
– all metrics collection (no filtering)
– application monitoring without instrumentation (magic of
decoding protocols)

Remember... but in reality:
Database App Cache/Frontend
Computing node
Computing node Computing node
Computing node Computing node
Computing node

2. Information aggregation
• Infrastructure monitoring should be transparent and
automatic (no instrumentation no configuration)
• You should handle your custom/biz metrics
• All metrics should be tagged automatically
• All metrics should be aggregated and segmented on a
service level basis

Orchestration platforms knows already...

Real example
https://github.com/kubernetes/kubernetes/issues/1405
1

3. Analysis & troubleshooting
• Imagine:
strace + wireshark + htop + lsof + iostat + vmstat + *
• Not available on containers, don’t understand
namespaces
• Metrics and logs can bite your in the ass, system
calls have all the truth
• Infrastructure gets more complex and volatile

Teams on Microservices
Francesc Zacarias, SRE @ Spotify

4. Teams by service
• Tags/Metadata from the orchestration platform, eg
Kubernetes:
– namespaces (dev, prod)
– services, deployments, RCs, pods
– custom tags
• ACLs out of the box (dashboards, alerts, etc) on
multi-tenant/PaaS scenarios

Sysdig
• 100% open-source
• 1M+ downloads
• Host analysis
• sysdig.org
• SaaS & on-prem
• 200+ customers
• Cluster analysis
• Dashboards, alerts,
events, teams
• sysdig.com

¡Danke
MicroXchg!
@bencerillo
@sysdig
www.sysdig.org
www.sysdig.com

What's hot

Automated hardware testing using docker for space

Docker, Inc.

Two things are for certain – space is hard, and Docker is not just for web content! Space software development traditionally lags behind state of the art software process for good reason – our missions are long (7+ years), we run on highly constrained embedded hardware, and the software cannot fail. Docker, along with a devops mentality, has helped us create a scalable, parallelizable and rapidly deployable test infrastructure for DART, NASA’s mission to hit an asteroid at 6 km/s. During the presentation, we will walk through how our dev cycle has changed from a human based testing system to an automated one. We will outline how we are using Docker (and NASA Goddard’s Core Flight Executive) for both our embedded development environment and our scalable test environment. Next, we will discuss what deployment means to us (and how different it is from web deployment). Lastly, we will explore lessons learned on how our hardware-centric testing approach was adapted into a software-based approach: what worked, what didn’t, what we wish we could do someday. How can you help? We are new to Docker. We are excited to share our experiences and hear from the Docker community on our use cases, technological hurdles that we faced, our solutions to these problems, and how we can harness Docker to the fullest extent.

Configuration Management and Transforming Legacy Applications in the Enterpri...

Docker, Inc.

Share the continuity of Société Générale's journey with Docker Enterprise from different points of view, from executives to devops, with CD platform as an enabler. Creating a Dockerfile that runs a container on a developer's laptop is pretty straightforward. But extending that to stacks of containers running on a dozen environments (development, integration, testing, staging, production, etc.) with different configuration and topologies can be a challenge. This talk will cover aspects of our journey to Docker Enterprise: What configuration should go in an image? Where to put different types of configuration? Images, environment variables, entrypoint, ...? How to store assets for building images and configuration for deployment in version control. We will discuss how Société Générale has implemented these, and what we plan next for Docker Enterprise deployment.

ContainerCon sysdig Slides

Loris Degioanni

This document discusses container monitoring and provides an overview of options for monitoring containers including command line tools, cAdvisor, Docker stats, and sysdig. It outlines some key metrics that are important to monitor like resource usage, network activity, file I/O, errors, application activity, and topology. It then dives deeper into explaining how to use tools like cAdvisor, Docker stats, and sysdig to monitor these metrics and view specific activities within containers. The presenter argues that sysdig provides a more robust monitoring solution that can see inside containers while respecting their isolated and lightweight nature.

Kubernetes 架構與虛擬化之差異

inwin stack

Kubernetes uses containers managed by container engines like Docker. It separates containers from the host machine using namespaces and cgroups for isolation. Docker containers share the host kernel and use aufs for the union filesystem. Virtual machines (VMs) run a full guest operating system with virtualization provided by hypervisors like KVM/QEMU. Containers are more lightweight than VMs as they share the host kernel and have smaller base images and faster launch times and resource usage.

Effective Data Pipelines with Docker & Jenkins - Brian Donaldson

Docker, Inc.

Ever find yourself needing data pipelines to feed a hungry data-driven culture, but not sure where to start, or what features are essential? In this talk, I will demonstrate a baseline data pipeline infrastructure built with Jenkins and Docker EE that checks all the boxes. Data pipelines often exist as that mysterious plumbing buried underground: occasionally inspected, but largely prone to silent failures and the ensuing hot fixes. Join the quest to daylight the infrastructure and benefit!

Chaos Engineering for Docker

Alexei Ledenev

Experiences with AWS immutable deploys and job processing

Docker, Inc.

How Docker is used at Gilt: At Gilt we use Docker primarily as a unit of immutability and to allow a standard way of deploying all kinds of software as opposed to its container properties. Why Gilt built Ionroller: An overview of the problems we tried to solve with Ionroller and immutable deploys. Pitfalls we've encountered with immutable deployments since Ionroller saw adoption in Gilt. Will cover issues such as DNS traffic migration, utilisation of resources ELBs not warmed up properly, Elasticbeanstalk using Nginx as proxy etc. Our experiences with Cloudformation and Codedeploy as an alternative to Ionroller and Elasticbeanstalk. Jobs: How we used to do batch jobs. Solutions we considered such as Mesos and Chronos. An overview of Sundial, an in house solution we built in the last few months and hope to open source for running containerized Docker jobs on Amazon ECS and why we chose it as our preferred solution.

containerd the universal container runtime

Docker, Inc.

containerd is a widely used container runtime that is now a CNCF project. It is designed to be embedded in larger systems rather than used directly by developers. containerd provides core primitives for managing containers on a host, such as container execution, image distribution, and storage. It focuses on simplicity, robustness, and portability. containerd will serve as a core container runtime for the CNCF ecosystem and is being integrated with projects like Kubernetes.

Introducing LinuxKit

Docker, Inc.

LinuxKit is an open source project that provides the components to build secure, portable, and lean container-based operating systems. It produces minimal and immutable images defined through a YAML configuration file. The project emphasizes security through limiting components, modern kernels, and container isolation with minimal privileges. It is maintained by Anthropic and supported on various platforms through integration with tools like Docker, Kubernetes, and InfraKit for managing clusters. Future work includes porting to more platforms and architectures while improving security and reliability.

Containerd Donation to CNCF Cloud Native Conference Berlin 2017

Patrick Chanezon

Docker has donated containerd, a core container runtime built by Docker with input from major cloud providers, to the Cloud Native Computing Foundation (CNCF). Containerd provides primitives for managing containers and works with projects like Docker Engine, Kubernetes, and CNCF projects. It has over 100 commits per month from many contributors and maintainers and is already widely used through Docker with millions of installs across industries.

DockerCon EU 2015: Cultural Revolution - How to Mange the Change Docker Brings

Docker, Inc.

The document discusses how organizations can manage the cultural changes that adopting Docker brings. It outlines four choices organizations face: whether to adopt Docker as a skunkworks project or enterprise-wide, whether to containerize monolithic or microservice applications, whether to use standard tools or develop proprietary solutions, and whether to adopt Docker openly or secretly. The document argues that adopting Docker openly as an enterprise standard can facilitate a DevOps culture, improve software quality, and increase hardware efficiency, ultimately benefiting the organization.

Deploying Kubernetes without scaring off your security team - KubeCon 2017

Major Hayden

From Code to Kubernetes

Daniel Oliveira Filho

Slides from the talk given to the Startup Berlin Slack Group that demonstrates how TruckIN is implementing its continuous delivery workflow using technologies and open-source tools. Topics that are covered: Automated Cloud Provisioning (Network, Subnets, VMs, Kubernetes Cluster, Firewall, Disks, Credentials, Private Docker Registry); Configuration Management (Salt Stack), Continuous Integration (Jenkins CI), Continuous Delivery/Deployment (Salt API/Reactor + Kubernetes) to a Google Cloud Kubernetes Cluster, Remote Application Debugging, Managing Google Cloud Kubernetes Cluster, Logging, Monitoring and ChatOps (Slack and operable.io)

DockerCon 2017 - Cilium - Network and Application Security with BPF and XDP

Thomas Graf

This talk will start with a deep dive and hands on examples of BPF, possibly the most promising low level technology to address challenges in application and network security, tracing, and visibility. We will discuss how BPF evolved from a simple bytecode language to filter raw sockets for tcpdump to the a JITable virtual machine capable of universally extending and instrumenting both the Linux kernel and user space applications. The introduction is followed by a concrete example of how the Cilium open source project applies BPF to solve networking, security, and load balancing for highly distributed applications. We will discuss and demonstrate how Cilium with the help of BPF can be combined with distributed system orchestration such as Docker to simplify security, operations, and troubleshooting of distributed applications.

Troubleshooting Tips from a Docker Support Engineer

Jeff Anderson

The document discusses various troubleshooting techniques for Docker including using tools like socat and curl to characterize networking and TLS issues, checking container processes and permissions, using volumes to store persistent data, and resolving issues with incorrect localhost references between containers. It also provides examples of troubleshooting issues with a Minecraft server, Ruby application, and Nginx proxy configuration.

What Have Namespaces Done for you Lately? Liz Rice, Aqua Security

Docker, Inc.

Kubernetes security

Saiyam Pathak

This document provides an overview of Kubernetes security concepts including the Kubernetes attack surface, TLS certificates, securing the Kubelet and etcd, authentication, authorization, admission controllers, tooling landscape, pod security policies, network policies, and secrets. It discusses key configuration recommendations and checks for securing different components like disabling privileged mode, setting TLS certificates, CIS benchmarks for etcd, and authentication methods in Kubernetes. The different types of admission controllers and CNCF security tooling are also briefly introduced.

Docker for Ops - Scott Coulton, Puppet

Docker, Inc.

In this talk, Scott Coulton will take you through Docker's cluster solution Swarm mode with his operations hat on. We will start from the beginning by describing what swarm mode is, what it does, and how it works behind the scenes. From there, we will look at very basic configurations of Swarm mode from the point of view of the operations team as well as a production-ready workflow including deployments of the cluster, logging and CD best practices. Attendees will be able to apply their learnings to their use cases.

Continuous Delivery With Containers

All Things Open

Building Containers: How Many Ways Are Too Many?

Vietnam Open Infrastructure User Group

This document discusses container build tools and summarizes some of their key features. It notes that while Docker works well, it has some inefficiencies like lack of concurrency support and inefficient caching. Alternatives discussed include BuildKit, Kaniko, img, and Buildah. The document recommends BuildKit as the most promising option as it uses a DAG-style language called LLB for more efficient concurrency and caching support compared to Dockerfiles, and its integration with Docker.

What's hot (20)

Automated hardware testing using docker for space

Configuration Management and Transforming Legacy Applications in the Enterpri...

ContainerCon sysdig Slides

Kubernetes 架構與虛擬化之差異

Effective Data Pipelines with Docker & Jenkins - Brian Donaldson

Chaos Engineering for Docker

Experiences with AWS immutable deploys and job processing

containerd the universal container runtime

Introducing LinuxKit

Containerd Donation to CNCF Cloud Native Conference Berlin 2017

DockerCon EU 2015: Cultural Revolution - How to Mange the Change Docker Brings

Deploying Kubernetes without scaring off your security team - KubeCon 2017

From Code to Kubernetes

DockerCon 2017 - Cilium - Network and Application Security with BPF and XDP

Troubleshooting Tips from a Docker Support Engineer

What Have Namespaces Done for you Lately? Liz Rice, Aqua Security

Kubernetes security

Docker for Ops - Scott Coulton, Puppet

Continuous Delivery With Containers

Building Containers: How Many Ways Are Too Many?

Similar to How to Monitor Microservices

Monitoring microservices: Docker, Mesos and Kubernetes visibility at scale

Alessandro Gallotta

This document discusses monitoring microservices running in containers at scale. It introduces Sysdig, an open source tool that captures system events and provides visibility into containers, processes, files, and more through a curses UI. Sysdig addresses the challenges of getting monitoring data out of orchestrated applications running in containers across nodes and making sense of that data. It collects kernel-level data without adding dependencies to containers. Sysdig Cloud provides correlated monitoring data at scale. The document also introduces Falco, an anomaly detection system built on Sysdig that monitors for security threats through behavior rules.

ThroughTheLookingGlass_EffectiveObservability.pptx

Grace Jansen

Our cloud-native environments are more complex than ever before! So how can we ensure that the applications we’re deploying to them are behaving as we intended them to? This is where effective observability is crucial. It enables us to monitor our applications in real-time and analyse and diagnose their behaviour in the cloud. However, until recently, we were lacking the standardization to ensure our observability solutions were applicable across different platforms and technologies. In this session, we’ll delve into what effective observability really means, exploring open source technologies and specifications, like OpenTelemetry, that can help us to achieve this while ensuring our applications remain flexible and portable.

DockerCon EU 2015: Monitoring Docker

Docker, Inc.

This document discusses monitoring Docker containers. It provides an overview of Datadog and how it can be used to monitor dynamic infrastructure like Docker. It then discusses the challenges in monitoring Docker environments due to their dynamic and distributed nature. Finally, it outlines different methods for collecting key metrics on Docker containers like CPU, memory, I/O and network metrics using Docker's pseudo-files, stats command and API.

Operational Visibiliy and Analytics - BU Seminar

Canturk Isci

The document discusses building operational visibility and analytics directly into cloud platforms. It describes an agentless system crawler that can provide deep visibility into cloud instances without requiring any action from end users. The crawler collects various system data which is then analyzed to provide operational insights and solve real-world problems. Specific applications discussed include vulnerability advising, configuration analysis, and license discovery. The goal is to design monitoring and analytics that are seamlessly integrated and optimized for cloud environments.

Big Brother: Kubernetes Edition

Knox Anderson

In this short demo-driven meetup, we'll help you get a handle on what's changing and how it will impact your DevOps practice. We'll cover: - What are the operational limitations of containers in production? - How do you get visibility inside containers without super-human effort? - How do you look into kubernetes performance, and not just container performance? - A live install of Sysdig Cloud on a running environment

Weave User Group Talk - DockerCon 2017 Recap

Patrick Chanezon

Docker moves very fast, with an edge channel released every month and a stable release every 3 months. Patrick will talk about how Docker introduced Docker EE and a certification program for containers and plugins with Docker CE and EE 17.03 (from March), the announcements from DockerCon (April), and the many new features planned for Docker CE 17.05 in May. This talk will be about what's new in Docker and what's next on the roadmap

The Art of Container Monitoring

Derek Chen

According to service scale, there are hundreds or thousands of running containers in your service. Should we monitor each container by microscope or monitor each microservice by magnifier? This depends which granularity can help us find and solve the problems. In this sharing, I will introduce how to use cAdvisor, Icinga2, InfluxDB and Grafana to build a self-hosted monitoring system. In addition, I also discuss with how to embrace open source and share some practical experiences.

Kaseya Connect 2012 - THE ABC'S OF MONITORING

Kaseya

Integration in the Cloud, by Rob Davies

Judy Breedlove

Kubernetes Monitoring & Best Practices

Ajeet Singh Raina

DCSF19 Container Security: Theory & Practice at Netflix

Docker, Inc.

Michael Wardrop, Netflix Usage of containers has undergone rapid growth at Netflix and it is still accelerating. Our container story started organically with developers downloading Docker and using it to improve their developer experience. The first production workloads were simple batch jobs, pioneering micro-services followed, then status as a first class platform running critical workloads. As the types of workloads changed and their importance increased, the security of our container ecosystem needed to evolve and adapt. This session will cover some security theory, architecture, along with practical considerations, and lessons we learnt along the way.

Docker Usage Patterns - Meetup Docker Paris - November, 10th 2015

Datadog

Monitoring Containerized Micro-Services In Azure

Alex Bulankou

This document discusses best practices for monitoring containerized microservices applications in Azure. It begins with an introduction to Application Insights and describes the agenda. It then discusses what is different about monitoring microservices compared to monolithic applications and some factors to consider when choosing a monitoring system. The document provides recommendations for setting up day-to-day monitoring operations, including maintaining a 15 minute daily triage process focusing on business metrics, application performance and health, and infrastructure and costs. It concludes with a demo of monitoring a sample microservices application using Application Insights and other tools.

Microservices Architecture - Cloud Native Apps

Araf Karsh Hamid

This document provides an overview of microservices architecture, including concepts, characteristics, infrastructure patterns, and software design patterns relevant to microservices. It discusses when microservices should be used versus monolithic architectures, considerations for sizing microservices, and examples of pioneers in microservices implementation like Netflix and Spotify. The document also covers domain-driven design concepts like bounded context that are useful for decomposing monolithic applications into microservices.

Containers as Infrastructure for New Gen Apps

Khalid Ahmed

Breaking the Monolith Road to Containers

Amazon Web Services

Following simple patterns of good application design can allow you to scale your application for your customers easily. This presentation dives into the 12 factor application design and demo how this applies to containers and deployments on Amazon ECS and Fargate. We'll take a look at tooling that can be used to simplify your workflow and help you adopt the principles of the 12 factor application.

Monitoring in Motion: Monitoring Containers and Amazon ECS

Amazon Web Services

Patterns and Pains of Migrating Legacy Applications to Kubernetes

Josef Adersberger

Running applications on Kubernetes can provide a lot of benefits: more dev speed, lower ops costs, and a higher elasticity & resiliency in production. Kubernetes is the place to be for cloud native apps. But what to do if you’ve no shiny new cloud native apps but a whole bunch of JEE legacy systems? No chance to leverage the advantages of Kubernetes? Yes you can! We’re facing the challenge of migrating hundreds of JEE legacy applications of a German blue chip company onto a Kubernetes cluster within one year. The talk will be about the lessons we've learned - the best practices and pitfalls we've discovered along our way.

Patterns and Pains of Migrating Legacy Applications to Kubernetes

QAware GmbH

Open Source Summit 2018, Vancouver (Canada): Talk by Josef Adersberger (@adersberger, CTO at QAware), Michael Frank (Software Architect at QAware) and Robert Bichler (IT Project Manager at Allianz Germany) Abstract: Running applications on Kubernetes can provide a lot of benefits: more dev speed, lower ops costs and a higher elasticity & resiliency in production. Kubernetes is the place to be for cloud-native apps. But what to do if you’ve no shiny new cloud-native apps but a whole bunch of JEE legacy systems? No chance to leverage the advantages of Kubernetes? Yes you can! We’re facing the challenge of migrating hundreds of JEE legacy applications of a German blue chip company onto a Kubernetes cluster within one year. The talk will be about the lessons we've learned - the best practices and pitfalls we've discovered along our way.

Kubernetes @ meetic

Sébastien Le Gall

3 years ago, Meetic chose to rebuild it's backend architecture using microservices and an event driven strategy. As we where moving along our old legacy application, testing features became gradually a pain, especially when those features rely on multiple changes across multiple components. Whatever the number of application you manage, unit testing is easy, as well as functional testing on a microservice. A good gherkin framework and a set of docker container can do the job. The real challenge is set in end-to-end testing even more when a feature can involve up to 60 different components. To solve that issue, Meetic is building a Kubernetes strategy around testing. To do such a thing we need to : - Be able to generate a docker container for each pull-request on any component of the stack - Be able to create a full testing environment in the simplest way - Be able to launch automated test on this newly created environment - Have a clean-up process to destroy testing environment after tests To separate the various testing environment, we chose to use Kubernetes Namespaces each containing a variant of the Meetic stack. But when it comes to Kubernetes, managing multiple namespaces can be hard. Yaml configuration files need to be shared in a way that each people / automated job can access to them and modify them without impacting others. This is typically why Meetic chose to develop it's own tool to manage namespace through a cli tool, or a REST API on which we can plug a friendly UI. In this talk we will tell you the story of our CI/CD evolution to satisfy the need to create a docker container for each new pull request. And we will show you how to make end-to-end testing easier using Blackbeard, the tool we developed to handle the need to manage namespaces inspired by Helm.

Similar to How to Monitor Microservices (20)

Monitoring microservices: Docker, Mesos and Kubernetes visibility at scale

ThroughTheLookingGlass_EffectiveObservability.pptx

DockerCon EU 2015: Monitoring Docker

Operational Visibiliy and Analytics - BU Seminar

Big Brother: Kubernetes Edition

Weave User Group Talk - DockerCon 2017 Recap

The Art of Container Monitoring

Kaseya Connect 2012 - THE ABC'S OF MONITORING

Integration in the Cloud, by Rob Davies

Kubernetes Monitoring & Best Practices

DCSF19 Container Security: Theory & Practice at Netflix

Docker Usage Patterns - Meetup Docker Paris - November, 10th 2015

Monitoring Containerized Micro-Services In Azure

Microservices Architecture - Cloud Native Apps

Containers as Infrastructure for New Gen Apps

Breaking the Monolith Road to Containers

Monitoring in Motion: Monitoring Containers and Amazon ECS

Patterns and Pains of Migrating Legacy Applications to Kubernetes

Kubernetes @ meetic

More from Sysdig

Wordpress y Docker, de desarrollo a produccion

How to Monitor Microservices

More Related Content

What's hot

What's hot (20)

Similar to How to Monitor Microservices

Similar to How to Monitor Microservices (20)

More from Sysdig

More from Sysdig (20)

Recently uploaded

Recently uploaded (20)

How to Monitor Microservices