Monitoring the ELK stack using Zabbix and Grafana (Dennis Kanbier / 26-11-2015)

Xifeo ICT | Dennis Kanbier
Monitoring the ELK stack
using Zabbix (and
Grafana)
Dennis Kanbier
26 Nov 2015 - NLZGG

Overview
• Introduction
• What is ELK and why do I want it?
• How to monitor ELK with Zabbix
• Impress people with pretty graphs
• Demo time ( please tweet using #nlzgg! )

Who am I?
• Dennis Kanbier
• dennis.kanbier@xifeo.nl
• Linux Consultant with Xifeo ICT
• “dkanbier” on Zabbix forums and IRC

What is the ELK stack
• Elasticsearch
• Search server
• Based on Apache Lucene
• Logstash
• Data pipeline
• Processes logs and other data
• Plugins
• Kibana
• Web frontend for Elasticsearch

Basic Logstash Pipeline

Apache log example
• 83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/
logstash-monitorama-2013/images/ahiruyaki.png HTTP/1.1" 200 203023
"http://semicomplete.com/presentations/logstash-monitorama-2013/"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36”
• A few interesting ﬁelds:
• client_ip: 83.149.9.216
• status code: 200
• timestamp: 04/Jan/2015:05:13:42 +0000
• and much more

{
"clientip" : "83.149.9.216",
"ident" : ,
"auth" : ,
"timestamp" : "04/Jan/2015:05:13:42 +0000",
"verb" : "GET",
"request" : “…/a-2013/images/ahiruyaki.png”,
"httpversion" : "HTTP/1.1",
"response" : "200",
"bytes" : "203023",
"referrer" : “http://semicomplete.com …“,
"agent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X…”
}
Logstash processes the log message and produces output:

Now what?
• Use the processed data to do fun things like:
• plot all entries using the [timestamp] field to see how
busy the web server is during the day
• which part of your website gets requested the most
using the [request] field
• monitor the status of your web server using the
[response] field
• Or do even more fun things using plugins…

logstash-ﬁlter-geoip
• Use the geo-ip ﬁlter plugin while logstash is processing the
message
• Looks up the geographical location of an IP address and adds
this information to the message
• Use Kibana to create a heat map based on the geographical
information in each message to see from which part in the world
your website gets visited
• Very nice to have in case of DDOS attacks originating from
places you don’t expect visitors from
• Or to determine where to up your advertising game

Example heat map

Developer issues

Not always the developers fault

Why developers love ELK
• Logs from all components and servers are available in one central place, easily searchable using
the Kibana web interface
• No more begging the scary Unix administrator for log files
• Log message are enriched with meaningful fields which enables users to:
• query on stack traces
• query on specific thread id’s
• query on log_level
• query on customer_id
• list faults per server/application/cluster/log_type/etc..
• You can easily plot logs, see how fast requests are handled
• Timestamps are normalised, even log entries from different time zones are searchable
chronologically

Example Kibana screen

Monitoring Logstash with
Zabbix
• Used 3 components to generate data and send it
to the Zabbix server:
• logstash-input-heartbeat
• logstash-ﬁlter-metrics
• logstash-output-zabbix

logstash-input-heartbeat
• Plugin to generate a heartbeat message at a
configurable interval
• The message contains a field called “clock” with the
epoch time of the moment it was send
• Assign a type of “heartbeat” to the message so we can
use that to filter out the message further in the pipeline
and send it to the Zabbix server

logstash.conf
input {
heartbeat {
message => "epoch"
interval => 20
type => "heartbeat"
}
}
filter {
}
output {
}

logstash-output-zabbix
• Used to send data to Zabbix
• Requires a configured Zabbix host and trapper
item to receive the data
• These fields must be added to the [@metadata]
field of the message you want to send to the
Zabbix server

logstash.conf
input {
heartbeat {
message => "epoch"
interval => 20
type => "heartbeat"
}
}
filter {
if [type] == "heartbeat" {
mutate {
add_field => { "[@metadata][zabbix_key]" => "logstash_heartbeat" }
add_field => { "[@metadata][zabbix_host]" => "logstash-indexer-1" }
}
}
}
output {
zabbix {
zabbix_server_host => “zabbix-server.example.com”
zabbix_host => “[@metadata][zabbix_host]”
zabbix_key => “[@metadata][zabbix_key]"
zabbix_value => clock
}

Results in Zabbix
• Zabbix host logstash-indexer-1 should already have the item
with key “logstash_heartbeat” conﬁgured as a Zabbix
trapper item.
• Logstash now sends the epoch value every 20 seconds to
the logstash_heartbeat trapper item
• You can use this to further process the data in Zabbix:
• trigger when there is too much time between epoch values
• trigger when you do not receive any values for a period of
time

Example item in Zabbix

logstash-filter-metrics
• Plugin to count messages and flush the result on a
specific interval
• Used to monitor logstash performance by counting
messages and store their count value in Zabbix as
Delta (speed per second)
• which gives us the messages this logstash
server handles per second

logstash.conf
input {
…
}
filter {
metrics {
meter => "events"
add_tag => "metric"
add_field => { "[@metadata][zabbix_key]" => "logstash_events" }
add_field => { "[@metadata][zabbix_host]" => “logstash-indexer-1“ }
flush_interval => 20
}
}
output {
if "metric" in [tags] {
zabbix {
zabbix_server_host => “zabbix-server.example.com”
zabbix_host => “[@metadata][zabbix_host]”
zabbix_key => “[@metadata][zabbix_key]"
zabbix_value => “[events][count]”
}
}
}

More results!
• We now have the item “logstash_events” which
shows how many messages we process per
second:

Now about that last slide….
• Presenting reports and graphs are not exactly
Zabbix’ strongest points
• Works all right to view single values like on the
previous slide
• But try to align multiple graphs nicely on one
screen to put on a wall and impress people

Grafana
• Build to present metric data in a simple and nice
manner
• Supports multiple data sources like Graphite and
InﬂuxdB
• Alexander Zobnin created grafana-zabbix, which
enables us to use the Zabbix database as a
Grafana backend through the Zabbix API
• https://github.com/alexanderzobnin/grafana-zabbix

Example dashboard

Things to consider
• Grafana queries the Zabbix database through the
API
• If you have a lot of graphs with a lot of items
these queries are quite expensive
• Especially if you have multiple dashboards
refreshing automatically on a high interval
• Still learning on how to best counter these
challenges

Live demo time!
Did you remember to tweet things with hashtag #nlzgg?

Questions?

Contact details
• dkanbier on Freenode IRC
• https://github.com/dkanbier
• http://denniskanbier.nl/blog for tutorials
• dennis.kanbier@xifeo.nl
Resources
• http://www.zabbix.com
• https://github.com/alexanderzobnin/grafana-zabbix
• https://www.elastic.co
• http://www.xifeo.nl

Monitoring the ELK stack using Zabbix and Grafana (Dennis Kanbier / 26-11-2015)

Related slideshows

More Related Content

Monitoring the ELK stack using Zabbix and Grafana (Dennis Kanbier / 26-11-2015)