Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo
hi.
hi.
dave@librato.com
@davejosephsen
github: djosephsen
hi.
dave@librato.com
@davejosephsen
github: djosephsen
Who put Stream Processing in my Monitoring
Conference?
Let’s talk about signal processing
RealTime Audio Processing
Stream Processing Inside Librato [Monitorama PDX 2015]
Stream Processing Inside Librato [Monitorama PDX 2015]
Stream Processing Inside Librato [Monitorama PDX 2015]
Stream Processing Inside Librato [Monitorama PDX 2015]
Stream Processing Inside Librato [Monitorama PDX 2015]
Stream Processing Inside Librato [Monitorama PDX 2015]
WhisperDB
WhisperDB
WhisperDB
WhisperDB
Stream Processing Inside Librato [Monitorama PDX 2015]
Stream Processing Inside Librato [Monitorama PDX 2015]
Stream Processing Inside Librato [Monitorama PDX 2015]
Stream Processing Inside Librato [Monitorama PDX 2015]
Nagios
Zabbix
Collectd
DAC (transform and normalize the signal)
(e.g., translate input to a standard json blob)
Zabbix
Filters (de-duplication, noise reduction)
(e.g., compute percentiles to filter outliers)
Multiplex (aggregation)
(e.g., aggregate http 200’s from multiple hosts)
Effects (ad-hoc external signal processing)
??
Effects (ad hoc external signal processing)
??
Stream Processing Inside Librato [Monitorama PDX 2015]
Stream Processing Inside Librato [Monitorama PDX 2015]
Stream Processing Inside Librato [Monitorama PDX 2015]
Stream Processing Inside Librato [Monitorama PDX 2015]
Stream Processing Inside Librato [Monitorama PDX 2015]
Say
“Exponential Smoothing”
again
Disk Latency
Queue Latency
Model A
Model B
Stream Processing Inside Librato [Monitorama PDX 2015]
ing
MultiVariate Data Science Menu
Specials Gaussian
Fuzzy
Discrete
Hybridized
Smoothing
ExponentialSmoothing
Holt-Winters
Chicken Fried Exponential Smoothing
Naieve Baysian
linear regression
Markov Chains
extreme studentized
Kung-Pow distance vector grouping
stream decompisiton
Organic, free-range Ansecomb Quartet
really really smooth Exponential Smoothing
xtra seasonality available on request
Downsampling
Statistical Convergence
Lacunary Sequence
Hashsum Distance
Lambda Convergence
Lambda ANYTHING
Seasonal Hybridized Whatever
HALF OFF!
Anything Named after a turn-of-the-century	

Member of the Russian Intelligencia
Spacial Forecasting
deviant Test
Quantile isolation
Probablistic merging
Hybridized/Fuzzy 	

Combo platter!
Collaborative n-gram cosine matching
Robust Sequential Map-Reduce
Micro-lot N-variate chi-square
Ask about our locally-sourced quantile robustness
Roasted Daily!
Scientist gonna science
Scientist gonna science
Engineer gonna.. yeah.
Scientist gonna scienceData Engineer gonna ??
Making a model is just the beginning
Stream Processing Inside Librato [Monitorama PDX 2015]
Stream Processing Inside Librato [Monitorama PDX 2015]
Stream Processing Inside Librato [Monitorama PDX 2015]
Stream Processing Inside Librato [Monitorama PDX 2015]
Stream Processing Inside Librato [Monitorama PDX 2015]
Stream Processing Inside Librato [Monitorama PDX 2015]
Stream Processing Inside Librato [Monitorama PDX 2015]
Nagios
GraphiosNagios
Graphios StatsdNagios
Graphios Statsd GraphiteNagios
raphios Statsd Graphite Grafana
raphios Statsd Graphite Grafana
Math
thingy
???
Effects (ad hoc external signal processing)
??
Stuff Statsd Graphite Grafana
Stuff Graphite GrafanaHeka
Stuff Graphite GrafanaHeka
Model A
Model B
Stuff Graphite GrafanaHeka
Model A
Model B
carbon
relay
carbon
relay
carbon
relay
carbon
relay
carbon
relay
carbon
relay
carbon
relay
carbon
relay
Stuff Graphite GrafanaHeka
Model A
Model B
carbon
relay
carbon
relay
carbon
relay
carbon
relay
carbon
relay
carbon
relay
carbon
relay
carbon
relay
Elastisearch
Elastisearch
Stream Processing Inside Librato [Monitorama PDX 2015]
Stream Processing Inside Librato [Monitorama PDX 2015]
Stream Processing Inside Librato [Monitorama PDX 2015]
Stream Processing Inside Librato [Monitorama PDX 2015]
Data in Truth out
Stream Processing Inside Librato [Monitorama PDX 2015]
Stream Processing Inside Librato [Monitorama PDX 2015]
X
XX
X
XX
X
X X
X
X
Kafka
SSA
Kafka
SSA
Kafka
SSA
r60
Kafka
SSA
r60
r900
Kafka
SSA
Stream Processing Inside Librato [Monitorama PDX 2015]
SSA
r60
Raw Writer
SSA
r60
Raw Writer
r900 r3600
Stream Processing Inside Librato [Monitorama PDX 2015]
Stream Processing Inside Librato [Monitorama PDX 2015]
SSA
r60
Raw Writer
Alerts
SSA
r60
Raw Writer
Alerts
SSA
MySQL
SSA
Alerts
MySQL
SSA
Alerts
MySQL
SSA
Alerts
MySQL
{
{
SSA
Alerts
MySQL
Storm is “pretty OK”
Lacks visibility into flow, performance, and failures
Netty? JVM Hops? Ser/Der?
Samza? Spark?
NIH?
( kafka + zk + mezos + disruptor+? )
Questions?

More Related Content

Stream Processing Inside Librato [Monitorama PDX 2015]