Dataday Texas 2016 - Datadog
•
8 likes•2,857 views
Data scientist Homin Lee talks about automatically detecing infrastructure anomalies and outliers in real-time with Datadog at Dataday Texas 2016.
Report
Share
More Related Content
What's hot
What's hot (20)
Similar to Dataday Texas 2016 - Datadog
Similar to Dataday Texas 2016 - Datadog (20)
More from Datadog
More from Datadog (20)
Recently uploaded
Recently uploaded (20)
Dataday Texas 2016 - Datadog
- 1. Automatically Detecting Anomalies and Outliers in Real-Time Homin Lee, Data Scientist
- 2. Outline ● Monitoring ● Alerting ● Outlier vs. Anomaly Detection ● Outlier Detection Algorithms ● Anomaly Detection Algorithms
- 4. Monitor Everything Datadog gathers performance data from all your application components.
- 8. Alerting
- 9. Alerting?
- 10. Alerting?
- 11. Outlier and Anomaly Detection
- 16. Outlier Detection Algorithms MAD median absolute deviation DBSCAN density-based spatial clustering of applications with noise
- 17. Robust Outlier Detection Algorithms
- 18. Median Absolute Deviation MAD(D) = median( { |di - median(D)| } )
- 19. Median Absolute Deviation MAD(D) = median( { |di - median(D)| } ) D = { 1, 2, 3, 4, 5, 6, 100 }
- 20. Median Absolute Deviation MAD(D) = median( { |di - median(D)| } ) D = { 1, 2, 3, 4, 5, 6, 100 } median = 4
- 21. Median Absolute Deviation MAD(D) = median( { |di - median(D)| } ) D = { 1, 2, 3, 4, 5, 6, 100 } median = 4 deviations = { -3, -2, -1, 0, 1, 2, 96 }
- 22. Median Absolute Deviation MAD(D) = median( { |di - median(D)| } ) D = { 1, 2, 3, 4, 5, 6, 100 } median = 4 deviations = { -3, -2, -1, 0, 1, 2, 96 } abs deviations = { 0, 1, 1, 2, 2, 3, 96 }
- 23. Median Absolute Deviation MAD(D) = median( { |di - median(D)| } ) D = { 1, 2, 3, 4, 5, 6, 100 } median = 4 deviations = { -3, -2, -1, 0, 1, 2, 96 } abs deviations = { 0, 1, 1, 2, 2, 3, 96 } MAD = 2
- 24. Median Absolute Deviation MAD(D) = median( { |di - median(D)| } ) D = { 1, 2, 3, 4, 5, 6, 100 } median = 4 deviations = { -3, -2, -1, 0, 1, 2, 96 } abs deviations = { 0, 1, 1, 2, 2, 3, 96 } MAD = 2 (std dev = 33.8)
- 25. Median Absolute Deviation Parameters: Tolerance, Pct } tol. = 3.0
- 26. DBSCAN
- 30. DBSCAN 1 dd/2d/4 3d/4 ~ median(dist from median series) × tolerance
- 31. MAD or DBSCAN?
- 32. MAD or DBSCAN?
- 33. Some subtleties
- 34. Some subtleties
- 35. Some subtleties
- 37. An Investigation past 30 minutes
- 38. An Investigation past 30 minutes past day
- 39. An Investigation past day past week past 30 minutes
- 40. An Investigation past day past week past 30 minutes past 5 weeks
- 41. An Investigation past day ANOMALY past 30 minutes past week past 5 weeks
- 42. Anomalies A time series point is an anomaly if: ● Given the past points in the series ( ), the point in question ( ) is unlikely given your model of the past;
- 43. Anomalies A time series point is an anomaly if: ● Given the past points in the series ( ), the point in question ( ) is unlikely given your model of the past; and you should alert on a set of anomalies if: ● they are a symptom of an issue you care about ( ).
- 44. Our Approach 1. Extract as much signal as we can from the time series. 2. Use robust statistical measures when creating the model. 3. Give the user control over when they get alerted.
- 45. What’s Normal?
- 46. What’s Normal?
- 47. What’s Normal?
- 48. What’s Normal?
- 49. Past Performance... past 30 minutes past day past week past 5 weeks
- 51. Decomposition
- 52. Decomposition
- 53. Decomposition
- 54. Decomposition
- 55. Decomposition
- 56. Autocorrelation
- 57. Signal vs. Noise
- 58. Signal vs. Noise
- 59. Signal vs. Noise vs. Signal
- 66. Alerting
- 67. Alerting
- 68. Recap ● Extract as much signal as you can. ● Use robust statistical measures. ● Alert judiciously. ● Don’t over-optimize.
- 70. Thanks!
- 71. Appendix
- 72. DASHBOARDS Build Real-Time Interactive Dashboards CORRELATION Search And Correlate Metrics And Events See It All In One Place Your Servers, Your Clouds, Your Metrics, Your Apps, Your team. Together.
- 73. COLLABORATION Share What You Saw, Write What You Did METRIC ALERTS Get Alerted On Critical Issues DEVELOPER API Instrument Your Apps, Write New Integrations See It All In One Place Your Servers, Your Clouds, Your Metrics, Your Apps, Your team. Together.
- 74. Flexible Pricing To Match Your Dynamic Infrastructure. Free Up to 5 Hosts 1 Day retention Custom metrics and events Discussion group supported Pro Up to 500 Hosts $15 Per Host / Month 13 Month retention Custom metrics and events Metric alerts* Email supported Enterprise 500+ Hosts Contact us for pricing: +1 866 329 4466 sales@datadoghq.com Customized retention Custom metrics and events Metric alerts* Email and phone supported