research-article

Learning a Hierarchical Monitoring System for Detecting and Diagnosing Service Issues

Authors:

Shwetabh Khanduja,

Vikas Bahirwani,

Sundararajan Sellamanickam,

Sathiya Keerthi,

Sudheer DhulipallaAuthors Info & Claims

KDD '15: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Pages 2029 - 2038

https://doi.org/10.1145/2783258.2788624

Published: 10 August 2015 Publication History

Abstract

We propose a machine learning based framework for building a hierarchical monitoring system to detect and diagnose service issues. We demonstrate its use for building a monitoring system for a distributed data storage and computing service consisting of tens of thousands of machines. Our solution has been deployed in production as an end-to-end system, starting from telemetry data collection from individual machines, to a visualization tool for service operators to examine the detection outputs. Evaluation results are presented on detecting 19 customer impacting issues in the past three months.

Supplementary Material

MP4 File (p2029.mp4)

Download
288.05 MB

References

[1]

N. Begum and E. J. Keogh. Rare time series motif discovery from unbounded streams. PVLDB, 8(2):149--160, 2014.

Digital Library

[2]

P. Bodík, M. Goldszmidt, A. Fox, D. B. Woodard, and H. Andersen. Fingerprinting the datacenter: automated classification of performance crises. In EuroSys 2010, pages 111--124, 2010.

Digital Library

[3]

M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander. LOF: Identifying density-based local outliers. SIGMOD Rec., 29(2):93--104, May 2000.

Digital Library

[4]

V. Chandola, A. Banerjee, and V. Kumar. Anomaly detection for discrete sequences: A survey. IEEE Trans. Knowl. Data Eng., 24(5):823--839, 2012.

Digital Library

[5]

H. Chen, G. Jiang, and K. Yoshihira. Failure detection in large-scale internet services by principal subspace mapping. IEEE Trans. Knowl. Data Eng., 19(10):1308--1320, 2007.

Digital Library

[6]

M. Y. Chen, A. X. Zheng, J. Lloyd, M. I. Jordan, and E. A. Brewer. Failure diagnosis using decision trees. In ICAC, pages 36--43, 2004.

Digital Library

[7]

B. J. Frey and D. Dueck. Clustering by passing messages between data points. Science, 315:972--976, 2007.

[8]

Q. Fu, J. Lou, Q. Lin, R. Ding, D. Zhang, Z. Ye, and T. Xie. Performance issue diagnosis for online service systems. In SRDS, pages 273--278, 2012.

Digital Library

[9]

Q. Fu, J. Lou, Y. Wang, and J. Li. Execution anomaly detection in distributed systems through unstructured log analysis. In ICDM, pages 149--158, 2009.

Digital Library

[10]

M. Gabel, R. Glad-Bachrach, N. Bjorner, and A. Schuster. Latent fault detection in cloud services. Technical Report Technical Report, Microsoft Research, 2011.

[11]

A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. in JMLR, 13(1):723--773, Mar. 2012.

Digital Library

[12]

S.-S. Ho and H. Wechsler. A martingale framework for detecting changes in data streams by testing exchangeability. IEEE PAMI, 32(12):2113--2127, 2010.

Digital Library

[13]

D. Lin, R. Raghu, V. Ramamurthy, J. Yu, R. Radhakrishnan, and J. Fernandez. Unveiling clusters of events for alert and incident management in large-scale enterprise it. In KDD, pages 1630--1639, 2014.

Digital Library

[14]

J. Lou, Q. Fu, S. Yang, Y. Xu, and J. Li. Mining invariants from console logs for system problem detection. In USENIX, 2010.

Digital Library

[15]

J. Lou, Q. Lin, R. Ding, Q. Fu, D. Zhang, and T. Xie. Software analytics for incident management of online services: An experience report. In IEEE/ACM ASE, 2013.

[16]

C. Luo, J. Lou, Q. Lin, Q. Fu, R. Ding, D. Zhang, and Z. Wang. Correlating events with time series for incident diagnosis. In KDD, pages 1583--1592, 2014.

Digital Library

[17]

N. Meinshausen and P. Buhlmann. High-dimensional graphs and variable selection with the lasso. The Annals of Statistics, 34(3):1436--1462, June 2006.

[18]

A. Mueen. Time series motif discovery: dimensions and applications. Data Mining and Knowledge Discovery, 4(2):152--159, 2014.

Digital Library

[19]

H. Qiu, Y. Liu, N. A. Subrahmanya, and W. Li. Granger causality for time-series anomaly detection. In ICDM, pages 1074--1079, 2012.

Digital Library

[20]

S. Roy, A. C. Koíg, I. Dvorkin, and M. Kumar. Perfaugur: Robust diagnostics for performance anomalies in cloud services. In ICDE, 2015, 2015.

[21]

B. Scholkopf, J. C. Platt, J. C. Shawe-Taylor, A. J. Smola, and R. C. Williamson. Estimating the support of a high-dimensional distribution. Neural Computation, 2001.

Digital Library

[22]

L. Tang and T. Li. Logtree: A framework for generating system events from raw textual logs. In ICDM, pages 491--500, 2010.

Digital Library

[23]

R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society (Series B), 58:267--288, 1996.

Cited By

Goel DHusain FSingh AGhosh SParayil ABansal CZhang XRajmohan Sd'Amorim M(2024)X-Lifecycle Learning for Cloud Incident Management using LLMsCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663861(417-428)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663861
Zhang DZhang XBansal CLas-Casas PFonseca RRajmohan Sd'Amorim M(2024)LM-PACE: Confidence Estimation by Large Language Models for Effective Root Causing of Cloud IncidentsCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663858(388-398)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663858
Zhang XGhosh SBansal CWang RMa MKang YRajmohan Sd'Amorim M(2024)Automated Root Causing of Cloud Incidents using In-Context Learning with GPT-4Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663846(266-277)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663846
Show More Cited By

Index Terms

Learning a Hierarchical Monitoring System for Detecting and Diagnosing Service Issues
1. Computing methodologies
  1. Machine learning

Recommendations

Robust Anomaly Detection for Multivariate Time Series through Stochastic Recurrent Neural Network
KDD '19: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

Industry devices (i.e., entities) such as server machines, spacecrafts, engines, etc., are typically monitored with multivariate time series, whose anomaly detection is critical for an entity's service quality management. However, due to the complex ...
Anomaly Detection and Failure Root Cause Analysis in (Micro) Service-Based Cloud Applications: A Survey
The proliferation of services and service interactions within microservices and cloud-native applications, makes it harder to detect failures and to identify their possible root causes, which is, on the other hand crucial to promptly recover and fix ...
Correlating events with time series for incident diagnosis
KDD '14: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining

As online services have more and more popular, incident diagnosis has emerged as a critical task in minimizing the service downtime and ensuring high quality of the services provided. For most online services, incident diagnosis is mainly conducted by ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '15: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 2015

2378 pages

ISBN:9781450336642

DOI:10.1145/2783258

General Chairs:
Longbing Cao
University of Technology, Sydney
,
Chengqi Zhang
University of Technology, Sydney
,
Program Chairs:
Thorsten Joachims
Cornell University
,
Geoff Webb
Monash University
,
Dragos D. Margineantu
Boeing Research
,
Graham Williams
Australian Taxation Office

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 August 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

KDD '15

Sponsor:

KDD '15: The 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 10 - 13, 2015

NSW, Sydney, Australia

Acceptance Rates

KDD '15 Paper Acceptance Rate 160 of 819 submissions, 20%;

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '25

Sponsor:
sigkdd
sigkdd

The 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 3 - 7, 2025

Toronto , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

51
Total Citations
View Citations
759
Total Downloads

Downloads (Last 12 months)38
Downloads (Last 6 weeks)15

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Goel DHusain FSingh AGhosh SParayil ABansal CZhang XRajmohan Sd'Amorim M(2024)X-Lifecycle Learning for Cloud Incident Management using LLMsCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663861(417-428)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663861
Zhang DZhang XBansal CLas-Casas PFonseca RRajmohan Sd'Amorim M(2024)LM-PACE: Confidence Estimation by Large Language Models for Effective Root Causing of Cloud IncidentsCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663858(388-398)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663858
Zhang XGhosh SBansal CWang RMa MKang YRajmohan Sd'Amorim M(2024)Automated Root Causing of Cloud Incidents using In-Context Learning with GPT-4Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663846(266-277)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663846
Yao ZPei CChen WWang HSu LJiang HXie ZNie XPei Dd'Amorim M(2024)Chain-of-Event: Interpretable Root Cause Analysis for Microservices through Automatically Learning Weighted Event Causal GraphCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663827(50-61)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663827
Srinivas PHusain FParayil AChoure ABansal CRajmohan SRoychoudhury APaiva AAbreu RStorey MAniche MNagappan N(2024)Intelligent Monitoring Framework for Cloud Services: A Data-Driven ApproachProceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice10.1145/3639477.3639753(381-391)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639477.3639753
Yao ZYe HPei CCheng GWang GLiu ZChen HCui HLi ZLi JXie GPei D(2024)SparseRCA: Unsupervised Root Cause Analysis in Sparse Microservice Testing Traces2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE62328.2024.00045(391-402)Online publication date: 28-Oct-2024
https://doi.org/10.1109/ISSRE62328.2024.00045
Pan LZhu H(2023)An Intelligent Framework for Log Anomaly Detection Based on Log Template ExtractionJournal of Cases on Information Technology10.4018/JCIT.33014525:1(1-23)Online publication date: 12-Sep-2023
https://doi.org/10.4018/JCIT.330145
Ganatra VParayil AGhosh SKang YMa MBansal CNath SMace JChandra SBlincoe KTonella P(2023)Detection Is Better Than Cure: A Cloud Incidents PerspectiveProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613898(1891-1902)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3613898
Zeighami SShahabi CSharan V(2023)NeuroSketch: Fast and Approximate Evaluation of Range Aggregate Queries with Neural NetworksProceedings of the ACM on Management of Data10.1145/35889541:1(1-26)Online publication date: 30-May-2023
https://dl.acm.org/doi/10.1145/3588954
Barbarioli BMersy GSintos SKrishnan S(2023)Hierarchical Residual Encoding for Multiresolution Time Series CompressionProceedings of the ACM on Management of Data10.1145/35889531:1(1-26)Online publication date: 30-May-2023
https://dl.acm.org/doi/10.1145/3588953
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents