research-article

An empirical investigation of missing data handling in cloud node failure prediction

Authors:

Saravanakumar Rajmohan, and

Qingwei LinAuthors Info & Claims

ESEC/FSE 2022: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

November 2022

Pages 1453 - 1464

https://doi.org/10.1145/3540250.3558946

Published: 09 November 2022 Publication History

Abstract

Cloud computing systems have become increasingly popular in recent years. A typical cloud system utilizes millions of computing nodes as the basic infrastructure. Node failure has been identified as one of the most prevalent causes of cloud system downtime. To improve the reliability of cloud systems, many previous studies collected monitoring metrics from nodes and built models to predict node failures before the failures happen. However, based on our experience with large-scale real-world cloud systems in Microsoft, we find that the task of predicting node failure is severely hampered by missing data. There is a large amount of missing data, and the online latest data utilized for prediction is even worse. As a result, the real-time performance of the node prediction model is limited. In this paper, we first characterize the missing data problem for node failure prediction. Then, we evaluate several existing data interpolation approaches, and find that node dimension interpolation approaches outperform time dimension ones and deep learning based interpolation is the best for early prediction. Our findings can help academics and engineers address the missing data problem in cloud node failure prediction and other data-driven software engineering scenarios.

References

[1]

K. Mohaideen Abdul Kadhar and G. Anand. 2021. Preparing the Data. Apress, Berkeley, CA. 99–119.

[2]

Moria Bergman, Tova Milo, Slava Novgorodov, and Wang-Chiew Tan. 2015. Query-oriented data cleaning with oracles. In Proceedings of the Conference on Management of Data (SIGMOD). ACM, 1199–1214.

Digital Library

[3]

Mirela Madalina Botezatu, Ioana Giurgiu, Jasmina Bogojeska, and Dorothea Wiesmann. 2016. Predicting Disk Replacement towards Reliable Data Centers. In Proceedings of the Knowledge Discovery and Data Mining (SIGKDD). ACM, 39–48.

Digital Library

[4]

Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. 2018. Brits: Bidirectional recurrent imputation for time series. Advances in neural information processing systems, 31 (2018).

[5]

Chengliang Chai, Lei Cao, Guoliang Li, Jian Li, Yuyu Luo, and Samuel Madden. 2020. Human-in-the-loop outlier detection. In Proceedings of the Conference on Management of Data (SIGMOD). ACM, 19–33.

Digital Library

[6]

Yujun Chen, Xian Yang, Qingwei Lin, Hongyu Zhang, Feng Gao, Zhangwei Xu, Yingnong Dang, Dongmei Zhang, Hang Dong, Yong Xu, Hao Li, and Yu Kang. 2019. Outage Prediction and Diagnosis for Cloud Service Systems. In Proceedings of the International World Wide Web Conferences (WWW). ACM, 2659–2665.

Digital Library

[7]

Thomas Cover and Peter Hart. 1967. Nearest neighbor pattern classification. IEEE Transactions on Information Theory (TIT), 13, 1 (1967), 21–27.

Digital Library

[8]

Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. 2005. A tutorial on the cross-entropy method. Annals of operations research, 134, 1 (2005), 19–67.

[9]

Supratim Deb, Zihui Ge, Sastry Isukapalli, Sarat Puthenpura, Shobha Venkataraman, He Yan, and Jennifer Yates. 2017. Aesop: Automatic policy learning for predicting and mitigating network service impairments. In Proceedings of the Knowledge Discovery and Data Mining (SIGKDD). ACM, 1783–1792.

Digital Library

[10]

Ori Hadary, Luke Marshall, Ishai Menache, Abhisek Pan, Esaias E Greeff, David Dion, Star Dorminey, Shailesh Joshi, Yang Chen, and Mark Russinovich. 2020. Protean: VM allocation service at scale. In Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI). USENIX, 845–861.

[11]

Shilin He, Qingwei Lin, Jian-Guang Lou, Hongyu Zhang, Michael R Lyu, and Dongmei Zhang. 2018. Identifying impactful service system problems via log analysis. In Proceedings of the Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). ACM, 60–70.

Digital Library

[12]

Xiao-Yuan Jing, Fumin Qi, Fei Wu, and Baowen Xu. 2016. Missing data imputation based on low-rank recovery and semi-supervised regression for software effort estimation. In Proceedings of the International Conference on Software Engineering (ICSE). ACM/IEEE, 607–618.

Digital Library

[13]

Sunghun Kim, Hongyu Zhang, Rongxin Wu, and Liang Gong. 2011. Dealing with noise in defect prediction. In Proceedings of the International Conference on Software Engineering (ICSE). IEEE, 481–490.

Digital Library

[14]

Anil C Kokaram, Robin D Morris, William J Fitzgerald, and Peter JW Rayner. 1995. Interpolation of missing data in image sequences. Transactions on Image Processing (TIP), 4, 11 (1995), 1509–1519.

Digital Library

[15]

Jing Li, Xinpu Ji, Yuhan Jia, Bingpeng Zhu, Gang Wang, Zhongwei Li, and Xiaoguang Liu. 2014. Hard Drive Failure Prediction Using Classification and Regression Trees. In Proceedings of the International Conference on Dependable Systems and Networks (DSN). IEEE/IFIP, 383–394.

Digital Library

[16]

Zhenhao Li, Tse-Hsun Chen, and Weiyi Shang. 2020. Where shall we log? studying and suggesting logging locations in code blocks. In Proceedings of the International Conference on Automated Software Engineering (ASE). IEEE/ACM, 361–372.

Digital Library

[17]

Qingwei Lin, Ken Hsieh, Yingnong Dang, Hongyu Zhang, Kaixin Sui, Yong Xu, Jian-Guang Lou, Chenggang Li, Youjiang Wu, and Randolph Yao. 2018. Predicting node failure in cloud service systems. In Proceedings of the Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). ACM, 480–490.

Digital Library

[18]

Qingwei Lin, Hongyu Zhang, Jian-Guang Lou, Yu Zhang, and Xuewei Chen. 2016. Log clustering based problem identification for online service systems. In Proceedings of the International Conference on Software Engineering Companion (ICSE-Companion). ACM/IEEE, 102–111.

Digital Library

[19]

Yudong Liu, Hailan Yang, Pu Zhao, Minghua Ma, Chengwu Wen, Hongyu Zhang, Chuan Luo, Qingwei Lin, Chang Yi, and Jiaojian Wang. 2022. Multi-task Hierarchical Classification for Disk Failure Prediction in Online Service Systems. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 3438–3446.

Digital Library

[20]

Sidi Lu, Bing Luo, Tirthak Patel, Yongtao Yao, Devesh Tiwari, and Weisong Shi. 2020. Making Disk Failure Predictions SMARTer!. In Proceedings of the Conference on File and Storage Technologies (FAST). USENIX, 151–167.

[21]

Chuan Luo, Pu Zhao, Bo Qiao, Youjiang Wu, Hongyu Zhang, Wei Wu, Weihai Lu, Yingnong Dang, Saravanakumar Rajmohan, Qingwei Lin, and Dongmei Zhang. 2021. NTAM: Neighborhood-Temporal Attention Model for Disk Failure Prediction in Cloud Platforms. In Proceedings of the International World Wide Web Conferences (WWW). ACM, 1181–1191.

Digital Library

[22]

Ao Ma, Rachel Traylor, Fred Douglis, Mark Chamness, Guanlin Lu, Darren Sawyer, Surendar Chandra, and Windsor Hsu. 2015. RAIDShield: characterizing, monitoring, and proactively protecting against disk failures. Transactions on Storage (TOS), 11, 4 (2015), 1–28.

Digital Library

[23]

Minghua Ma, Zheng Yin, Shenglin Zhang, Sheng Wang, Christopher Zheng, Xinhao Jiang, Hanwen Hu, Cheng Luo, Yilin Li, and Nengjun Qiu. 2020. Diagnosing root causes of intermittent slow queries in cloud databases. Proceedings of the VLDB Endowment, 13, 8 (2020), 1176–1189.

Digital Library

[24]

Minghua Ma, Shenglin Zhang, Junjie Chen, Jim Xu, Haozhe Li, Yongliang Lin, Xiaohui Nie, Bo Zhou, Yong Wang, and Dan Pei. 2021. Jump-Starting Multivariate Time Series Anomaly Detection for Online Service Systems. In Proceedings of the Annul Technical Conference (ATC). USENIX, 413–426.

[25]

Minghua Ma, Shenglin Zhang, Dan Pei, Xin Huang, and Hongwei Dai. 2018. Robust and rapid adaption for concept drift in software system anomaly detection. In Proceedings of the 29th International Symposium on Software Reliability Engineering (ISSRE). 13–24.

[26]

Sonu Mehta, Ranjita Bhagwan, Rahul Kumar, Chetan Bansal, Chandra Maddila, B Ashok, Sumit Asthana, Christian Bird, and Aditya Kumar. 2020. Rex: Preventing bugs and misconfiguration in large services using correlated change analysis. In Proceedings of the Symposium on Networked Systems Design and Implementation (NSDI). USENIX, 435–448.

[27]

Bertrand Muquet, Zhengdao Wang, Georgios B Giannakis, Marc De Courville, and Pierre Duhamel. 2002. Cyclic prefixing or zero padding for wireless multicarrier transmissions? Transactions on communications, 50, 12 (2002), 2136–2148.

[28]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 32, MIT Press, 8024–8035.

[29]

Daniel Peña and George C Tiao. 1991. A note on likelihood estimation of missing values in time series. The American Statistician, 45, 3 (1991), 212–213.

[30]

ALEXANDER L Read. 1999. Linear interpolation of histograms. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 425, 1-2 (1999), 357–360.

[31]

Weiyi Shang, Zhen Ming Jiang, Hadi Hemmati, Brain Adams, Ahmed E Hassan, and Patrick Martin. 2013. Assisting developers of big data analytics applications when deploying on hadoop clouds. In Proceedings of the International Conference on Software Engineering (ICSE). IEEE, 402–411.

[32]

Qinbao Song and Martin Shepperd. 2007. Missing data imputation techniques. International journal of business intelligence and data mining, 2, 3 (2007), 261–291.

Digital Library

[33]

Ming Sun, Ya Su, Shenglin Zhang, Yuanpu Cao, Yuqing Liu, Dan Pei, Wenfei Wu, Yongsu Zhang, Xiaozhou Liu, and Junliang Tang. 2021. CTF: Anomaly Detection in High-Dimensional Time Series with Coarse-to-Fine Model Transfer. In Proceedings of the International Conference on Computer Communications (INFOCOM). IEEE, 1–10.

Digital Library

[34]

Xiaoyi Sun, Krishnendu Chakrabarty, Ruirui Huang, Yiquan Chen, Bing Zhao, Hai Cao, Yinhe Han, Xiaoyao Liang, and Li Jiang. 2019. System-Level Hardware Failure Prediction using Deep Learning. In Proceedings of the Design Automation Conference (DAC). ACM, 20.

Digital Library

[35]

Olga Troyanskaya, Michael Cantor, Gavin Sherlock, Pat Brown, Trevor Hastie, Robert Tibshirani, David Botstein, and Russ B Altman. 2001. Missing value estimation methods for DNA microarrays. Bioinformatics, 17, 6 (2001), 520–525.

[36]

Kashi Venkatesh Vishwanath and Nachiappan Nagappan. 2010. Characterizing cloud computing hardware reliability. In Proceedings of the Symposium on Cloud Computing (SoCC). ACM, 193–204.

Digital Library

[37]

Wikipedia. 2021. Facebook Outage. https://en.wikipedia.org/wiki/2021_Facebook_outage

[38]

Rongxin Wu, Hongyu Zhang, Sunghun Kim, and Shing-Chi Cheung. 2011. Relink: recovering links between bugs and changes. In Proceedings of the Symposium on the Foundation of Software Engineering European Software Engineering Conference (ESEC/FSE). ACM, 15–25.

Digital Library

[39]

Yang Wu, Mingchen Zhao, Andreas Haeberlen, Wenchao Zhou, and Boon Thau Loo. 2014. Diagnosing missing events in distributed systems with negative provenance. SIGCOMM Computer Communication Review, 44, 4 (2014), 383–394.

Digital Library

[40]

Chang Xu, Gang Wang, Xiaoguang Liu, Dongdong Guo, and Tie-Yan Liu. 2016. Health Status Assessment and Failure Prediction for Hard Drives with Recurrent Neural Networks. Transactions on Computers, 65, 11 (2016), 3502–3508.

Digital Library

[41]

Yong Xu, Kaixin Sui, Randolph Yao, Hongyu Zhang, Qingwei Lin, Yingnong Dang, Peng Li, Keceng Jiang, Wenchi Zhang, Jian-Guang Lou, Murali Chintalapati, and Dongmei Zhang. 2018. Improving Service Availability of Cloud Systems by Predicting Disk Error. In Proceedings of the USENIX Annul Technical Conference (ATC). USENIX, 481–494.

[42]

Jiaxuan You, Xiaobai Ma, Daisy Yi Ding, Mykel J. Kochenderfer, and Jure Leskovec. 2020. Handling Missing Data with Graph Representation Learning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). 33, MIT Press, 19075–19087.

[43]

Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. 2019. Self-attention generative adversarial networks. In Proceedings of the International Conference on Machine Learning (ICML). ACM, 7354–7363.

[44]

Jianguo Zhang, Ji Wang, Lifang He, Zhao Li, and Philip S. Yu. 2018. Layerwise Perturbation-Based Adversarial Training for Hard Drive Health Degree Prediction. In Proceedings of the International Conference on Data Mining (ICDM). IEEE, 1428–1433.

[45]

Qiao Zhang, Guo Yu, Chuanxiong Guo, Yingnong Dang, Nick Swanson, Xinsheng Yang, Randolph Yao, Murali Chintalapati, Arvind Krishnamurthy, and Thomas Anderson. 2018. Deepview: Virtual Disk Failure Diagnosis and Pattern Detection for Azure. In Proceedings of the Symposium on Networked Systems Design and Implementation (NSDI). USENIX, Renton, WA. 519–532. isbn:978-1-939133-01-4

Cited By

Ma LYang WXu BJiang SFei BLiang JZhou MXiao YRoychoudhury APaiva AAbreu RStorey M(2024)KnowLog: Knowledge Enhanced Pre-trained Language Model for Log UnderstandingProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3623304(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3623304
Duan CYang FZhao PZheng LDagli YLiu YLin QZhang DChua TNgo CKa-Wei Lee RKumar RLauw H(2024)SOIL: Score Conditioned Diffusion Model for Imbalanced Cloud Failure PredictionCompanion Proceedings of the ACM on Web Conference 202410.1145/3589335.3648303(65-72)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589335.3648303
Song YZheng PTian YWang B(2024)ACPR: Adaptive Classification Predictive Repair Method for Different Fault ScenariosIEEE Access10.1109/ACCESS.2023.334688112(4631-4641)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2023.3346881
Show More Cited By

Index Terms

An empirical investigation of missing data handling in cloud node failure prediction

Recommendations

Predicting Node failure in cloud service systems
ESEC/FSE 2018: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

In recent years, many traditional software systems have migrated to cloud computing platforms and are provided as online services. The service quality matters because system failures could seriously affect business and user experience. A cloud service ...
Read More
A co-training approach for time series prediction with missing data
MCS'07: Proceedings of the 7th international conference on Multiple classifier systems

In this paper we consider the problem of missing data in time series analysis. We propose a semi-supervised co-training method to handle the problem of missing data. We transform the time series data to set of labeled and unlabeled data. Different ...
Read More
Multiple predictively equivalent risk models for handling missing data at time of prediction: With an application in severe hypoglycemia risk prediction for type 2 diabetes
Graphical abstract
Consider a scenario where we want to predict SH risk for a patient with SBP and A1C missing. When we only have one (hypothetical) risk model based on age, SBP and A1C, we could handle the missing SBP and AIC by (1) measuring ...
Highlights
- Missing data at the time of prediction limits the use of risk models
- We ...
Abstract
The presence of missing data at the time of prediction limits the application of risk models in clinical and research settings. Common ways of handling missing data at the time of prediction include measuring the missing value and ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEC/FSE 2022: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

November 2022

1822 pages

ISBN:9781450394130

DOI:10.1145/3540250

General Chair:
Abhik Roychoudhury
National University of Singapore, Singapore
,
Program Chairs:
Cristian Cadar
Imperial College London, UK
,
Miryung Kim
University of California at Los Angeles, USA

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 November 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ESEC/FSE '22

Sponsor:

ESEC/FSE '22: 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

November 14 - 18, 2022

Singapore, Singapore

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
230
Total Downloads

Downloads (Last 12 months)101
Downloads (Last 6 weeks)11

Other Metrics

View Author Metrics

Citations

Cited By

Ma LYang WXu BJiang SFei BLiang JZhou MXiao YRoychoudhury APaiva AAbreu RStorey M(2024)KnowLog: Knowledge Enhanced Pre-trained Language Model for Log UnderstandingProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3623304(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3623304
Duan CYang FZhao PZheng LDagli YLiu YLin QZhang DChua TNgo CKa-Wei Lee RKumar RLauw H(2024)SOIL: Score Conditioned Diffusion Model for Imbalanced Cloud Failure PredictionCompanion Proceedings of the ACM on Web Conference 202410.1145/3589335.3648303(65-72)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589335.3648303
Song YZheng PTian YWang B(2024)ACPR: Adaptive Classification Predictive Repair Method for Different Fault ScenariosIEEE Access10.1109/ACCESS.2023.334688112(4631-4641)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2023.3346881
Chen YZhang CMa MLiu YDing RLi BHe SRajmohan SLin QZhang D(2023)ImDiffusion: Imputed Diffusion Models for Multivariate Time Series Anomaly DetectionProceedings of the VLDB Endowment10.14778/3632093.363210117:3(359-372)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.14778/3632093.3632101
Yang FYin WWang LLi TZhao PLiu BWang PQiao BLiu YBjörkman MRajmohan SLin QZhang DChandra SBlincoe KTonella P(2023)Diffusion-Based Time Series Data Imputation for Cloud Failure Prediction at Microsoft 365Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613866(2050-2055)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3613866
Ding RZhang CWang LXu YMa MWu XZhang MChen QGao XGao XFan HRajmohan SLin QZhang DChandra SBlincoe KTonella P(2023)TraceDiag: Adaptive, Interpretable, and Efficient Root Cause Analysis on Large-Scale Microservice SystemsProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613864(1762-1773)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3613864
Zhao CMa MZhong ZZhang STan ZXiong XYu LFeng JSun YZhang YPei DLin QZhang DSingh ASun YAkoglu LGunopulos DYan XKumar ROzcan FYe J(2023)Robust Multimodal Failure Detection for Microservice SystemsProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3580305.3599902(5639-5649)Online publication date: 6-Aug-2023
https://dl.acm.org/doi/10.1145/3580305.3599902
Li HMa MLiu YQin SQiao BYao RChaturvedi HTran TChintalapati MRajmohan SLin QZhang D(2023)CODEC: Cost-Effective Duration Prediction System for Deadline Scheduling in the Cloud2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE59848.2023.00069(298-308)Online publication date: 9-Oct-2023
https://doi.org/10.1109/ISSRE59848.2023.00069
Zeng ZZhang YXu YMa MQiao BZou WChen QZhang MZhang XZhang HGao XFan HRajmohan SLin QZhang DLiu AMuccini H(2023)TraceArk: Towards Actionable Performance Anomaly Alerting for Online Service SystemsProceedings of the 45th International Conference on Software Engineering: Software Engineering in Practice10.1109/ICSE-SEIP58684.2023.00029(258-269)Online publication date: 17-May-2023
https://dl.acm.org/doi/10.1109/ICSE-SEIP58684.2023.00029
Yan XHsieh KLiyanage YMa MChintalapati MLin QDang YZhang DLiu AMuccini H(2023)Aegis: Attribution of Control Plane Change Impact across Layers and Components for Cloud SystemsProceedings of the 45th International Conference on Software Engineering: Software Engineering in Practice10.1109/ICSE-SEIP58684.2023.00026(222-233)Online publication date: 17-May-2023
https://dl.acm.org/doi/10.1109/ICSE-SEIP58684.2023.00026
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents