research-article

Anomaly Detection for Container Cluster based on JointCloud Platform

Authors:

Chunge ZhuAuthors Info & Claims

ICCDA '19: Proceedings of the 2019 3rd International Conference on Compute and Data Analysis

Pages 26 - 30

https://doi.org/10.1145/3314545.3314567

Published: 14 March 2019 Publication History

Abstract

In order to accurately discover container exception data of large-scale container clusters to guide the maintenance of container clusters, a new anomaly detection model for container clusters is proposed in this article. The model combines the advantages of supervised learning and unsupervised learning to accurately and efficiently label container anomaly data in a large-scale data environment. Experiments show that the labeling rate of the raw data is as high as 95.6%, and the accuracy of anomaly detection is as high as 87.0%. Simultaneously, the common five classification algorithms are used to compare the anomaly detection effect between the labeled data and the raw data, and the validity of the model is further verified.

References

[1]

Malte Schwarzkopf, Andy Konwinski, Michael Abd-El-Malek, and John Wilkes. 2013. Omega: flexible, scalable schedulers for large compute clusters. In Proceedings of the 8th ACM European Conference on Computer Systems (Prague, Czech Republic, Apr 15 - 17, 2013). EuroSys'13. ACM, New York, NY, USA, 351--364.

Digital Library

[2]

Abhishek Verma, Luis Pedrosa, Madhukar Korupolu, David Oppenheimer, Eric Tune, and John Wilkes. 2015. Large-scale cluster management at Google with Borg. In Proceedings of the Tenth European Conference on Computer Systems (Bordeaux, France, Aprial 21 - 25, 2015). EuroSys'15. ACM, New York, NY, USA, Article 18, 17 pages.

Digital Library

[3]

Eric Boutin, Jaliya Ekanayake, Wei Lin, Bing Shi, Jingren Zhou, Zhengping Qian, Ming Wu, and Lidong Zhou. 2014. Apollo: scalable and coordinated scheduling for cloud-scale computing. In Proceedings of the 11th USENIX conference on Operating Systems Design and Implementation (Broomfield, CO, USA, October 6 - 8, 2014). OSDI'14. USENIX Association, Berkeley, CA, USA, 285--300.

Digital Library

[4]

Apache Aurora. http://aurora.apache.org/.

[5]

Amazon ECS. https://aws.amazon.com/cn/ecs.

[6]

Docker Swarm. https://docs.docker.com/engine/swarm.

[7]

Openstack. https://www.openstack.org.

[8]

Brendan Burns, Brian Grant, David Oppenheimer, Eric Brewer, and John Wilkes. 2016. Borg, Omega, and Kubernetes. Commun. ACM 59, 5 (April 2016), 50--57.

Digital Library

[9]

Apache Mesos. http://mesos.apache.org.

[10]

Amazon Web Services outage causes widespread internet problems. https://www.cbsnews.com/news/amazon-web-services-cloud-outage-internet-crashes.

[11]

Microsoft Azure Customers Faced Seven Hour Outage. https://winbuzzer.com/2017/03/16/microsoft-azure-customers-faced-seven-hour-outage-yesterday-xcxwbn.

[12]

Chengwei Wang. 2009. EbAT: online methods for detecting utility cloud anomalies. In Proceedings of the 6th Middleware Doctoral Symposium (Urbana Champaign, Illinois, USA, November 30, 2009).MDS'09. ACM, New York, NY, USA, Article 4, 6 pages.

Digital Library

[13]

Du Q., Xie T., He Y. 2018. Anomaly Detection and Diagnosis for Container-Based Microservices with Performance Monitoring. In International Conference on Algorithms and Architectures for Parallel Processing. (Guangzhou, China, November 15 - 17, 2018). ICA3PP'18. Springer Nature Switzerland AG, 560--572.

[14]

Tenenbaum, J. B., De Silva, V., & Langford, J. 2000. A Global Geometric Framework for Nonlinear Dimensionality Reduction. Science, Vol. 290, Issue 5500 (Jan. 2000), 2319--2323.

[15]

Roweis S T, Saul L K. 2000. Nonlinear Dimensionality Reduction by Locally Linear Embedding. Science, Vol. 290, Issue 5500 (Jan. 2000), 2323--2326.

[16]

Wong R C, Leung C H. 2008. Automatic Semantic Annotation of Real-World Web Images. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 30, Issue 11 (Dec. 2008), 1933--1944.

Digital Library

[17]

Kuroda K, Hagiwara M. 2002. An image retrieval system by impression words and specific object names-IRIS. Neurocomputing, Vol. 43, Issue 1 (Mar. 2002), 259--276.

[18]

He, S., Zhu, J., He, P., & Lyu, M. R. 2016. Experience Report: System Log Analysis for Anomaly Detection. In International symposium on software reliability engineering (Ottawa, ON, Canada, October 23-27, 2016.). ISSRE'16. IEEE 207--218.

[19]

Zheng, D., Li, F., & Zhao, T. 2016. Self-adaptive statistical process control for anomaly detection in time series. Expert Systems With Applications, Vol.57 (Sep. 2016), 324--336.

Digital Library

[20]

Guan, Q., Fu, S., Debardeleben, N., & Blanchard, S. 2013. Exploring Time and Frequency Domains for Accurate and Automated Anomaly Detection in Cloud Computing Systems. In Pacific rim international symposium on dependable computing(Vancouver, BC, Canada, Dec. 02-04, 2013). PRDC'13. IEEE: 196--205.

Digital Library

[21]

Quinlan J R. 1986. Induction of Decision Trees. Machine Learning, Vol. 1, Issue 1 (Jan. 1986), 81--106.

Digital Library

[22]

Huaimin Wang, Peichang Shi, Yiming Zhang. 2017. JointCloud: A Cross-Cloud Cooperation Architecture for Integrated Internet Service Customization. International conference on distributed computing systems (Atlanta, GA, USA, June 05-08, 2017). ICDCS'17. IEEE: 1846--1855.

[23]

Huaimin Wang. Bo Ding. 2016. Growing construction and adaptive evolution of complex software systems. SCIENCE CHINA Information Sciences, Vol.59, Issue 6 (Apr. 2016), 743--761.

Cited By

Joraviya NGohil BRao U(2024)DL-HIDS: deep learning-based host intrusion detection system using system calls-to-image for containerized cloud environmentThe Journal of Supercomputing10.1007/s11227-024-05895-380:9(12218-12246)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s11227-024-05895-3
Joraviya NGohil BRao U(2024)Ab‐HIDS: An anomaly‐based host intrusion detection system using frequency of N‐gram system call features and ensemble learning for containerized environmentConcurrency and Computation: Practice and Experience10.1002/cpe.824936:23Online publication date: 6-Aug-2024
https://doi.org/10.1002/cpe.8249
Janecek MEzzati-Jivan NAzhari S(2021)Container Workload Characterization Through Host System Tracing2021 IEEE International Conference on Cloud Engineering (IC2E)10.1109/IC2E52221.2021.00015(9-19)Online publication date: Oct-2021
https://doi.org/10.1109/IC2E52221.2021.00015

Index Terms

Anomaly Detection for Container Cluster based on JointCloud Platform
1. Computing methodologies
  1. Machine learning
    1. Machine learning algorithms

Recommendations

Autoencoding Binary Classifiers for Supervised Anomaly Detection
PRICAI 2019: Trends in Artificial Intelligence
Abstract
We propose the Autoencoding Binary Classifiers (ABC), a novel supervised anomaly detector based on the Autoencoder (AE). There are two main approaches in anomaly detection: supervised and unsupervised. The supervised approach accurately detects ...
Anomaly Detection in Cybersecurity Datasets via Cooperative Co-evolution-based Feature Selection
Anomaly detection from Big Cybersecurity Datasets is very important; however, this is a very challenging and computationally expensive task. Feature selection (FS) is an approach to remove irrelevant and redundant features and select a subset of features, ...
Anomaly pattern detection in categorical datasets
KDD '08: Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining

We propose a new method for detecting patterns of anomalies in categorical datasets. We assume that anomalies are generated by some underlying process which affects only a particular subset of the data. Our method consists of two steps: we first use a "...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICCDA '19: Proceedings of the 2019 3rd International Conference on Compute and Data Analysis

March 2019

163 pages

ISBN:9781450366342

DOI:10.1145/3314545

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 March 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Key R&D Program of China

Conference

ICCDA 2019

ICCDA 2019: 2019 The 3rd International Conference on Compute and Data Analysis

March 14 - 17, 2019

HI, Kahului, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
149
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Joraviya NGohil BRao U(2024)DL-HIDS: deep learning-based host intrusion detection system using system calls-to-image for containerized cloud environmentThe Journal of Supercomputing10.1007/s11227-024-05895-380:9(12218-12246)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s11227-024-05895-3
Joraviya NGohil BRao U(2024)Ab‐HIDS: An anomaly‐based host intrusion detection system using frequency of N‐gram system call features and ensemble learning for containerized environmentConcurrency and Computation: Practice and Experience10.1002/cpe.824936:23Online publication date: 6-Aug-2024
https://doi.org/10.1002/cpe.8249
Janecek MEzzati-Jivan NAzhari S(2021)Container Workload Characterization Through Host System Tracing2021 IEEE International Conference on Cloud Engineering (IC2E)10.1109/IC2E52221.2021.00015(9-19)Online publication date: Oct-2021
https://doi.org/10.1109/IC2E52221.2021.00015

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten