Article

Eigenspace-based anomaly detection in computer systems

Authors:

Hisashi KASHIMAAuthors Info & Claims

KDD '04: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining

Pages 440 - 449

https://doi.org/10.1145/1014052.1014102

Published: 22 August 2004 Publication History

Abstract

We report on an automated runtime anomaly detection method at the application layer of multi-node computer systems. Although several network management systems are available in the market, none of them have sufficient capabilities to detect faults in multi-tier Web-based systems with redundancy. We model a Web-based system as a weighted graph, where each node represents a "service" and each edge represents a dependency between services. Since the edge weights vary greatly over time, the problem we address is that of anomaly detection from a time sequence of graphs.In our method, we first extract a feature vector from the adjacency matrix that represents the activities of all of the services. The heart of our method is to use the principal eigenvector of the eigenclusters of the graph. Then we derive a probability distribution for an anomaly measure defined for a time-series of directional data derived from the graph sequence. Given a critical probability, the threshold value is adaptively updated using a novel online algorithm.We demonstrate that a fault in a Web application can be automatically detected and the faulty services are identified without using detailed knowledge of the behavior of the system.

References

[1]

A. Banerjee, I. Dhillon, J. Ghosh, and S. Sra. Generative model-based clustering of directional data. In Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 19--28, 2003.

Digital Library

[2]

P. Barford, J. Kline, D. Plonka, and A. Ron. A signal analysis of network traffic anomalies. In Proceedings of the Second ACM SIGCOMM Workshop on Internet Measurment, pages 71--82, 2002.

Digital Library

[3]

A. Berman and R. J. Plemmons. Nonnegative Matrices in the Mathematical Sciences, volume 9 of Classics in applied mathematics. SIAM, 1994.

[4]

I. S. Dhillon. Co-clustering documents and words using bipartite spectral graph partitioning. In Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 269--274, 2001.

Digital Library

[5]

C. H. Q. Ding, X. He, and H. Zha. A spectral method to separate disconnected and nearly-disconnected web graph components. In Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 275--280, 2001.

Digital Library

[6]

A. G. Ganek and T. A. Corbi. The dawning of the autonomic computing era. IBM Systems Journal, 42(1):5--18, 2003.

Digital Library

[7]

M. Gupta, A. Neogi, M. K. Agarwal, and G. Kar. Discovering dynamic dependencies in enterprise environments for problem determination. In Proceedings of 14th IFIP/IEEE Workshop on Distributed Systems: Operations and Management, pages 221--233, 2003.

[8]

H. Hajji. Baselining network traffic and online faults detection. In Proceedings of IEEE International Conference on Communications, volume 1, pages 301--308, 2003.

[9]

J. Hopcroft, O. Khan, B. Kulis, and B. Selman. Natural communities in large linked networks. In Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 541--546, 2003.

Digital Library

[10]

J. Huan, W. Wang, and J. Prins. Efficient mining of frequent subgraphs in the presence of isomorphism. In Proceedings of the Third IEEE International Conference on Data Mining, pages 549--552, 2003.

Digital Library

[11]

IBM. Trade3; http://www-306.ibm.com/software/webservers/appserv/benchmark3.html.

[12]

A. Inokuchi and H. Kashima. Mining significant pairs of patterns from graph structures with class labels. In Proceedings of the Third IEEE International Conference on Data Mining, pages 83--90, 2003.

Digital Library

[13]

A. Inokuchi, T. Washio, and H. Motoda. Complete mining of frequent patterns from graphs: Mining graph data. Machine Learning, 50:321--354, 2003.

Digital Library

[14]

M. Kuramochi and G. Karypis. Discovering frequent geometric subgraphs. In Proceedings of the Second IEEE International Conference on Data Mining, pages 258--265, 2002.

Digital Library

[15]

A. Y. Ng, A. X. Zheng, and M. I. Jordan. Link analysis, eigenvectors and stability. In Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence, pages 903--910, 2001.

Digital Library

[16]

C. Noble and D. Cook. Graph-based anomaly detection. In Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 631--636, 2003.

Digital Library

[17]

S. Sarkar and K. Boyer. Quantitative measures for change based on feature organization: Eigenvalues and eigenvectors. Computer Vision and Image Understanding, 71:110--136, 1998.

Digital Library

[18]

G. Strang. Linear Algebra and its Applications. Academic Press, 1976.

[19]

The Open Group. Application response measurement --- ARM; http://www.opengroup.org/tech/management/arm/.

[20]

M. Thottan and C. Ji. Anomaly detection in IP networks. IEEE Transactions on Signal Processing, 51(8):2191-- 2204, 2003.

Digital Library

[21]

H. Wang, D. Zhang, and K. G.Shin. Detecting SYN flooding attacks. In Proceedings IEEE INFOCOM 2002, pages 1530 --1539, 2002.

[22]

T. Washio and H. Motoda. State of the art of graph-based data mining. In SIGKDD Explorations Special Issue on Multi-Relational Data Mining, volume 5, pages 59--68, 2003.

Digital Library

[23]

K. Yamanishi and J. Takeuchi. A unifying framework for detecting outliers and change points from non-stationary time series data. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 676--681, 2002.

Digital Library

[24]

K. Yamanishi, J. Takeuchi, G. Williams, and P. Milne. On-line unsupervised outlier detection using finite mixtures with discounting learning algorithms. In Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 320--324, 2000.

Digital Library

Cited By

Wang RXi LZhang FFan HYu XLiu LYu SLeung V(2025)Context Correlation Discrepancy Analysis for Graph Anomaly DetectionIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.348837537:1(174-187)Online publication date: Jan-2025
https://doi.org/10.1109/TKDE.2024.3488375
Hsieh KWong MSegarra SMani SEberl TPanasyuk ANetravali RChandra RKandula SVanbever LZhang I(2024)NetVigilProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691922(1771-1789)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691922
Panda SFitzgerald BHazra B(2024) Real-time anomaly detection of the stochastically excited systems on spherical ( ) manifold Probabilistic Engineering Mechanics10.1016/j.probengmech.2024.10368978(103689)Online publication date: Oct-2024
https://doi.org/10.1016/j.probengmech.2024.103689
Show More Cited By

Index Terms

Eigenspace-based anomaly detection in computer systems

Recommendations

Graph-based anomaly detection
KDD '03: Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining

Anomaly detection is an area that has received much attention in recent years. It has a wide variety of applications, including fraud detection and network intrusion detection. A good deal of research has been performed in this area, often using strings ...
Anomaly Detection in Embedded Systems
Special issue on fault-tolerant embedded systems

By employing fault tolerance, embedded systems can withstand both intentional and unintentional faults. Many fault-tolerance mechanisms are invoked only after a fault has been detected by whatever fault-detection mechanism is used, hence, the process of ...
Maximal and minimal entry in the principal eigenvector for the distance matrix of a graph

Let G=(V,E) be a simple, connected and undirected graph with vertex set V(G) and edge set E(G). Also let D(G) be the distance matrix of a graph G (Janezic et al., 2007) [13]. Here we obtain Nordhaus-Gaddum-type result for the spectral radius of distance ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '04: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining

August 2004

874 pages

ISBN:1581138881

DOI:10.1145/1014052

General Chairs:
Won Kim
Cyber Database Solutions
,
Ronny Kohavi
Amazon.com
,
Program Chairs:
Johannes Gehrke
Cornell University
,
William DuMouchel
AT&T Labs Research

Copyright © 2004 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 August 2004

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

KDD04

Sponsor:

KDD04: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 22 - 25, 2004

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '25

Sponsor:
sigkdd
sigkdd

The 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 3 - 7, 2025

Toronto , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

129
Total Citations
View Citations
1,826
Total Downloads

Downloads (Last 12 months)59
Downloads (Last 6 weeks)9

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang RXi LZhang FFan HYu XLiu LYu SLeung V(2025)Context Correlation Discrepancy Analysis for Graph Anomaly DetectionIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.348837537:1(174-187)Online publication date: Jan-2025
https://doi.org/10.1109/TKDE.2024.3488375
Hsieh KWong MSegarra SMani SEberl TPanasyuk ANetravali RChandra RKandula SVanbever LZhang I(2024)NetVigilProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691922(1771-1789)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691922
Panda SFitzgerald BHazra B(2024) Real-time anomaly detection of the stochastically excited systems on spherical ( ) manifold Probabilistic Engineering Mechanics10.1016/j.probengmech.2024.10368978(103689)Online publication date: Oct-2024
https://doi.org/10.1016/j.probengmech.2024.103689
Alshlahy WRhouma D(2024)Detection of misbehaving individuals in social networks using overlapping communities and machine learningJournal of King Saud University - Computer and Information Sciences10.1016/j.jksuci.2024.102110(102110)Online publication date: Jul-2024
https://doi.org/10.1016/j.jksuci.2024.102110
Chen GArroyo JAthreya ACape JVogelstein JPark YWhite CLarson JYang WPriebe C(2024)Multiple network embedding for anomaly detection in time series of graphsComputational Statistics & Data Analysis10.1016/j.csda.2024.108070(108070)Online publication date: Oct-2024
https://doi.org/10.1016/j.csda.2024.108070
Riza LPutra ZZain MTrihutama FUtama JSamah KHerdiwijaya DNQZ RMumpuni EPriyatikanto R(2024)Real-time anomaly detection in sky quality meter data using probabilistic exponential weighted moving averageInternational Journal of Data Science and Analytics10.1007/s41060-024-00535-8Online publication date: 20-Apr-2024
https://doi.org/10.1007/s41060-024-00535-8
Liu SMa SChen HCui LDing J(2024)Combining KNN with AutoEncoder for Outlier DetectionJournal of Computer Science and Technology10.1007/s11390-023-2403-y39:5(1153-1166)Online publication date: 5-Dec-2024
https://doi.org/10.1007/s11390-023-2403-y
Fukushima SYamanishi K(2023)Balancing Summarization and Change Detection in Graph Streams2023 IEEE International Conference on Data Mining (ICDM)10.1109/ICDM58522.2023.00118(1025-1030)Online publication date: 1-Dec-2023
https://doi.org/10.1109/ICDM58522.2023.00118
Gong YDong XZhang JChen M(2023)Latent evolution model for change point detection in time-varying networksInformation Sciences10.1016/j.ins.2023.119376646(119376)Online publication date: Oct-2023
https://doi.org/10.1016/j.ins.2023.119376
Yamanishi KYamanishi K(2023)MDL Change DetectionLearning with the Minimum Description Length Principle10.1007/978-981-99-1790-7_6(209-263)Online publication date: 15-Sep-2023
https://doi.org/10.1007/978-981-99-1790-7_6
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten