research-article

Spatio-temporal decomposition, clustering and identification for alert detection in system logs

Authors:

A. Nur Zincir-Heywood,

Evangelos E. Milios,

Markus LatzelAuthors Info & Claims

SAC '12: Proceedings of the 27th Annual ACM Symposium on Applied Computing

Pages 621 - 628

https://doi.org/10.1145/2245276.2245395

Published: 26 March 2012 Publication History

Abstract

In this work, we propose an approach based on analyzing the spatio-temporal partitions of a system log, generated by supercomputers consisting of several nodes, for alert detection without employing semantic analysis. In this case, "Spatial" refers to the source of the log event and "Temporal" refers to the time the log event was reported. Our research shows that these spatio-temporal partitions can be clustered to separate normal activity from anomalous activity, with high accuracy. Therefore, our proposed method provides an effective alert detection mechanism.

References

[1]

M. Aharon, G. Barash, I. Cohen, and E. Mordechai. One Graph Is Worth a Thousand Logs: Uncovering Hidden Structures in Massive System Event Logs. Lecture Notes in Computer Science, 5781/2009: 227--243, 2009.

Digital Library

[2]

Q. Fu, J.-G. Lou, Y. Wang, and J. Li. Execution anomaly detection in distributed systems through unstructured log analysis. In Data Mining, 2009. ICDM '09. Ninth IEEE International Conference on, pages 149--158, December 2009.

Digital Library

[3]

L. Huang, X. Ke, K. Wong, and S. Mankovskii. Symptom-based problem determination using log data abstraction. In Proceedings of the 2010 Conference of the Center for Advanced Studies on Collaborative Research, CASCON '10, pages 313--326, New York, NY, USA, 2010. ACM.

Digital Library

[4]

K. Julisch. Clustering Intrusion Detection Alarms to Support Root Cause Analysis. ACM Transactions on Information and System Security, 6(4): 443--471, November 2003.

Digital Library

[5]

M. I. Krzywinski, J. E. Schein, I. Birol, J. Connors, R. Gascoyne, D. Horsman, S. J. Jones, and M. A. Marra. Circos: An information aesthetic for comparative genomics. Genome Research, 2009.

[6]

T. Li, F. Liang, S. Ma, and W. Peng. An Integrated Framework on Mining Log Files for Computing System Management. In Proceedings of of ACM KDD 2005, pages 776--781, 2005.

Digital Library

[7]

C. Lim, N. Singh, and S. Yajnik. A Log Mining Approach to Failure Analysis of Enterprise Telephony Systems. In Proceedings of The 38th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN 2008), June 2008.

[8]

A. Makanju, A. N. Zincir-Heywood, and E. E. Milios. System State Discovery via Information Content Clustering of System Logs. In Proceedings of the 2011 International Conference on Availability, Reliability and Security, ARES 2011. Vienna, Austria., August 2011.

Digital Library

[9]

A. Oliner, A. Aiken, and J. Stearley. Alert Detection in System Logs. In Proceedings of the International Conference on Data Mining (ICDM). Pisa, Italy., pages 959--964, Los Alamitos, CA, USA, 2008. IEEE Computer Society.

Digital Library

[10]

A. Oliner and J. Stearley. What Supercomputers say: A Study of Five System Logs. In 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, 2007 (DSN '07), pages 575--584, June 2007.

Digital Library

[11]

W. Peng, C. Perng, T. Li, and H. Wang. Event summarization for system management. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD '07, pages 1028--1032, New York, NY, USA, 2007. ACM.

Digital Library

[12]

J. E. Prewett. Analyzing Cluster Log Files using Logsurfer. In Proceedings of the 4th Annual Conference on Linux Clusters, 2003.

[13]

S. Sabato, E. Yom-Tov, A. Tsherniak, and S. Rosset. Analyzing System Logs: A New View of What's Important. In Proceedings of the 2nd USENIX Workshop on Tackling Computer Systems Problems with Machine Learning Techniques, pages 6: 1--6: 7, Berkeley, CA, USA, 2007. USENIX Association.

Digital Library

[14]

R. Sterritt. Towards autonomic computing: effective event management. In Software Engineering Workshop, 2002. Proceedings. 27th Annual NASA Goddard/IEEE, pages 40--47, Dec. 2002.

Digital Library

[15]

USENIX. USENIX - The Computer Failure Data Repository. Published to the web. Last Accessed October 2011.

[16]

R. Vaarandi. A Breadth-First Algorithm for Mining Frequent Patterns from Event Logs. In Proceedings of the 2004 IFIP International Conference on Intelligence in Communication Systems (LNCS), volume 3283, pages 293--308, 2004.

[17]

W. Xu. Detecting Large Scale System Problems by Mining Console Logs. PhD thesis, University of California, Berkeley, 2010.

Digital Library

Cited By

Qi JLuan ZHuang SFung CYang HLi HZhu DQian D(2023)LogEncoder: Log-Based Contrastive Representation Learning for Anomaly DetectionIEEE Transactions on Network and Service Management10.1109/TNSM.2023.323952220:2(1378-1391)Online publication date: Jun-2023
https://doi.org/10.1109/TNSM.2023.3239522
Vaarandi RBlumbergs BKont M(2018)An unsupervised framework for detecting anomalous messages from syslog log filesNOMS 2018 - 2018 IEEE/IFIP Network Operations and Management Symposium10.1109/NOMS.2018.8406283(1-6)Online publication date: Apr-2018
https://doi.org/10.1109/NOMS.2018.8406283
Makanju AZincir-Heywood AMilios E(2012)Interactive learning of alert signatures in High Performance Cluster system logs2012 IEEE Network Operations and Management Symposium10.1109/NOMS.2012.6211882(52-60)Online publication date: Apr-2012
https://doi.org/10.1109/NOMS.2012.6211882

Index Terms

Spatio-temporal decomposition, clustering and identification for alert detection in system logs
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Unsupervised learning
        Cluster analysis

Recommendations

Clustering event logs using iterative partitioning
KDD '09: Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining

The importance of event logs, as a source of information in systems and network management cannot be overemphasized. With the ever increasing size and complexity of today's event logs, the task of analyzing event logs has become cumbersome to carry out ...
A Lightweight Algorithm for Message Type Extraction in System Application Logs

Message type or message cluster extraction is an important task in the analysis of system logs in computer networks. Defining these message types automatically facilitates the automatic analysis of system logs. When the message types that exist in a log ...
Storage and retrieval of system log events using a structured schema based on message type transformation
SAC '11: Proceedings of the 2011 ACM Symposium on Applied Computing

Message types are semantic groupings of the free form messages in system log events. The message types that exist in a log file, if known, can be used in several log management and analysis tasks. In this work, we explore the use of message types as a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SAC '12: Proceedings of the 27th Annual ACM Symposium on Applied Computing

March 2012

2179 pages

ISBN:9781450308571

DOI:10.1145/2245276

Conference Chairs:
Sascha Ossowski
University Rey Juan Carlos, Spain
,
Paola Lecca
The Microsoft Research - University of Trento COSBI, Italy

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAPP: ACM Special Interest Group on Applied Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 March 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SAC 2012

Sponsor:

SIGAPP

SAC 2012: ACM Symposium on Applied Computing

March 26 - 30, 2012

Trento, Italy

Acceptance Rates

SAC '12 Paper Acceptance Rate 270 of 1,056 submissions, 26%;

Overall Acceptance Rate 1,650 of 6,669 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
152
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)1

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Qi JLuan ZHuang SFung CYang HLi HZhu DQian D(2023)LogEncoder: Log-Based Contrastive Representation Learning for Anomaly DetectionIEEE Transactions on Network and Service Management10.1109/TNSM.2023.323952220:2(1378-1391)Online publication date: Jun-2023
https://doi.org/10.1109/TNSM.2023.3239522
Vaarandi RBlumbergs BKont M(2018)An unsupervised framework for detecting anomalous messages from syslog log filesNOMS 2018 - 2018 IEEE/IFIP Network Operations and Management Symposium10.1109/NOMS.2018.8406283(1-6)Online publication date: Apr-2018
https://doi.org/10.1109/NOMS.2018.8406283
Makanju AZincir-Heywood AMilios E(2012)Interactive learning of alert signatures in High Performance Cluster system logs2012 IEEE Network Operations and Management Symposium10.1109/NOMS.2012.6211882(52-60)Online publication date: Apr-2012
https://doi.org/10.1109/NOMS.2012.6211882

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents