Anomaly Detection for Web Log based Data: A Survey

Meena Siwach Suman Mann

Research Scholar Department of Information Technology
USICT, GGSIPU Maharaja Surajmal Institute of Technology
Maharaja Surajmal Institute of Technology Delhi, India
Delhi, India sumanmann2007@gmail.com
meenusiwach@gmail.com

Abstract— Web Log data contains crucial information such they might represent unexpected behavior that was
as User Name, IP Address, Access Request, Time Stamp, previously unknown. Data mining techniques are employed
Result Status, number of Bytes Transferred, Use Agent and when data has to be processed to uncover relationships or
URL that Referred. In post-error investigations, log files are forecast known or unknown outcomes. [6]
useful as you can use log files to figure out what's causing an
error or a security breach. This is because the log files record
data at the same time as the information system's activity. Classification, Clustering and Techniques of machine
Regularly reviewing logs may aid in the detection of harmful learning are among them. Hybrid techniques are also being
assaults on the system. This review aims to lay out a developed to improve the accuracy of anomaly detection.
comprehensive outline of the various research methods related Detecting irregular behavior or anomalies will lead to further
to anomaly detection in system logs, and to evaluate their investigation and classification into new types of incursions.
effectiveness. We have discussed the various advantages and [7]
drawbacks of different types of anomaly detection techniques.
Anomalies can be categorized into three types as shown
We also talk about the computational complexity of these
techniques in real applications.
in figure 1:
2022 IEEE Delhi Section Conference (DELCON) | 978-1-6654-5883-2/22/$31.00 ©2022 IEEE | DOI: 10.1109/DELCON54057.2022.9753130

Keywords—Anomaly Detection, Machine Learning, Deep

Learning, LSTM, RNN

I. INTRODUCTION
Anomaly detection in system logs is becoming
increasingly important for large enterprises as they are prone
to exploitation due to the increasing number of security bugs
and vulnerability in the system. A hostile action or
infiltration could be the source of the abnormality. The
analyst is interested in the anomalous behavior detected in
the dataset, and this is the most crucial aspect for anomaly
detection [1]. Outliers are observations that diverge so much
from the norm that they raise the possibility that they were
originated by a separate mechanism. Anomalies can be
generated by data errors, but they can also be indicative of a
new, previously unknown underlying procedures [4]. The
traditional log anomaly detection method is no longer useful
anymore due to the increasing complexity of the attacks.
Instead, the attacks are being analyzed by classification,
clustering and using techniques of machine learning. In
numerous applications, anomaly detection gives very
important and vital information. Data mining techniques are
employed when data needs to be processed to uncover Fig. 1. Type of Anomalies
relationships or forecast known or unknown outcomes [3].
Many hybrid techniques have been developed to find A. Point Anomaly
known and unknown assaults more precisely. This study Point anomalies are irregularities or deviations that occur
examines a variety of data mining strategies for detection of at random and may or may not have a specific meaning.
anomalies to give a better knowledge of the options available
[2]. These data mining approaches are used in anomaly B. Contextual Anomaly
detection to detect startling activity buried inside data by
It is also termed as conditional anomaly. It is a data point
increasing the likelihood of being intruded.
that can be regarded abnormal in a particular observation
context [11].
II. ANOMALY DETECTION
It is the method of identifying patterns in a dataset that C. Collective Anomaly
do not behave normally or predictably. Anomalies or These are anomalous collections of discrete data points in
outliers are terms used to describe these unusual behaviors. which each individual point appears to be normal in isolation
Anomalies are not usually classified as attacks, although but exhibits strange properties when observed in a group.

978-1-6654-5883-2/22/$31.00 ©2022 IEEE

Authorized licensed use limited to: Sharda University. Downloaded on April 26,2022 at 05:41:57 UTC from IEEE Xplore. Restrictions apply.
 III. MACHINE LEARNING AND TECHNIQUES
be supervised, semi-supervised and unsupervised as shown
It is a subset of artificial intelligence that let a system to in figure 2. In order to accelerate the learning process,
learn from concepts and ideas without having to be several advanced techniques have been proposed.
programmed. Types of learning processes can

Fig. 2. Types of Machine Learning

A. Supervised Learning C. Unsupervised Learning

In this learning, the user should first determine what kind In unsupervised learning, the algorithm doesn't have pre-
of data should be used for training purposes, before writing assigned labels or score levels for the training data. Instead, it
any training examples. For instance, in the case of must first identify all the natural patterns in the data set.
handwriting analysis, the data should contain all or a portion There are various examples where this process works, such
of a word or a sentence. The training set should be as clustering and principal component analysis. Compared to
representative of the real-life use of the function. The outputs supervised learning techniques, which usually require a large
should also be derived from various sources, such as human amount of human labor to prepare and verify the initial tags,
experts and measurements. unsupervised learning allows for a minimal amount of
A feature vector is a representation of an object, which preparation and audit to be conducted. It also allows for the
has a number of features that describe the object. The goal is exploitation of previously undetected patterns.
to provide enough data to forecast the outcome. The
algorithm may be run on the acquired training set. After IV. MACHINE LEARNING ALGORITHMS
learning and parameter adjustment, the learnt function's
performance should be tested on a different test set from the Machine learning algorithms are designed to perform
training program. [11]. tasks without being explicitly programmed. They're often
utilized in applications like anomaly detection, email
filtering, and computer vision, where developing traditional
B. Semi-supervised Learning algorithms to fulfil the required tasks is difficult or
It is a kind of learning that combines the unlabeled data impossible. This paper reviews some machine learning
of training with the labeled data of supervised learning. It is a algorithms and approaches of anomaly detection in log files.
special type of weak supervision. Unlabeled data, which can
be combined with small amounts of labeled data, can A. Decision Tree (DT)
improve a learning algorithm's accuracy by up to 30%. This
is especially true in cases where the acquisition of large, fully It is a supervised learning technique that may be used in
labeled training sets is not feasible [7]. various areas such as data mining, machine learning, and
artificial intelligence. A DT can be represented graphically as
The learning problem can be viewed as an exam that the a simple tree, and the outcome choice can be clearly
teacher uses to test her knowledge. In the transductive or described, as attributes in a group and value that the node can
inductive setting, these issues are practice problems that the take [18]. You can see in figure 3, the output of a DT graph
teacher will use in the course. after training of certain model.

Authorized licensed use limited to: Sharda University. Downloaded on April 26,2022 at 05:41:57 UTC from IEEE Xplore. Restrictions apply.
 predictive modelling, incorporates deep learning as a key
component.

Fig. 3. Representing a DT graph

B. Support Vector Machine (SVM)

It is a Machine that draws a hyperplane in high-
dimensional space. When there is no clear classifier which
can be found between the classes, SVM proceeds by moving
the data into a high dimension space. Rather than employing
positive and negative examples, the one class SVM is based Fig. 5. Deep Learning Hybrid model Architecture
on a single set of examples that belongs to a specific class
and no negative instances [14]. Deep learning is highly useful for data scientists who are
responsible with gathering, analyzing, and interpreting
massive amounts of data; it speeds up and simplifies the
C. k-Nearest Neighbors (kNN) process [12]. Here is an example, showing a DL based
kNN is a supervised machine learning algorithm that is architecture of an DT-SVM hybrid model, in figure 5.
commonly used for solving classification problems. It is also
commonly used for building recommender systems and face
DEEP DEEP LEARNING ALGORITHMS
detection applications. When compared to deep generative
models, KNN has been observed to perform better in
instances with fewer anomalies [19]. Deep learning algorithms help in improving the learning
performance and simplification of calculation process.
However, they can be very challenging to implement and
maintain in terms of long training times and high accuracy.
Here are some basic DL algorithms.

A. RNN
It is an artificial neural system that is typically built with
limited memory. They can only store and make predictions
about their outputs [2].

B. CNN
Fig. 4. Illustration of K-NN It is used for the classification's problems. It is a type of
artificial intelligence used for classification of data types
such as images and videos, with the help of the neurons and
D. Principal Component Analysis (PCA) their connectivity patterns.
It is used to for the dimensionality reduction of the large CNN works by adding layers in the imagery datasets and
size data sets to n- dimension data set. PCA works on the by also dividing the images into some layers.
principle of conversion of data without losing any significant
information of the data. PCA clusters datasets in an
unsupervised fashion C. LSTMs
Long-Term Memory networks are RNNs that can look
V. DEEP LEARNING back to long-term temporal dependencies. They can perform
Deep Learning (DL) is a recent update on Artificial various tasks related to natural language processing [5].
Neural Networks that takes use of available low-cost Bidirectional LSTM (Bi-LSTM) is an LSTM extension that
computing. They are focused with creating far larger and separates a standard LSTM's hidden layer of neurons into
more complicated neural networks, and many of the two opposed directions, forward and backward. Bi-LSTM
approaches are concerned with very large datasets of tagged may capture log sequence knowledge from both directions of
analogue data. Data science, which covers statistics and input.

Authorized licensed use limited to: Sharda University. Downloaded on April 26,2022 at 05:41:57 UTC from IEEE Xplore. Restrictions apply.
D. Auto-Encoder F. DSN
An autoencoder is an unsupervised neural network Deep Stacking Networks (DSNs) isn't like other deep
technique. It constricts the data and encodes it effectively, learning frameworks that we’ve studied before. It is made up
then reconstructs the output from encoded data that is closer of a huge size of deep separate networks, all with its own set
to the original input. It can also reconstruct the output from of hidden layers. Training, according to the DSN, is not a
the encoded data [15]. single, isolated problem, but rather a collection of unique
training problems [10].
E. DBN
DBNs are a kind of deep neural network that consists of VI. VARIOUS HYBRID APPROACHES
numerous graphical representations layers also termed as Many researchers have worked on anomaly detection in
RBMs (Restricted Boltzmann Machine). The fundamental different fields using various deep learning or machine
sensory information is represented by the input layer, while learning algorithms. They proposed that employing a single
the abstract description of this data is represented by the algorithm produces insufficient results because new threats
hidden layer. The output layer's sole purpose is to do are discovered on a regular basis, so utilizing a single method
network categorization [10]. is insufficient. In Table 1, we’ve discussed some hybrid
approaches and their pros and cons concluded by various
authors:

TABLE I. HYBRID APPROACHES BY DIFFERENT AUTHORS

Sr. No Paper Reference Authors Techniques Pros Cons

1 [13] Kamilia Menghour, Ant colony optimization (ACO) The bulk of databases When the dataset is
Labiba Souici-Meslati and produced the better results sufficiently huge, the time
Particle swarm optimization in regards of classification complexity increases.
(PSO) performance.

2 [16] Sepp Hochreiter, Jürgen Long-Short Term Memory Good performance for When using a naive bayes
Schmidhuber (LSTM) / Bidirectional-LSTM sequential form of data classifier in various contexts,
it is difficult to predict the
results.

3 [17] Peddabachigari, Support Vector Machines (SVM) Yields greater results on the When compared to SVM, this
Abraham,Grosan and and Decision Tree (DT) KDD cup dataset. method produces similar
Thomas Higher accuracy with SVM results.

4 [15] Hinton, G.E. & Autoencoder (Ae) Can be used for unbalanced As a result of failing to
Salakhutdinov, R.R. dataset with few anomalies. rebuild the anomalous data
samples, a substantial
reconstruction error is
produced.

5 [8] Farid, Harbi, and Naive bayes and Decision False positives were reduced Improvements in the False
Rahman Tree (DT) to a minimum, and balance Positive Rate are required for
detection rates were remote to user attacks.
increased.

6 [9] Fu, Song & Liu, Jianguo One-Class and Two-Class It’s a self-adaptive by Failure detection accuracy is
& Pannu, Husanbir Support Vector Machines learning from observed not possible to achieve at
failure occurrences and also, 100%.
it is independent of previous
failure history.

After analyzing these approaches as discussed in Table I, unfamiliar problems, as well as removing each other's
we can conclude that combining various machine learning flaws, which considerably improves overall accuracy in
algorithms aids in refining, generalizing, and adapting to anomaly detection.

Authorized licensed use limited to: Sharda University. Downloaded on April 26,2022 at 05:41:57 UTC from IEEE Xplore. Restrictions apply.
 VII. CONCLUSION [12] Jerone T. A. Andrews, Edward J. Morton, and Lewis D. Griffin,
"Detecting Anomalous Data Using Auto- Encoders," International
In this study, we looked at various learning techniques Journal of Machine Learning and Computing vol.6, no. 1, pp. 21-26,
that have been used to find anomalies in web log data. A 2016
comparative evaluation of issues addressed and DL strategies [13] Menghour K, Labiba Souici-Meslati: “Hybrid ACO-PSO Based
used was presented. We also summarized and compared Approaches for Feature Selection”
different Hybrid techniques and their pros and cons, [14] Tang D. H., Cao Z.,Machine “Learning-based Intrusion Detection
researched by various authors. Among all the hybrid Algorithm”; Journal of Computational Information Systems;5(6);
2009; p. 1825-1831.
approaches, the supervised deep learning models like DT and
[15] Hinton, G.E. & Salakhutdinov, R.R.. (2006). “Reducing the
SVM provided satisfactory results with higher accuracy in Dimensionality of Data with Neural Networks”. Science (New York,
detecting the anomalies using the KDD cup dataset. N.Y.). 313. 504-7. 10.1126/science.1127647.
[16] Hochreiter S., Schmidhuber J.; “Long Short-Term Memory”. Neural
Comput 1997; 9 (8): 1735–1780
REFERENCES
[17] Peddabachigari S., Abraham A., Grosan C., Thomas J., “Modeling
intrusion detection system using hybrid intelligent systems”; Journal
[1] Dokas P., . Ertoz L., Kumar V., Lazarevic A., Srivastava J., Tan P. of network and computer applications; 30(1); 2007; p. 114-132.
N., “Data mining for network intrusion detection,” In Proceedings of
[18] Batta M.. “Machine Learning Algorithms “- A Review, International
NSF Workshop on Next Generation Data Mining; 2002; p. 21-30
Journal of Science and Research (IJSR),ISSN: 2319-7064
[2] Agrawal S., Agrawal J: “Survey on Anomaly Detection using Data
[19] V´ıt ˇSkv´ara, Tom´aˇs Pevn`y, and V´aclav ˇSm´ıdl. “Are generative
Mining Techniques,” Procedia Computer Science 60 (2015) 708 –
deep models for novelty detection truly better? “arXiv preprint
713
arXiv:1807.05027, 2018.
[3] Chandola V., Banerjee A. , Kumar V., “Anomaly detection: A
[20] https://www.educba.com/deep-learning-networks/
survey,” ACM Computing Surveys (CSUR); 41(3); 2009;p. 15.
[21] Anish Batra, Guneet Singh Sethi, Suman Mann,” Personalized
[4] D. Hawkins. “Identification of Outliers”. Chapman and Hall, London,
Automation of Electrical and Electronic Devices Using Sensors and
1980.
Artificial Intelligence—the Intelligizer System” Computational
[5] “A Survey on Log Anomaly Detection using Deep Learning”, 2020 Intelligence: Theories, Applications and Future Directions - Volume
8th International Conference on Reliability, Infocom Technologies I, 2019, Volume 798.
and Optimization (Trends and Future Directions) (ICRITO) Amity
[22] Sakshi Hooda, Suman Mann, “Sepsis-Diagnosed Patients’ In-Hospital
University, Noida, India. June 4-5, 2020
Mortality Prediction Using Machine Learning: The Use Of Local Big
[6] Rashmikant Dalal K., Rele M, "Cyber Security: Threat Detection Data-Driven Technique in the Emergency Department” International
Model based on Machine learning Algorithm", Communication and journal of grid and distributed computing, vol13, Issue 1 2020.
Electronics Systems (ICCES) 2018 3rd International Conference on,
[23] Mann, Suman, et al. "Estimation of age groups using facial
pp. 239-243, 2018.
recognition features" International journal of engineering and
[7] Chalapathy R, Chawla S: “deep learning for anomaly detection: a computer science,2018 pp 23945-23951.
survey,” arXiv:1901.03407v2 [cs.LG] 23 Jan 2019
[24] Hooda, S., and S. Mann. "A Focus on the ICU’s Mortality Prediction
[8] Farid D. M., Harbi N., Rahman M. Z., “Combining naive bayes and Using a CNN-LSTM Model." International Journal of Psychosocial
decision tree for adaptive intrusion detection”; International Journal Rehabilitation 24, no. 6 (2020): 8045-8050.
of Network Security & Its Applications (IJNSA);2( 2);2010;p. 12-25.
[25] Vasu Negi, Suman Mann , Vivek Chauhan, “ Devanagari Character
[9] Fu, Song & Liu, Jianguo & Pannu, Husanbir. (2012). “A Hybrid Recognition Using Artificial Neural Network”, International Journal
Anomaly Detection Framework in Cloud Computing Using One- of Engineering and Technology, 2017, 2161-2167
Class and Two-Class Support Vector Machines”. 7713. 726-738.
[26] D. Gupta, S. K. Jha and S. Mann Maharaja Surajmal, "Internet
10.1007/978-3-642- 35527-1_600
Crimes-It's Analysis and Prevention Approaches," 2021 9th
[10] Dargan, S., Kumar, M., Ayyagari, M.R. et al. “A Survey of Deep International Conference on Reliability, Infocom Technologies and
Learning and Its Applications”: A New Paradigm to Optimization (Trends and Future Directions) (ICRITO), 2021, pp. 1-
Machine Learning. Arch ComputatMethods Eng 27, 1071–1092 4, doi: 10.1109/ICRITO51393.2021.9596396.
(2020)
[27] Suman Mann, Deepa Gupta, Yukti Arora, Shivanka Priyanka
[11] Song X, Wu M, Jermaine C, and Ranka S. “Conditional anomaly Chugh, Akash Gupta, Smart Hospitals Using Artificial Intelligence
detection.” IEEE Transactions on Knowledge and Data Engineering, and Internet of Things for COVID-19 Pandemic, chapter in Smart
19(5):631–645, 2007. Healthcare Monitoring Using IoT with 5G, 2021.

Authorized licensed use limited to: Sharda University. Downloaded on April 26,2022 at 05:41:57 UTC from IEEE Xplore. Restrictions apply.