Anomaly Detection

There is often the need to update an installed Intrusion Detection System (IDS) due to new attack methods or upgraded computing environments. Since many cur- rent IDSs are constructed by manual encoding of ex- pert security knowledge, changes to IDSs are expensive and require a large amount of programming and debug- ging. We describe a data mining framework for adap-

Dendritic Cells (DCs) are innate immune system cells which have the power to activate or suppress the immune system. The behaviour of human of human DCs is abstracted to form an algorithm suitable for anomaly detection. We test this algorithm on the real-time problem of port scan detection. Our results show a significant difference in artificial DC behaviour for an

As computer systems continue to grow in scale and complexity, performance problems become common and a major concern for large-scale computing. Performance anomalies caused by application bugs, hardware or software faults, or resource contention can have great impact on system-wide performance and could lead to significant economic losses for service providers. While many detection methods have been presented in the past, the newly emerging challenges are detection
scalability and practical use. In this paper, we propose a scalable, non-parametric method for effectively detecting performance anomalies in large-scale systems. The design is generic for anomaly detection in a variety of parallel and distributed systems exhibiting peer-comparable property. It adopts a divide-and-conquer approach to address the scalability challenge and explores the use of non-parametric clustering and two-phase majority voting to improve detection flexibility and accuracy. We derive probabilistic models to quantitatively evaluate our decentralized design. Experiments with a suite of applications on production systems demonstrate that this method outperforms existing methods in terms of detection accuracy with a negligible runtime overhead.

Abstract This paper advocates a problem-oriented approach for the design of artificial immune systems (AIS) for data mining. By problem-oriented approach we mean that, in real-world data mining applications the design of an AIS should take into account the ...

In this whitepaper we briey describe Masibty, a novel anomaly-based web application rewall we devised. It has a modular and extensible structure. We give an overview of the anomaly detection models we im- plemented in it, and show that it is able to detect dierent kind of real world attacks on common web applications. We also evaluate its perfor- mance

In this paper we analyze the use of different types of statistical tests for the correlation of anomaly detection alerts. We show that the Granger Causality Test, one of the few proposals that can be extended to the anomaly detection domain, strongly depends on good choices of a parameter which proves to be both sensitive and difficult to estimate. We

The construction and maintenance of a medical thesaurus is a non-trivial task, due to the inherent complexity of a proper medical terminology. We present a methodology for transaction-based anomaly detection in the process of thesaurus maintenance. Our experiences are based on lexicographic work with the MorphoSaurus lexicons, which are the basis for a mono- and cross-lingual biomedical information retrieval system. Any "edit"or "delete" actions within these lexicons that undo an action defined earlier were defined as anomalous. We identify four types of such anomalies. We also analyzed to which extent the anomalous lexicon entries had been detected by an alternative, corpus-based approach.

The annual incidence of insider attacks continues to grow, and there are indications this trend will continue. While there are a number of existing tools that can accurately identify known attacks, these are reactive (as opposed to proactive) in their enforcement, and may be eluded by previously unseen, adversarial behaviors. This paper proposes an approach that combines Structural Anomaly Detection (SA) from social and information networks and Psychological Profiling (PP) of individuals. SA uses technologies including graph analysis, dynamic tracking, and machine learning to detect structural anomalies in large-scale information network data, while PP constructs dynamic psychological profiles from behavioral patterns. Threats are finally identified through a fusion and ranking of outcomes from SA and PP. The proposed approach is illustrated by applying it to a large data set from a massively multi-player online game, World of War craft (WoW). The data set contains behavior traces from over 350,000 characters observed over a period of 6 months. SA is used to predict if and when characters quit their guild (a player association with similarities to a club or workgroup in non-gaming contexts), possibly causing damage to these social groups. PP serves to estimate the five-factor personality model for all characters. Both threads show good results on the gaming data set and thus validate the proposed approach.

Anomaly detection is one of the major areas of research with the tremendous development of computer networks. Any intrusion detection model designed should have the ability to visualize high dimensional data with high processing and accurate detection rate. Integrated Intrusion detection models combine the advantage of low false positive rate and shorter detection time. Hence this paper proposes an anomaly detection model by deploying consistency based feature selection, J48 decision tree and self organizing map (SOM). Experimental analysis has been carried on KDD99 data set and each of the features selected using the integrated mechanism has been able to identify the attacks in the data set.

Relational reasoning, which has been defined as the ability to discern meaningful patterns within any informational stream, is a foundational cognitive ability associated with education, including in scientific domains. This study entailed the analysis of instructional conversations in which an attending clinical neurologist and his team of residents made diagnostic and therapeutic decisions about actual patients in a hospital setting. The primary goal was to investigate the role of 4 manifestations of relational reasoning (i.e., analogy, anomaly, antinomy, and antithesis) in medical education and diagnostic and therapeutic decision making. Results indicated that the degree to which members of the medical team used the 4 forms of relational reasoning depended on their role and expertise, as well as the time point in the problem-solving process. Specific reasoning patterns that emerged in the discourse and a prototypical model of the reasoning process are described and implications for research and practice are considered.
Keywords: relational reasoning, medical education, discourse analysis, apprenticeship model

In binary classification there are two types of errors, and in many applications these may have very different costs. We consider two learning frameworks that address this issue: minimax classification, where we seek to minimize the maximum of the false alarm and miss rates, and Neyman-Pearson (NP) classification, where we seek to minimize the miss rate while ensuring the false alarm rate is less than a specified level α. We show that our approach, based on cost-sensitive support vector machines, significantly outperforms methods typically used in practice. Our results also illustrate the importance of heuristics for improving the accuracy of error rate estimation in this setting. We then reduce anomaly detection to NP classification by considering a second class of points, allowing us to estimate minimum volume sets using algorithms for NP classification. Comparing this approach with traditional one-class methods, we find that our approach has several advantages.

Attacks, such as port scans, DDoS and worms, threaten the functionality and reliability of IP networks. Early and accurate detection is crucial to mitigate their impact. We use the Method of Remaining Elements (MRE) to detect anomalies based on the characterization of traffic features through a proportional uncertainty measure. MRE has the functionality and performance to detect abnormal behavior and

Due to the increasing deployment of vehicles in human societies and the necessity for smart traffic control, anomaly detection is among the various tasks widely employed in traffic monitoring. As the issue of urban traffic and their relative smart monitoring systems have gained popularity among researchers in recent years, there exist several studies in this regard. In most of these studies, classification is performed based on the behavior of drivers, where a set of default trajectories are used in order to learn the system and classify the related data. However, two under-studied challenges are the lack of access to sufficient data to provide an efficient model, along with the lack of access to anomaly data that covers all possible abnormal trajectories. While the former challenge can be tackled through long-term data recording, the latter requires appropriate considerations. To this aim, we have utilized a combination of optimized convolutional neural network and fuzzy neural network classifiers, along with autoencoding neural networks. The final combination occurs at the decision level. First, the CNN-ANFIS classifier assigns the input trajectory to one of the predefined categories. Then, the trained autoencoder networks examine the result in order to find whether the trajectory is normal or abnormal. Obtaining 87.5% accuracy on QMUL and 99.5% on the T15 datasets confirms the superior performance of the proposed method.

Uncertain data management, querying and mining have become important because the majority of real world data is accompanied with uncertainty these days. Uncertainty in data is often caused by the deficiency in underlying data collecting equipments or sometimes manually introduced to preserve data privacy. The uncertainty information in the data is useful and can be used to improve the quality of the underlying results. Therefore in this dissertation, three problems are being solved related to outlier detection on uncertain data. 1) Distancebased outlier detection on uncertain data: In this research, we give a novel definition of distance-based outliers on uncertain data. Since the distance probability computation is expensive, a cell-based approach is proposed to index the dataset objects and to speed up the outlier detection process. The cell-based approach identifies and prunes the cells containing only inliers based on its bounds on outlier score (#D-neighbors). Similarly it can ...

... Japkowicz [6] used an autoencoder neural network to detect faults in gearboxes. ... Several types of neural networks have been proposed for novelty detection including Autoassociators [6 ... A control system was created using a National Instruments PXI chassis and analogue input ...

... Arshad Ali, Modood Ahmad Khan, S. Azam H. Bukhari and Waqar Mahmood1 drarshad@niit. edu.pk, 55Modood@niit.edu.pk, 55azam@niit.edu ... Cottrell, Connie Logg, Jiri Nivartili, William Jerrod of SLAC-Stanford, Ejaz Ahmed, Zaheer A. Khan, Fawad Nazir and Sajjad Haider of ...

Techniques for anomaly detection in the maritime domain are developed in this thesis using an area metric that measures the degree of similarity, or distinction, between ships’ tracks using the area between ships’ tracks. A modified k-means algorithm is applied to the areas calculated between ships tracks to extract sea lanes and classify these ships’ tracks into sea lanes. Bayes’ theorem of conditional probabilities is used to calculate the probability of ship tracks being in different sea lanes. Two types of anomaly detection are examined in this thesis. Ships not sailing in standard sea lanes are detected using Bayes’ theorem, and ships following one another are detected using user-defined thresholds on the minimum area between sea tracks. The development of the metric that compares ships’ track data using area is the most significant result of this thesis.http://archive.org/details/maritimedomainaw1094560384Navy, PakistanApproved for public release; distribution is unlimited

ABSTRACT Many approaches for anomaly detection use statistical based methods that build profiles of normality. In these cases, anomalies are defined as deviations from normal models build from representative data. Detection systems based solely on these approaches typically generate high false alarm rates due to the difficulty of creating flawless models. In order to support the comprehension, validation and update of such models, this paper is devoted to the visualization of normal behavioral models of sea traffic and their usability evaluation. First, we present geographical projections of the different probability density functions that represent the normal traffic behavior and second, we outline results from a usability assessment carried out in order to evaluate the ability of such visualizations to support representative tasks related to the establishment of normal situational picture.

ABSTRACT The utilization of uniform eddy current techniques to detect anomalies in conductive plates represents an important issue. This article presents novel uniform eddy current probe architecture with a planar excitation coil and a magnetic field sensing array of GMR magnetometer sensors. A set of PXI modular instruments, the designed and implemented uniform eddy current probe and an X-Y scanning system are part of a non-destructive testing (NDT) system used to detect, localize and characterize defects associated with the tested conductive plates. Sinusoidal excitation is applied to the planar spiral rectangular coil of the sensor while a set of GMR magnetometer sensors detects the induced magnetic field in specimens under test. The usage of the GMR magnetometer sensor array as part of the uniform eddy current probe reduces the testing time and increases the reliability of the system, avoiding defect detection failure. Considering the dispersion of the GMR sensors characteristics a practical approach concerning the magnetic biasing and appropriate image processing techniques including image filtering, background subtraction image merging, 2D correlation for the merged image and a 2D template matching are included in the paper.

Smart meters have become a core part of the Internet of Things, and its sensory network is increasing globally. For example, in the UK there are over 15 million smart meters operating across homes and businesses. One of the main advantages of the smart meter installation is the link to a reduction in carbon emissions. Research shows that, when provided with accurate and real-time energy usage readings, consumers are more likely to turn off unneeded appliances and change other behavioural patterns around the home (e.g., lighting, thermostat adjustments). In addition, the smart meter rollout results in a lessening in the number of vehicle callouts for the collection of consumption readings from analogue meters and a general promotion of renewable sources of energy supply. Capturing and mining the data from this fully maintained (and highly accurate) sensing network, provides a wealth of information for utility companies and data scientists to promote applications that can further supp...

【Abstract】Intrusion Detection System(IDS) has beenharassed by false positive and false negative problem. Common IDS using single detection mode is hard to solve this problem effectively. This paper analyzes the characteristics of network flow and presents a new method, called hybrid IDS, combining misuse detection mode and anomaly detection mode, the method can overcome the shortcomings of IDS using single mode.

The rapid progress of modern technologies generates a massive amount of highthroughput data, called Big Data, which provides opportunities to find new insights using machine learning (ML) algorithms. Big Data consist of many features (attributes). However, irrelevant features may degrade the classification performance of ML algorithms. Feature selection (FS) is a combinatorial optimisation technique used to select a subset of relevant features that represent the dataset. For example, FS is an effective preprocessing step of anomaly detection techniques in Big Cybersecurity Datasets. Evolutionary algorithms (EAs) are widely used search strategies for feature selection. A variant of EAs, called a cooperative co-evolutionary algorithm (CCEA) or simply cooperative co-evolution (CC), which uses a divide-and-conquer approach, is a good choice for large-scale optimisation problems. The goal of this thesis is to investigate and develop three key research issues related to feature selection in Big Data and anomaly detection using feature selection in Big Cybersecurity Data. The first research problem of this thesis is to investigate and develop a feature selection framework using CCEA. The objective of feature selection is twofold: selecting a suitable subset of features or in other words, reducing the number of features to decrease computations and improving classification accuracy, which are contradictory, but can be achieved using a single objective function. Using only classification accuracy as the objective function for FS, EAs, such as CCEA, achieves higher accuracy, even with a higher number of features. Hence, this thesis proposes a penalty-based wrapper single objective function. This function has been used to evaluate the FS process using CCEA, henceforth called Cooperative Co-Evolutionary Algorithm-Based Feature Selection (CCEAFS). Experimental analysis was performed using six widely used classifiers on six different datasets, with and without FS. The experimental results indicate that the proposed objective function is efficient at reducing the number of features in the final feature subset without significantly reducing classification accuracy. Furthermore, the performance results have been compared with four other state-of-the-art techniques. CC decomposes a large and complex problem into several subproblems, optimises each subproblem independently, and collaborates different subproblems only to build a complete solution of the problem. The existing decomposition solutions have poor performance because of some limitations, such as not considering feature interactions, dealing with only an even number of features, and decomposing the dataset statically. However, for real-world problems without any prior information about how the features in a dataset interact, it is difficult to find a suitable problem decomposition technique for feature selection. Hence, the second research problem of this thesis is to investigate and develop a decomposition method that can decompose Big Datasets dynamically, and can ensure the probability of grouping interacting features into the same subcomponent. Accordingly, this thesis proposes a random feature grouping (RFG) with three variants. RFG has been used in the CC-based FS process, hence called Cooperative Co-Evolution-Based Feature Selection with Random Feature Grouping (CCFSRFG). Experiment analysis performed using six widely used ML classifiers on seven different datasets, with and without FS, indicates that, in most cases, the proposed CCFSRFG-1 outperforms CCEAFS and CCFSRFG-2, and also does so when using all features. Furthermore, the performance results have been compared with five other state-of-theart techniques. Anomaly detection from Big Cybersecurity Datasets is very important; however, this is a very challenging and computationally expensive task. Feature selection in cybersecurity datasets may improve and quantify the accuracy and scalability of both supervised and unsupervised anomaly detection techniques. The third research problem of this thesis is to investigate and develop an anomaly detection approach using feature selection that can improve the anomaly detection performance, and also reduce the execution time. Accordingly, this thesis proposes an Anomaly Detection Using Feature Selection (ADUFS) to deal with this research problem. Experiments were performed on five different benchmark cybersecurity datasets, with and without feature selection, and the performance of both supervised and unsupervised anomaly detection techniques were investigated by ADUFS. The experimental results indicate that, instead of using the original dataset, a dataset with a reduced number of features yields better performance in terms of true positive rate (TPR) and false positive rate (FPR) than the existing techniques for anomaly detection. In addition, all anomaly detection techniques require less computational time when using datasets with a suitable subset of features…