Detection of Anomalies in Multivariate Time Series

11 min readNov 13, 2023

Introduction

Finding anomalies in multivariate time series data is essential in a number of fields, including industrial operations, finance, and health monitoring. For example, in finance, it can be used to spot anomalous market patterns or discover fraud. Vital sign monitoring aids in the early identification of patient deterioration in the hospital setting. Similar to this, predictive maintenance and equipment failure prevention are crucial in industrial settings.

One cannot stress how important it is to detect anomalies accurately and efficiently. Identifying abnormalities fast and accurately can result in significant cost savings, increased safety, and enhanced decision-making in the modern environment where data is continuously being generated and collected. Though uncommon in most cases, anomalies can point to important, frequently unanticipated developments. The difference between preventing a disaster and responding to its aftermath can be found in the timely detection of these anomalies.

Challenges in Anomaly Detection:

Anomaly detection in multivariate time series poses particular difficulties, despite its significance. The fundamental challenge stems from anomalies’ very character as uncommon occurrences. Because of their rarity, there is frequently not enough information available to train reliable detection algorithms. Additionally, anomalies may not be visible across all variables in a multivariate environment, which makes them even more difficult to find. They might only show up in intricate and subtle patterns in certain data sets.

Furthermore, accurate and time-efficient models that can handle vast amounts of data are necessary for the practical use of anomaly detection. This becomes more difficult in situations when data streams are high-dimensional and dynamic. Because these data are dynamic, models must be able to learn and change on their own over time, without the need for human intervention.

Given these difficulties, conventional anomaly detection techniques frequently fail. Their inability to identify anomalies completely or their propensity to generate excessive false alarms render them unsuitable for use in practical scenarios. Thus, novel methods that can effectively identify anomalies in multivariate time series despite the limitations of data sparsity and complexity are desperately needed.

In this study, we suggest a new ensemble method for multivariate time series anomaly detection. By combining feature bagging, nested rotations, and a semi-supervised learning strategy, this method tackles the issues raised above. Our approach guarantees the applicability of anomaly detection in large-scale, real-world scenarios while at the same time improving its accuracy.

Related Work

Review of Existing Techniques:

Numerous approaches have been developed as a result of years of intensive research on anomaly detection. These techniques can be broadly divided into three categories: hybrid approaches, machine learning-based techniques, and statistical techniques.

Statistical Methods: For many years, the foundation of anomaly detection has been established by using conventional statistical methods like clustering techniques, regression models, and control charts. Although these techniques work well in some situations, they frequently make a lot of assumptions about the distribution of the data and might not be appropriate for complicated, high-dimensional time series data.
Machine Learning-Based Methods: The development of machine learning has led to the development of increasingly complex techniques. Among the notable ones are:
Deep Neural Networks (DNNs): By learning intricate patterns in data, DNNs — especially those using architectures like autoencoders — have demonstrated promise in the detection of anomalies. Nevertheless, they frequently call for sizable computational resources and a lot of data.
Long Short-Term Memory (LSTM) Autoencoders: These are specialized neural networks that can identify patterns in time series data by identifying dependencies. They may have trouble with high-dimensional data or subtle anomalies, but they have been successful in capturing temporal anomalies.
Convolutional Autoencoders: These are skilled at handling spatial patterns within data and make use of convolutional neural networks (CNNs). Although strong, their impact may be diminished in situations that are solely temporal or do not involve images.
Hybrid Approaches: To take advantage of both approaches’ advantages, there has been a recent trend toward combining statistical and machine learning techniques. The goal of these hybrid models is to strike a compromise between real-world data limitations and the requirement for reliable pattern recognition.

Limitations of Current Approaches:

Despite the advancements, several limitations persist in the field of anomaly detection, particularly when it comes to multivariate time series:

Handling High-Dimensional Data: A lot of current models suffer from the dimensionality curse, which makes models perform worse the more features they have.
Data Sparsity and Imbalance: While anomalies are uncommon occurrences, models may be skewed in favor of the majority class, or typical observations, as a result of this imbalance.
Subtle Anomalies: Models built to identify more pronounced anomalies frequently fail to identify anomalies that arise from small but significant deviations in a subset of features.
Adaptability and Scalability: A lot of models aren’t flexible enough to scale well with big datasets or adjust to changing data streams, which makes them less useful in real-world scenarios.

The previously mentioned drawbacks emphasize the necessity for novel strategies that can manage the complexity of multivariate time series data. In order to close these gaps, the research suggests an ensemble technique that improves anomaly detection in complex datasets by combining nested rotations, feature bagging, and a semi-supervised method.

Problem Formulation

Defining Anomalies in Multivariate Time Series:

In the realm of multivariate time series, anomalies can be broadly classified into three types based on their nature and the patterns they exhibit:

Point Anomalies: These are individual data points that show a notable departure from the overall data. When examining each variable separately in a multivariate setting, these anomalies might not be noticeable; however, they will become noticeable when examining the relationships between various variables.
Contextual Anomalies: These happen when a data point is unusual in one context (like space or time), but may not be regarded as an outlier in another. Understanding the underlying patterns and circumstances that establish what is typical for a particular context is necessary for spotting such anomalies.
Collective Anomalies: These comprise a group of data points that diverge from the typical pattern of the entire dataset. Since these anomalies might not be visible when examining individual data points, they are especially difficult to find.

Understanding these types of anomalies is crucial in formulating a detection strategy. The complexity arises when anomalies are subtle, hidden in high-dimensional data, or when they manifest only under certain conditions.

Classification Approach to Anomaly Detection:

The study treats anomaly detection as a binary classification problem, which involves identifying data points as either normal or anomalous. This approach simplifies the problem but introduces unique challenges:

Imbalanced Data: The dataset is naturally skewed, with a far larger percentage of normal data points than anomalous ones, due to the rarity of anomalies. The performance of classification algorithms may be biased in favor of the majority class due to this imbalance.
Feature Selection and Reduction: Not every feature in a multivariate setting will be as useful in identifying anomalies as others. A crucial component is determining which features are most pertinent or minimizing the dimensionality of the data without sacrificing important information.
Temporal Dependencies: Temporal dependencies are a common feature of multivariate time series data, which means that one state may depend on another. For the model to effectively discriminate between normal and anomalous patterns, these dependencies must be taken into consideration.
Model Complexity: It is necessary to weigh the model’s computational efficiency and interpretability against its complexity. Although extremely complicated models can be more accurate, they can also be more computationally demanding and challenging to understand.

The study’s suggested solution seeks to capitalize on the advantages of feature bagging, ensemble learning, and semi-supervised techniques in order to address these issues. This formulation paves the way for the introduction of an anomaly detection technique for multivariate time series data that is more resilient and flexible.

Proposed Approach

Feature Bagging:

Concept: In feature bagging, different subsets of the original dataset’s features are created, and each subset is then used to train a different detection model. The idea behind this method is that distinct feature subsets can offer a variety of insights into the data, which could enhance the overall detection of anomalies.
Application: Feature bagging is used in this study to create a variety of models that each concentrate on a different aspect of the data. The possibility of finding anomalies that might be noticeable in some feature subsets but not in others is increased by this diversity.
Image Suggestion: A figure that describes the feature bagging procedure, breaking down the original dataset into several subsets and demonstrating how each subset is used to train a distinct model.

Nested Rotations:

Concept: To optimize the variance of the features, data is transformed using Principal Component Analysis (PCA) in nested rotations. Through this transformation, the data may reveal hidden structures that are more suggestive of anomalies.
Application: The data is preprocessed by the study using nested rotations before being fed into the anomaly detection models. By doing this, the models become more adept at spotting minute irregularities in the multivariate time series.
Image Suggestion: A graphical depiction of the PCA procedure that displays the initial feature space and the feature space that has been altered by nested rotations.

Ensemble Technique:

Concept: To arrive at a final prediction, the ensemble technique combines the results of several models. By utilizing each model’s advantages and minimizing its disadvantages, this strategy improves performance as a whole.
Application: The ensemble technique is used in this study to combine the output from models trained on various feature subsets (produced by feature bagging). An anomaly detection system that is more reliable and accurate is the result of this aggregation.
Image Suggestion: The proposed image is an infographic that illustrates the ensemble process, which combines the outputs of various models to generate a final anomaly detection result.

Semi-Supervised Approach:

Concept: Labeled and unlabeled data are used in combination for training in a semi-supervised approach. This is especially helpful in situations where it is difficult to obtain labeled data, particularly for anomalies.
Application: To efficiently integrate the outputs of the base models, the study uses a semi-supervised approach. Even with limited labeled anomaly data, the best way to aggregate these outputs is discovered through the application of logistic regression.
Image Suggestion: A flowchart illustrating the steps involved in semi-supervised learning, including the use of both labeled and unlabeled data during the training stage.

Experimental Evaluation

Experimental Setup

Objective: The main aim of the experiments is to assess the efficacy of the suggested anomaly detection method in a multivariate time series setting that is representative of real life.
Data Sources: The Skoltech Anomaly Benchmark dataset, which is extensive and indicative of actual situations in a range of industries, is used in this study.
Methodologies: Preprocessing the data with nested rotations, applying feature bagging to generate a variety of subsets, training several models on these subsets, and finally combining the results with the suggested semi-supervised ensemble method are the techniques.
Image Suggestion: A schematic diagram or flowchart that illustrates the experimental setup, from data preprocessing to model training and aggregation, is a suggested image.

Results and Performance Metrics:

Performance Metrics: The efficacy of the suggested approach is assessed using critical metrics like accuracy, precision, recall, and F1-score.
Comparison with Baseline Models: To show the improvements, the outcomes of the suggested method are compared with baseline models like autoencoders and standard neural networks.
Statistical Analysis: To make sure that improvements in performance metrics are real and not the result of chance variations in the data, statistical tests are carried out.
Image Suggestion: Bar charts or graphs that illustrate the performance metrics of the suggested approach in relation to the baseline models. A different chart might show how significant the results are statistically.

Discussion of Results:

Effectiveness of the Method: The findings show that the suggested ensemble technique is effective in identifying irregularities in multivariate time series, particularly in situations where the anomalies are subtle or the result of intricate feature interactions.
Advantages Over Baseline Methods: The accuracy and capacity to identify minute abnormalities are two areas where the method significantly outperforms baseline models.
Limitations and Challenges: To present a fair assessment, all limitations noted throughout the experiments — such as problems with scalability or computational complexity — are addressed.
Image Suggestion: A heatmap or confusion matrix that displays the true positives, true negatives, false positives, and false negatives can be used as an image suggestion to help visualize the classification performance.

Real-World Implications:

Practical Applications: These results’ real-world implications are explored, showing how this approach could be used for efficient anomaly detection across a range of industries.
Scalability and Adaptability: The method’s ability to scale to larger datasets and adapt to various time series data types is also investigated.
Image Suggestion: An infographic or case study illustration showing how the technique is actually used in the real world, like financial fraud detection or industrial equipment monitoring.

Conclusions and Next Steps

Summary of Key Findings:

Effectiveness of the Proposed Method: The study showed that the suggested ensemble technique is efficient at identifying irregularities in multivariate time series data. Nested rotations, feature bagging, and a semi-supervised learning strategy combined to produce notable gains over conventional techniques.
Advantages Over Existing Techniques: The analysis emphasized the suggested method’s benefits for managing high-dimensional data, decreasing false positives, and enhancing detection sensitivity, particularly when dealing with mild abnormalities.

Contributions to the Field:

Innovative Approach: This study presented a novel method for addressing the difficulties associated with multivariate time series anomaly detection by combining a number of approaches.
Bridging Gaps in Existing Methods: The study effectively tackled a number of shortcomings in current approaches, such as problems with imbalance, data sparsity, and the complexity of high-dimensional datasets.

Future Research Directions:

Algorithmic Improvements: Future studies could look into ways to make the algorithm even better, like maximizing feature selection or raising the effectiveness of nested rotations.
Scalability and Real-Time Processing: Improvements in the method’s scalability for bigger datasets and its adaptation for real-time anomaly detection require further work.
Application to Different Domains: There is a wealth of opportunities for further research because the suggested method can be applied to a variety of domains, including healthcare, finance, and industrial process control.
Deep Learning Integration: Examining the assimilation of deep learning methodologies with the suggested group methodology may yield additional enhancements in the precision and effectiveness of identification.

Potential Applications:

Industry-Specific Applications: The technique can be applied to a wide range of industries, including fraud detection, complex system monitoring, and predictive maintenance.
Integration with IoT and Big Data Platforms: To improve monitoring and decision-making in smart cities, manufacturing, and other industries, the strategy could be integrated with IoT and big data platforms.

Challenges and Limitations:

Addressing Computational Complexity: Although the suggested approach works well, its computational complexity might be a hindrance in some situations. The goal of future research should be to maximize computational efficiency.
Generalization across Datasets: To validate the method’s efficacy in a variety of scenarios, more research should look into how generalizable it is across various time series dataset types.

References

Iliopoulos, A., Violos, J., Diou, C., & Varlamis, I. (2023). “Detection of Anomalies in Multivariate Time Series Using Ensemble Techniques.” [Online]. Available: https://arxiv.org/abs/2308.03171
M. Braei and S. Wagner, “Anomaly detection in univariate time-series: A survey on the state-of-the-art,” arXiv preprint arXiv:2004.00433, 2020.
A. B. Nassif, M. A. Talib, Q. Nasir, and F. M. Dakalbab, “Machine Learning for Anomaly Detection: A Systematic Review,” IEEE Access, vol. 9, pp. 78658–78700, 2021.
G. Mamalakis, C. Diou, A. L. Symeonidis, and L. Georgiadis, “Of daemons and men: reducing the false positive rate in intrusion detection systems with file system footprint analysis,” Neural Computing and Applications, vol. 31, pp. 7755–7767, 2019.
I. Ullah and Q. H. Mahmoud, “Design and Development of RNN Anomaly Detection Model for IoT Networks,” IEEE Access, vol. 10, pp. 62722–62750, 2022.
P. Gogoi, D. K. Bhattacharyya, B. Borah, and J. K. Kalita, “A survey of outlier detection methods in network anomaly identification,” The Computer Journal, vol. 54, no. 4, pp. 570–588, 2011.