Abstract
Anomaly detection is the process of identifying unexpected events or abnormalities in data, and it has been applied in many different areas such as system monitoring, fraud detection, healthcare, intrusion detection, etc. Providing real-time, lightweight, and proactive anomaly detection for time series with neither human intervention nor domain knowledge could be highly valuable since it reduces human effort and enables appropriate countermeasures to be undertaken before a disastrous event occurs. To our knowledge, RePAD (Real-time Proactive Anomaly Detection algorithm) is a generic approach with all above-mentioned features. To achieve real-time and lightweight detection, RePAD utilizes Long Short-Term Memory (LSTM) to detect whether or not each upcoming data point is anomalous based on short-term historical data points. However, it is unclear that how different amounts of historical data points affect the performance of RePAD. Therefore, in this paper, we investigate the impact of different amounts of historical data on RePAD by introducing a set of performance metrics that cover novel detection accuracy measures, time efficiency, readiness, and resource consumption, etc. Empirical experiments based on real-world time series datasets are conducted to evaluate RePAD in different scenarios, and the experimental results are presented and discussed.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Lee, M.-C., Lin, J.-C., Gran, E.G.: RePAD: real-time proactive anomaly detection for time series. In: Proceedings of the 34th International Conference on Advanced Information Networking and Applications (AINA 2020), pp. 1291–1302 (2020)
Hochenbaum, J., Vallis, O.S., Kejariwal, A.: Automatic anomaly detection in the cloud via statistical learning. arXiv preprint arXiv:1704.07706 (2017)
Aggarwal, C.C., Yu, P.S.: Outlier detection with uncertain data. In: Proceedings of the 2008 SIAM International Conference on Data Mining. Society for Industrial and Applied Mathematics, pp. 483–493 (2008)
Xu, J., Shelton, C.R.: Intrusion detection using continuous time Bayesian networks. J. Artif. Intell. Res. 39, 745–774 (2010)
Fisher, W.D., Camp, T.K., Krzhizhanovskaya, V.V.: Crack detection in earth dam and levee passive seismic data using support vector machines. Procedia Comput. Sci. 80, 577–586 (2016)
Wu, J., Zeng, W., Yan, F.: Hierarchical temporal memory method for time-series-based anomaly detection. Neurocomputing 273, 535–546 (2018)
Staudemeyer, R.C.: Applying long short-term memory recurrent neural networks to intrusion detection. South African Comput. J. 56(1), 136–154 (2015)
Bontemps, L., McDermott, J., Le-Khac, N.A.: Collective anomaly detection based on long short-term memory recurrent neural networks. In: International Conference on Future Data and Security Engineering, pp. 141–152. Springer, Cham, November 2016
Lavin, A., Ahmad, S.: Evaluating real-time anomaly detection algorithms – the numenta anomaly benchmark. In: 14th International Conference on Machine Learning and Applications (2015).
Xu, H., et al.: Unsupervised anomaly detection via variational auto-encoder for seasonal KPIs in web applications. In: Proceedings of the 2018 World Wide Web Conference, pp. 187–196 (2018)
numenta/NAB.: The Numenta Anomaly Benchmark [Online code repository]. https://github.com/numenta/NAB Accessed 03 Mar 2021
Ren, H., et al.: Time-series anomaly detection service at microsoft. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3009–3017 (2019)
Simula Research Laboratory: the eX3 research infrastructure. https://www.ex3.simula.no Accessed 03 Mar 2021
Lin, J.-C., Lee, M.-C.: Performance evaluation of job schedulers under hadoop YARN. Concurrency Computat. Pract. Exper. (CCPE) 28(9), 2711–2728 (2016)
Lee, M.-C., Lin, J.-C., Yahyapour, R.: Hybrid job-driven scheduling for virtual mapreduce clusters. IEEE Trans. Parallel Distrib. Syst. (TPDS) 27(6), 1687–1699 (2016)
Lee, M.-C., Lin, J.-C., Gran, E.G.: ReRe: a lightweight real-time ready-to-go anomaly detection approach for time series. In: Proceedings of the 44th IEEE Computer Society Signature Conference on Computers, Software, and Applications (COMPSAC 2020), pp. 322–327 (2020)
Acknowledgments
This work was supported by the project eX3 - Experimental Infrastructure for Exploration of Exascale Computing funded by the Research Council of Norway under contract 270053 and the scholarship under project number 80430060 supported by Norwegian University of Science and Technology.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Lee, MC., Lin, JC., Gran, E.G. (2021). How Far Should We Look Back to Achieve Effective Real-Time Time-Series Anomaly Detection?. In: Barolli, L., Woungang, I., Enokido, T. (eds) Advanced Information Networking and Applications. AINA 2021. Lecture Notes in Networks and Systems, vol 225. Springer, Cham. https://doi.org/10.1007/978-3-030-75100-5_13
Download citation
DOI: https://doi.org/10.1007/978-3-030-75100-5_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-75099-2
Online ISBN: 978-3-030-75100-5
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)