Abstract
This paper introduces a new methodology for detecting anomalies in time series data, with a primary application to monitoring the health of (micro-) services and cloud resources. The main novelty in our approach is that instead of modeling time series consisting of real values or vectors of real values, we model time series of probability distributions. This extension allows the technique to be applied to the common scenario where the data is generated by requests coming in to a service, which is then aggregated at a fixed temporal frequency. We show the superior accuracy of our method on synthetic and public real-world data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
A collective anomaly consists of a subset of points that deviates from the rest of the dataset even though individually each point may appear normal.
- 2.
The code is available at https://github.com/awslabs/gluon-ts/tree/distribution_ anomaly_detection/distribution_anomaly_detection.
References
Alexandrov, A., et al.: Gluonts: probabilistic time series models in python. arXiv preprint arXiv:1906.05264 (2019)
Bendre, S.: Outliers in statistical data (1994)
Caron, F., Davy, M., Doucet, A., Duflos, E., Vanheeghe, P.: Bayesian inference for linear dynamic models with dirichlet process mixtures. IEEE Trans. Signal Process. 56(1), 71–84 (2007)
Chang, Y., Kaufmann, R.K., Kim, C.S., Miller, J.I., Park, J.Y., Park, S.: Evaluating trends in time series of distributions: a spatial fingerprint of human effects on climate. J. Econom. 214(1), 274–294 (2020)
Chen, T., et al.: Mxnet: a flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv:1512.01274 (2015)
Du, M., Li, F., Zheng, G., Srikumar, V.: Deeplog: anomaly detection and diagnosis from system logs through deep learning. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 1285–1298 (2017)
Faloutsos, C., Gasthaus, J., Januschowski, T., Wang, Y.: Forecasting big time series: old and new. Proc. VLDB Endow. 11(12), 2102–2105 (2018)
Gasthaus, J., et al.: Probabilistic forecasting with spline quantile function RNNs. In: The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1901–1910 (2019)
González, J.P., San Roque, A.M., Perez, E.A.: Forecasting functional time series with a new hilbertian armax model: application to electricity price forecasting. IEEE Trans. Power Syst. 33(1), 545–556 (2017)
Guha, S., Mishra, N., Roy, G., Schrijvers, O.: Robust random cut forest based anomaly detection on streams. In: International Conference on Machine Learning, pp. 2712–2721 (2016)
Hawkins, D.M.: Identification of Outliers, vol. 11. Springer, Heidelberg (1980)
Hochenbaum, J., Vallis, O.S., Kejariwal, A.: Automatic anomaly detection in the cloud via statistical learning. arXiv preprint arXiv:1704.07706 (2017)
Hyndman, R.J.: Computing and graphing highest density regions. Am. Stat. 50(2), 120–126 (1996)
Hyndman, R.J., Ullah, M.S.: Robust forecasting of mortality and fertility rates: a functional data approach. Comput. Stat. Data Anal. 51(10), 4942–4956 (2007)
Kieu, T., Yang, B., Guo, C., Jensen, C.S.: Outlier detection for time series with recurrent autoencoder ensembles (2019)
Malhotra, P., Vig, L., Shroff, G., Agarwal, P.: Long short term memory networks for anomaly detection in time series. In: Proceedings, vol. 89, pp. 89–94. Presses universitaires de Louvain (2015)
Meng, W., et al.: Loganomaly: unsupervised detection of sequential and quantitative anomalies in unstructured logs. In: IJCAI, pp. 4739–4745 (2019)
Moayedi, H.Z., Masnadi-Shirazi, M.: Arima model for network traffic prediction and anomaly detection. In: 2008 International Symposium on Information Technology, vol. 4, pp. 1–6. IEEE (2008)
Munir, M., Siddiqui, S.A., Chattha, M.A., Dengel, A., Ahmed, S.: Fusead: unsupervised anomaly detection in streaming sensors data by fusing statistical and deep learning models. Sensors 19(11), 2451 (2019)
Park, J.Y., Qian, J.: Functional regression of continuous state distributions. J. Econom. 167(2), 397–412 (2012)
Ren, H., et al.: Time-series anomaly detection service at microsoft. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3009–3017 (2019)
Rodriguez, A., Ter Horst, E., et al.: Bayesian dynamic density estimation. Bayesian Anal. 3(2), 339–365 (2008)
Salinas, D., Flunkert, V., Gasthaus, J., Januschowski, T.: Deepar: probabilistic forecasting with autoregressive recurrent networks. Int. J. Forecast. 36(3), 1181–1191 (2019)
Siffer, A., Fouque, P.A., Termier, A., Largouet, C.: Anomaly detection in streams with extreme value theory. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1067–1075 (2017)
Szabó, Z., Sriperumbudur, B.K., Póczos, B., Gretton, A.: Learning theory for distribution regression. J. Mach. Learn. Res. 17(1), 5272–5311 (2016)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017)
Yeh, C.C.M., et al.: Matrix profile i: all pairs similarity joins for time series: a unifying view that includes motifs, discords and shapelets. In: 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 1317–1322. IEEE (2016)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Ayed, F., Stella, L., Januschowski, T., Gasthaus, J. (2021). Anomaly Detection at Scale: The Case for Deep Distributional Time Series Models. In: Hacid, H., et al. Service-Oriented Computing – ICSOC 2020 Workshops. ICSOC 2020. Lecture Notes in Computer Science(), vol 12632. Springer, Cham. https://doi.org/10.1007/978-3-030-76352-7_14
Download citation
DOI: https://doi.org/10.1007/978-3-030-76352-7_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-76351-0
Online ISBN: 978-3-030-76352-7
eBook Packages: Computer ScienceComputer Science (R0)