A Transferable Time Series Forecasting Service Using Deep Transformer Model for Online Systems
ASE '22: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering
Article No.: 4, Pages 1 - 12
Abstract
Many real-world online systems expect to forecast the future trend of software quality to better automate operational processes, optimize software resource cost and ensure software reliability. To achieve that, all kinds of time series metrics collected from online software systems are adopted to characterize and monitor the quality of software services. To meet relevant software engineers’ requirements, we focus on time series forecasting and aim to provide an event-driven and self-adaptive forecasting service. In this paper, we present TTSF-transformer, a transferable time series forecasting service using deep transformer model. TTSF-transformer normalizes multiple metric frequencies to ensure the model sharing across multi-source systems, employs a deep transformer model with Bayesian estimation to generate the predictive marginal distribution, and introduces transfer learning and incremental learning into the training process to ensure the performance of long-term prediction. We conduct experiments on real-world time series metrics from two different types of game business in Tencent®. The results show that TTSF-transformer significantly outperforms other state-of-the-art methods and is suitable for wide deployment in large online systems.
References
[1]
Adebiyi A Ariyo, Adewumi O Adewumi, and Charles K Ayo. 2014. Stock price prediction using the ARIMA model. In 2014 UKSim-AMSS 16th International Conference on Computer Modelling and Simulation. IEEE, 106–112.
[2]
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150(2020).
[3]
John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain adaptation with structural correspondence learning. In Proceedings of the 2006 conference on empirical methods in natural language processing. 120–128.
[4]
G. E. P. Box and G. M. Jenkins. 1994. Time Series Analysis: Forecasting and Control. Prentice Hall.
[5]
Yitian Chen, Yanfei Kang, Yixiong Chen, and Zizhuo Wang. 2020. Probabilistic forecasting with temporal convolutional neural network. Neurocomputing 399(2020), 491–501.
[6]
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers.
[7]
Hal Daumé III. 2009. Frustratingly easy domain adaptation. arXiv preprint arXiv:0907.1815(2009).
[8]
Chirag Deb, Fan Zhang, Junjing Yang, Siew Eang Lee, and Kwok Wei Shah. 2017. A review on time series forecasting techniques for building energy consumption. Renewable and Sustainable Energy Reviews 74 (2017), 902–924.
[9]
Lixin Duan, Dong Xu, and Ivor Wai-Hung Tsang. 2012. Domain adaptation from multiple sources: A domain-dependent regularization approach. IEEE Transactions on neural networks and learning systems 23, 3(2012), 504–518.
[10]
R Mohammdi Farsani and E Pazouki. 2021. A transformer self-attention model for time series forecasting. Journal of Electrical and Computer Engineering Innovations (JECEI) 9, 1(2021), 1–10.
[11]
Alan R Hevner, Salvatore T March, Jinsoo Park, and Sudha Ram. 2004. Design science in information systems research. MIS quarterly (2004), 75–105.
[12]
Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex Smola. 2006. Correcting sample selection bias by unlabeled data. Advances in neural information processing systems 19 (2006).
[13]
Tao Huang, Pengfei Chen, and Ruipeng Li. 2022. A Semi-Supervised VAE Based Active Anomaly Detection Framework in Multivariate Time Series for Online Systems. In Proceedings of the ACM Web Conference 2022. 1797–1806.
[14]
Rob Hyndman, Anne B Koehler, J Keith Ord, and Ralph D Snyder. 2008. Forecasting with exponential smoothing: the state space approach. Springer Science & Business Media.
[15]
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences 114, 13(2017), 3521–3526.
[16]
Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer.
[17]
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. 2019. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in Neural Information Processing Systems 32 (2019), 5243–5253.
[18]
Zhizhong Li and Derek Hoiem. 2017. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence 40, 12(2017), 2935–2947.
[19]
Bryan Lim and Stefan Zohren. 2021. Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A 379, 2194(2021), 20200209.
[20]
David Lopez-Paz and Marc’Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. Advances in neural information processing systems 30 (2017).
[21]
Minghua Ma, Shenglin Zhang, Dan Pei, Xin Huang, and Hongwei Dai. 2018. Robust and rapid adaption for concept drift in software system anomaly detection. In 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 13–24.
[22]
Yemeserach Mekonnen, Srikanth Namuduri, Lamar Burton, Arif Sarwat, and Shekhar Bhansali. 2019. Machine learning techniques in wireless sensor network based precision agriculture. Journal of the Electrochemical Society 167, 3 (2019), 037522.
[23]
Manfred Mudelsee. 2019. Trend analysis of climate time series: A review of methods. Earth-science reviews 190 (2019), 310–322.
[24]
Hoang Nguyen, Le-Minh Kieu, Tao Wen, and Chen Cai. 2018. Deep learning methods in transportation domain: a review. IET Intelligent Transport Systems 12, 9 (2018), 998–1004.
[25]
Carl Edward Rasmussen. 2003. Gaussian processes in machine learning. In Summer school on machine learning. Springer, 63–71.
[26]
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. 2017. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2001–2010.
[27]
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. 2020. DeepAR: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting 36, 3 (2020), 1181–1191.
[28]
Matthias Seeger. 2004. Gaussian processes for machine learning. International journal of neural systems 14, 02 (2004), 69–106.
[29]
Omer Berat Sezer, Mehmet Ugur Gudelek, and Ahmet Murat Ozbayoglu. 2020. Financial time series forecasting with deep learning: A systematic literature review: 2005–2019. Applied soft computing 90 (2020), 106181.
[30]
Sima Siami-Namini, Neda Tavakoli, and Akbar Siami Namin. 2018. A comparison of ARIMA and LSTM in forecasting time series. In 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA). IEEE, 1394–1401.
[31]
Ajit P Singh and Geoffrey J Gordon. 2008. Relational learning via collective matrix factorization. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. 650–658.
[32]
Masashi Sugiyama, Taiji Suzuki, Shinichi Nakajima, Hisashi Kashima, Paul von Bünau, and Motoaki Kawanabe. 2008. Direct importance estimation for covariate shift adaptation. Annals of the Institute of Statistical Mathematics 60, 4(2008), 699–746.
[33]
Sean J Taylor and Benjamin Letham. 2018. Forecasting at scale. The American Statistician 72, 1 (2018), 37–45.
[34]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
[35]
Deqing Wang, Chenwei Lu, Junjie Wu, Hongfu Liu, Wenjie Zhang, Fuzhen Zhuang, and Hui Zhang. 2019. Softly associative transfer learning for cross-domain classification. IEEE transactions on cybernetics 50, 11 (2019), 4709–4721.
[36]
Jindong Wang, Yiqiang Chen, Shuji Hao, Wenjie Feng, and Zhiqi Shen. 2017. Balanced distribution adaptation for transfer learning. In 2017 IEEE international conference on data mining (ICDM). IEEE, 1129–1134.
[37]
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768(2020).
[38]
Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering. Springer Science & Business Media.
[39]
Neo Wu, Bradley Green, Xue Ben, and Shawn O’Banion. 2020. Deep transformer models for time series forecasting: The influenza prevalence case. arXiv preprint arXiv:2001.08317(2020).
[40]
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. 2020. Connecting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 753–763.
[41]
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of AAAI.
Index Terms
- A Transferable Time Series Forecasting Service Using Deep Transformer Model for Online Systems
Recommendations
Online time-series forecasting using spiking reservoir
AbstractIoT-based automated systems require efficient online time-series analysis and forecasting and there is a growing requirement to enable such processing at the low-cost constrained edge devices. Classical approaches such as Online ...
Time series forecasting by a seasonal support vector regression model
The support vector regression (SVR) model is a novel forecasting approach and has been successfully used to solve time series problems. However, the applications of SVR models in a seasonal time series forecasting has not been widely investigated. This ...
A heuristic time-invariant model for fuzzy time series forecasting
Many forecasting models based on the concepts of fuzzy time series have been proposed in the past decades. These models have been applied to predict enrollments, temperature, crop production and stock index, etc. In this paper, we present a simple ...
Comments
Information & Contributors
Information
Published In
October 2022
2006 pages
ISBN:9781450394758
DOI:10.1145/3551349
Copyright © 2022 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 05 January 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
ASE '22
ASE '22: 37th IEEE/ACM International Conference on Automated Software Engineering
October 10 - 14, 2022
MI, Rochester, USA
Acceptance Rates
Overall Acceptance Rate 82 of 337 submissions, 24%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 305Total Downloads
- Downloads (Last 12 months)151
- Downloads (Last 6 weeks)8
Reflects downloads up to 12 Sep 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format