research-article

Multi-Horizon Time Series Forecasting with Temporal Attention Learning

Authors:

Heng HuangAuthors Info & Claims

KDD '19: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

Pages 2527 - 2535

https://doi.org/10.1145/3292500.3330662

Published: 25 July 2019 Publication History

Abstract

We propose a novel data-driven approach for solving multi-horizon probabilistic forecasting tasks that predicts the full distribution of a time series on future horizons. We illustrate that temporal patterns hidden in historical information play an important role in accurate forecasting of long time series. Traditional methods rely on setting up temporal dependencies manually to explore related patterns in historical data, which is unrealistic in forecasting long-term series on real-world data. Instead, we propose to explicitly learn constructing hidden patterns' representations with deep neural networks and attending to different parts of the history for forecasting the future.

In this paper, we propose an end-to-end deep-learning framework for multi-horizon time series forecasting, with temporal attention mechanisms to better capture latent patterns in historical data which are useful in predicting the future. Forecasts of multiple quantiles on multiple future horizons can be generated simultaneously based on the learned latent pattern features. We also propose a multimodal fusion mechanism which is used to combine features from different parts of the history to better represent the future. Experiment results demonstrate our approach achieves state-of-the-art performance on two large-scale forecasting datasets in different domains.

References

[1]

Mart'in Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et almbox. {n. d.}. Tensorflow: a system for large-scale machine learning.

[2]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014).

[3]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Machine Learning .

[4]

Filippo Maria Bianchi, Enrico Maiorino, Michael C Kampffmeyer, Antonello Rizzi, and Robert Jenssen. 2017. An overview and comparative analysis of recurrent neural networks for short term load forecasting. arXiv preprint arXiv:1705.04378 (2017).

[5]

G. E. P. Box and G. M. Jenkins. 1968. Some Recent Advances in Forecasting and Control. Journal of the Royal Statistical Society (1968).

[6]

Yagmur Gizem Cinar, Hamid Mirisaee, Parantapa Goswami, Eric Gaussier, Ali A"it-Bachir, and Vadim Strijov. 2017. Position-based content attention for time series forecasting with sequence-to-sequence rnns. In Advances in Neural Information Processing Systems .

[7]

Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. 2015. Long-term recurrent convolutional networks for visual recognition and description. In IEEE Conference on Computer Vision and Pattern Recognition .

[8]

Jeffrey L Elman. 1990. Finding structure in time. Cognitive science (1990).

[9]

Valentin Flunkert, David Salinas, and Jan Gasthaus. 2017. DeepAR: Probabilistic forecasting with autoregressive recurrent networks. arXiv preprint arXiv:1704.04110 .

[10]

Jerome H Friedman. 2001. Greedy function approximation: a gradient boosting machine. Annals of Statistics (2001).

[11]

Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition .

Digital Library

[12]

Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 (2013).

[13]

Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In IEEE Conference on Acoustics, Speech and Signal Processing .

[14]

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation (1997).

[15]

Charles C Holt. 2004. Forecasting seasonals and trends by exponentially weighted moving averages. International Journal of Forecasting (2004).

[16]

Tao Hong, Pierre Pinson, Shu Fan, Hamidreza Zareipour, Alberto Troccoli, and Rob J Hyndman. 2016. Probabilistic energy forecasting: Global Energy Forecasting Competition 2014 and beyond. International Journal of Forecasting (2016).

[17]

Chiori Hori, Takaaki Hori, Teng-Yok Lee, Ziming Zhang, Bret Harsham, John R Hershey, Tim K Marks, and Kazuhiko Sumi. 2017. Attention-based multimodal fusion for video description. In IEEE International Conference on Computer Vision .

[18]

Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In IEEE Conference on Computer Vision and Pattern Recognition .

[19]

Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations .

[20]

Roger Koenker and Gilbert Bassett Jr. 1978. Regression quantiles. Econometrica: journal of the Econometric Society (1978).

[21]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems .

Digital Library

[22]

Martin L"angkvist, Lars Karlsson, and Amy Loutfi. 2014. A review of unsupervised feature learning and deep learning for time-series modeling. Pattern Recognition Letters (2014).

[23]

Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research (2008).

[24]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems .

Digital Library

[25]

Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 (2016).

[26]

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. (2017).

[27]

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems .

Digital Library

[28]

Souhaib Ben Taieb and Amir F Atiya. 2016. A bias and variance analysis for multistep-ahead time series forecasting. IEEE Transactions on Neural Networks and Learning Systems (2016).

[29]

Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. 2017. A Multi-Horizon Quantile Recurrent Forecaster. arXiv preprint arXiv:1711.11053 (2017).

[30]

Peter R Winters. 1960. Forecasting sales by exponentially weighted moving averages. Management science (1960).

Digital Library

[31]

Qifa Xu, Xi Liu, Cuixia Jiang, and Keming Yu. 2016. Quantile autoregression neural network model with applications to evaluating value at risk. Applied Soft Computing (2016).

Digital Library

[32]

Hsiang-Fu Yu, Nikhil Rao, and Inderjit S Dhillon. 2016. Temporal regularized matrix factorization for high-dimensional time series prediction. In Advances in Neural Information Processing Systems .

Digital Library

[33]

Songfeng Zheng. 2010. Boosting based conditional quantile estimation for regression and binary classification. In Mexican International Conference on Artificial Intelligence .

Digital Library

Cited By

M. Sanjai Dr. C. Meenakshi (2024)Efficient Clothing Fashion Prediction using Machine LearningInternational Journal of Advanced Research in Science, Communication and Technology10.48175/IJETIR-1231(169-173)Online publication date: 10-Jul-2024
https://doi.org/10.48175/IJETIR-1231
Fan YTang QGuo YWei Y(2024)BiLSTM-MLAM: A Multi-Scale Time Series Prediction Model for Sensor Data Based on Bi-LSTM and Local Attention MechanismsSensors10.3390/s2412396224:12(3962)Online publication date: 19-Jun-2024
https://doi.org/10.3390/s24123962
Liu XWang W(2024)Deep Time Series Forecasting Models: A Comprehensive SurveyMathematics10.3390/math1210150412:10(1504)Online publication date: 11-May-2024
https://doi.org/10.3390/math12101504
Show More Cited By

Index Terms

Multi-Horizon Time Series Forecasting with Temporal Attention Learning
1. Applied computing
2. Computing methodologies
  1. Machine learning
    1. Machine learning algorithms
    2. Machine learning approaches
      1. Neural networks

Recommendations

Sales forecasting for Chemical Products by Using SARIMA Model
ICBDE '22: Proceedings of the 5th International Conference on Big Data and Education

Sales forecasting is widely used in enterprise resource management, which provides valuable information for efficient management. Sales forecasting facilitates the company to produce and stock products on demand. Based on the analysis of time series, ...
Time series forecasting by recurrent product unit neural networks

Time series forecasting (TSF) consists on estimating models to predict future values based on previously observed values of time series, and it can be applied to solve many real-world problems. TSF has been traditionally tackled by considering ...
Time series forecasting with the WARIMAX-GARCH method

It is well-known that causal forecasting methods that include appropriately chosen Exogenous Variables (EVs) very often present improved forecasting performances over univariate methods. However, in practice, EVs are usually difficult to obtain and in ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '19: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

July 2019

3305 pages

ISBN:9781450362016

DOI:10.1145/3292500

General Chairs:
Ankur Teredesai
KenSci
,
Vipin Kumar
University of Minnesota
,
Program Chairs:
Ying Li
EV Analysis Corporation
,
Rómer Rosales
LinkedIn
,
Evimaria Terzi
Boston University
,
George Karypis
University of Minnesota

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

KDD '19

Sponsor:

KDD '19: The 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 4 - 8, 2019

AK, Anchorage, USA

Acceptance Rates

KDD '19 Paper Acceptance Rate 110 of 1,200 submissions, 9%;

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '24

Sponsor:
sigkdd
sigkdd

The 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 25 - 29, 2024

Barcelona , Spain

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

119
Total Citations
View Citations
4,445
Total Downloads

Downloads (Last 12 months)490
Downloads (Last 6 weeks)43

Reflects downloads up to 26 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

M. Sanjai Dr. C. Meenakshi (2024)Efficient Clothing Fashion Prediction using Machine LearningInternational Journal of Advanced Research in Science, Communication and Technology10.48175/IJETIR-1231(169-173)Online publication date: 10-Jul-2024
https://doi.org/10.48175/IJETIR-1231
Fan YTang QGuo YWei Y(2024)BiLSTM-MLAM: A Multi-Scale Time Series Prediction Model for Sensor Data Based on Bi-LSTM and Local Attention MechanismsSensors10.3390/s2412396224:12(3962)Online publication date: 19-Jun-2024
https://doi.org/10.3390/s24123962
Liu XWang W(2024)Deep Time Series Forecasting Models: A Comprehensive SurveyMathematics10.3390/math1210150412:10(1504)Online publication date: 11-May-2024
https://doi.org/10.3390/math12101504
Kondath NMyat ASoh YTung WEugene KAn H(2024)Enhancing Day-Ahead Cooling Load Prediction in Tropical Commercial Buildings Using Advanced Deep Learning Models: A Case Study in SingaporeBuildings10.3390/buildings1402039714:2(397)Online publication date: 1-Feb-2024
https://doi.org/10.3390/buildings14020397
Genet RInzirillo H(2024)TKAN: Temporal Kolmogorov-Arnold NetworksSSRN Electronic Journal10.2139/ssrn.4825654Online publication date: 2024
https://doi.org/10.2139/ssrn.4825654
Mohd Razak SCornelio JCho YLiu HVaidya RJafarpour B(2024)Dynamic Physics-Guided Deep Learning for Long-Term Production Forecasting in Unconventional ReservoirsSPE Journal10.2118/221474-PA(1-19)Online publication date: 29-Jul-2024
https://doi.org/10.2118/221474-PA
Mendis KWickramasinghe MMarasinghe P(2024)Multivariate Time Series Forecasting: A ReviewProceedings of the 2024 2nd Asia Conference on Computer Vision, Image Processing and Pattern Recognition10.1145/3663976.3664241(1-9)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3663976.3664241
Yi KZhang QHe HShi KHu LAn NNiu Z(2024)Deep Coupling Network for Multivariate Time Series ForecastingACM Transactions on Information Systems10.1145/365344742:5(1-28)Online publication date: 27-Apr-2024
https://dl.acm.org/doi/10.1145/3653447
Liu CHe SLiu HChen JDong H(2024)WindTrans: Transformer-Based Wind Speed Forecasting Method for High-Speed RailwayIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2023.333715025:6(4947-4963)Online publication date: Jul-2024
https://doi.org/10.1109/TITS.2023.3337150
Kumari PK SJagadeesh PT D(2024)ConciseNet: A Robust Model for Enhancing Signal Reconstruction in Wireless Sensor Networks2024 Fourth International Conference on Advances in Electrical, Computing, Communication and Sustainable Technologies (ICAECT)10.1109/ICAECT60202.2024.10469289(1-4)Online publication date: 11-Jan-2024
https://doi.org/10.1109/ICAECT60202.2024.10469289
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents