research-article

POLLA: Enhancing the Local Structure Awareness in Long Sequence Spatial-temporal Modeling

Authors:

Jianxin LiAuthors Info & Claims

ACM Transactions on Intelligent Systems and Technology (TIST), Volume 12, Issue 6

Article No.: 69, Pages 1 - 24

https://doi.org/10.1145/3447987

Published: 29 November 2021 Publication History

Abstract

The spatial-temporal modeling on long sequences is of great importance in many real-world applications. Recent studies have shown the potential of applying the self-attention mechanism to improve capturing the complex spatial-temporal dependencies. However, the lack of underlying structure information weakens its general performance on long sequence spatial-temporal problem. To overcome this limitation, we proposed a novel method, named the Proximity-aware Long Sequence Learning framework, and apply it to the spatial-temporal forecasting task. The model substitutes the canonical self-attention by leveraging the proximity-aware attention, which enhances local structure clues in building long-range dependencies with a linear approximation of attention scores. The relief adjacency matrix technique can utilize the historical global graph information for consistent proximity learning. Meanwhile, the reduced decoder allows for fast inference in a non-autoregressive manner. Extensive experiments are conducted on five large-scale datasets, which demonstrate that our method achieves state-of-the-art performance and validates the effectiveness brought by local structure information.

References

[1]

2019. A deep spatial-temporal data-driven approach considering microclimates for power system security assessment. Appl. Energy 237 (2019), 36–48.

[2]

Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. 2019. MixHop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In ICML’19, Proceedings of Machine Learning Research, Vol. 97. 21–29.

[3]

Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150. Retrieved from https://arxiv.org/abs/2004.05150l.

[4]

Salah Bouktif, Ali Fiaz, Ali Ouni, and Mohamed Adel Serhani. 2018. Optimal deep learning LSTM model for electric load forecasting using feature selection and genetic algorithm: Comparison with machine learning approaches. Energies 11, 7 (2018), 1–20.

[5]

Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. arXiv:1904.10509. Retrieved from https://arxiv.org/abs/1904.10509.

[6]

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. In NIPS’16. 3837–3845.

Digital Library

[7]

Edouard Delasalles, Ali Ziat, Ludovic Denoyer, and Patrick Gallinari. 2019. Spatio-temporal neural networks for space-time data modeling and relation discovery. Knowl. Inf. Syst. 61, 3 (2019), 1241–1267.

Digital Library

[8]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT’19, Volume 1 (Long and Short Papers). Association for Computational Linguistics, 4171–4186.

[9]

Harris Drucker, Christopher J. C. Burges, Linda Kaufman, Alexander J. Smola, and Vladimir Vapnik. 1996. Support vector regression machines. In NIPS’96. MIT Press, 155–161.

Digital Library

[10]

Bowen Du, Hao Peng, Senzhang Wang, Md. Zakirul Alam Bhuiyan, Lihong Wang, Qiran Gong, Lin Liu, and Jing Li. 2020. Deep irregular convolutional residual LSTM for urban traffic passenger flows prediction. IEEE Trans. Intell. Transp. Syst. 21, 3 (2020), 972–985. https://doi.org/10.1109/TITS.2019.2900481

[11]

Jie Feng, Yong Li, Chao Zhang, Funing Sun, Fanchao Meng, Ang Guo, and Depeng Jin. 2018. DeepMove: Predicting human mobility with attentional recurrent networks. In WWW’18. ACM, 1459–1468.

Digital Library

[12]

Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv:1308.0850. Retrieved from https://arxiv.org/abs/1308.0850.

[13]

Shengnan Guo, Youfang Lin, Shijie Li, Zhaoming Chen, and Huaiyu Wan. 2019. Deep spatial-temporal 3D convolutional neural networks for traffic data forecasting. IEEE Trans. Intell. Transp. Syst. 20, 10 (2019), 3913–3926.

[14]

James Douglas Hamilton. 2020. Time Series Analysis. Princeton University Press.

[15]

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput. 9, 8 (1997), 1735–1780.

Digital Library

[16]

Chao Huang, Junbo Zhang, Yu Zheng, and Nitesh V. Chawla. 2018. DeepCrime: Attentive hierarchical recurrent networks for crime prediction. In CIKM’18. ACM, 1423–1432.

Digital Library

[17]

Peter J. Huber. 1992. Robust estimation of a location parameter. In Breakthroughs in Statistics. Springer, 492–518.

[18]

Yiannis Kamarianakis and Poulicos Prastacos. 2003. Forecasting traffic flow conditions in an urban network: Comparison of multivariate and univariate approaches. Transport. Res. Rec. 1857, 1 (2003), 74–84.

[19]

Yiannis Kamarianakis and Poulicos Prastacos. 2005. Space-time modeling of traffic flow. Comput. Geosci. 31, 2 (2005), 119–133.

Digital Library

[20]

Antonios Karatzoglou, Nikolai Schnell, and Michael Beigl. 2018. A convolutional neural network approach for modeling semantic trajectories and predicting future locations. In ICANN’18Lecture Notes in Computer Science, Vol. 11139. Springer, 61–72.

[21]

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020. Transformers are RNNs: Fast autoregressive transformers with linear attention. arXiv:2006.16236. Retrieved from https://arxiv.org/abs/2006.16236.

[22]

Seongchan Kim, Seungkyun Hong, Minsu Joh, and Sa-Kwang Song. 2017. DeepRain: ConvLSTM network for precipitation prediction using multichannel radar data. arXiv:1711.02316. Retrieved from https://arxiv.org/abs/1711.02316.

[23]

Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In ICLR 2017. OpenReview.net.

[24]

Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. arXiv:2001.04451. Retrieved from https://arxiv.org/abs/2001.04451.

[25]

Ruoyu Li, Sheng Wang, Feiyun Zhu, and Junzhou Huang. 2018. Adaptive graph convolutional neural networks. In AAAI’18. AAAI Press, 3546–3553.

Digital Library

[26]

Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. 2019. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In NIPS’19. 5244–5254.

Digital Library

[27]

Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. 2018. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. In ICLR’18. OpenReview.net.

[28]

Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. 2016. Predicting the next location: A recurrent model with spatial and temporal contexts. In AAAI’16. AAAI Press, 194–200.

Digital Library

[29]

Zhongjian Lv, Jiajie Xu, Kai Zheng, Hongzhi Yin, Pengpeng Zhao, and Xiaofang Zhou. 2018. LC-RNN: A deep learning model for traffic speed prediction. In IJCAI’18. 3470–3476.

Digital Library

[30]

Li Mengzhang and Zhu Zhanxing. 2020. Spatial-temporal fusion graph neural networks for traffic flow forecasting. arXiv:2012.09641 [cs.LG]. Retrieved from https://arxiv.org/abs/2012.09641.

[31]

Cheonbok Park, Chunggi Lee, Hyojin Bahng, Yunwon Tae, Seungmin Jin, Kihwan Kim, Sungahn Ko, and Jaegul Choo. 2020. ST-GRAT: A novel spatio-temporal graph attention networks for accurately forecasting dynamically changing road speed. In CIKM’20. ACM, 1215–1224.

Digital Library

[32]

Cheonbok Park, Chunggi Lee, Hyojin Bahng, Taeyun won, Kihwan Kim, Seungmin Jin, Sungahn Ko, and Jaegul Choo. 2019. STGRAT: A spatio-temporal graph attention network for traffic forecasting. arXiv:1911.13181. Retrieved from https://arxiv.org/abs/1911.13181.

[33]

Hao Peng, Hongfei Wang, Bowen Du, Md. Zakirul Alam Bhuiyan, Hongyuan Ma, Jianwei Liu, Lihong Wang, Zeyu Yang, Linfeng Du, Senzhang Wang, and Philip S. Yu. 2020. Spatial temporal incidence dynamic graph neural networks for traffic flow forecasting. Inf. Sci. 521 (2020), 277–290.

Digital Library

[34]

Chiara Plizzari, Marco Cannici, and Matteo Matteucci. 2020. Spatial temporal transformer network for skeleton-based action recognition. arXiv:2008.07404. Retrieved from https://arxiv.org/abs/2008.07404.

[35]

Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. 2015. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In NIPS’15. 802–810.

Digital Library

[36]

David I. Shuman, Sunil K. Narang, Pascal Frossard, Antonio Ortega, and Pierre Vandergheynst. 2013. The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains. IEEE Sign. Process. Mag. 30, 3 (2013), 83–98.

[37]

Karen Simonyan and Andrew Zisserman. 2014. Two-stream convolutional networks for action recognition in videos. In NIPS’14. 568–576.

Digital Library

[38]

Chao Song, Youfang Lin, Shengnan Guo, and Huaiyu Wan. 2020. Spatial-temporal synchronous graph convolutional networks: A new framework for spatial-temporal network data forecasting. In AAAI’20. AAAI Press, 914–921.

[39]

Sean J Taylor and Benjamin Letham. 2018. Forecasting at scale. Am. Stat. 72, 1 (2018), 37–45.

[40]

Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Transformer dissection: An unified understanding for transformer’s attention via the lens of kernel. In EMNLP-IJCNLP’19. Association for Computational Linguistics, 4344–4353.

[41]

Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. 2016. WaveNet: A generative model for raw audio. In Proceedings of the 9th ISCA Speech Synthesis Workshop. ISCA, 125.

[42]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS’17. 5998–6008.

Digital Library

[43]

Bao Wang, Xiyang Luo, Fangbo Zhang, Baichuan Yuan, Andrea L. Bertozzi, and P. Jeffrey Brantingham. 2018. Graph-based deep modeling and real time forecasting of sparse spatio-temporal data. arXiv:1804.00684. Retrieved from https://arxiv.org/abs/1804.00684.

[44]

Leye Wang, Xu Geng, Xiaojuan Ma, Feng Liu, and Qiang Yang. 2018. Crowd flow prediction by deep spatio-temporal transfer learning. arXiv:1802.00386. Retrieved from https://arxiv.org/abs/1802.00386.

[45]

Senzhang Wang, Jiannong Cao, and Philip S. Yu. 2019. Deep learning for spatio-temporal data mining: A survey. arXiv:1906.04928. Retrieved from https://arxiv.org/abs/1906.04928.

[46]

Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv:2006.04768. Retrieved from https://arxiv.org/abs/2006.04768.

[47]

Billy M Williams and Lester A Hoel. 2003. Modeling and forecasting vehicular traffic flow as a seasonal ARIMA process: Theoretical basis and empirical results. J. Transport. Eng. 129, 6 (2003), 664–672.

[48]

Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. 2019. Graph WaveNet for deep spatial-temporal graph modeling. In IJCAI’19. 1907–1913.

Digital Library

[49]

Sijie Yan, Yuanjun Xiong, and Dahua Lin. 2018. Spatial temporal graph convolutional networks for skeleton-based action recognition. In AAAI’18. AAAI Press, 7444–7452.

Digital Library

[50]

Zhixian Yan. 2010. Traj-ARIMA: A spatial-time series model for network-constrained trajectory. In IWCTS’21. ACM, 11–16.

Digital Library

[51]

Bing Yu, Haoteng Yin, and Zhanxing Zhu. 2018. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. In IJCAI’18. 3634–3640.

Digital Library

[52]

Zhuoning Yuan, Xun Zhou, and Tianbao Yang. 2018. Hetero-ConvLSTM: A deep learning approach to traffic accident prediction on heterogeneous spatio-temporal data. In KDD’18. ACM, 984–992.

Digital Library

[53]

Y. Zhang, S. Wang, B. Chen, J. Cao, and Z. Huang. 2021. TrafficGAN: Network-scale deep traffic prediction with generative adversarial nets. IEEE Trans. Intell. Transport. Syst. 22, 1 (2021), 219–230.

Digital Library

[54]

Chuanpan Zheng, Xiaoliang Fan, Cheng Wang, and Jianzhong Qi. 2020. GMAN: A graph multi-attention network for traffic prediction. In AAAI’20. AAAI Press, 1234–1241.

[55]

Chuanpan Zheng, Xiaoliang Fan, Chenglu Wen, Longbiao Chen, Cheng Wang, and Jonathan Li. 2020. DeepSTD: Mining spatio-temporal disturbances of multiple context factors for citywide traffic flow prediction. IEEE Trans. Intell. Transport. Syst. 21, 9 (2020), 3744–3755.

[56]

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI’21. AAAI Press.

Cited By

Li ZGao ZZhang XZhang GXu L(2024)Time-aware personalized graph convolutional network for multivariate time series forecastingExpert Systems with Applications10.1016/j.eswa.2023.122471240(122471)Online publication date: May-2024
https://doi.org/10.1016/j.eswa.2023.122471
Li ZGao ZZhang GLiu JXu L(2024)Dynamic personalized graph neural network with linear complexity for multivariate time series forecastingEngineering Applications of Artificial Intelligence10.1016/j.engappai.2023.107291127:PAOnline publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1016/j.engappai.2023.107291
Li JZhu TZhou HSun QJiang CZhang SHu C(2022)AIQoSer: Building the efficient Inference-QoS for AI Services2022 IEEE/ACM 30th International Symposium on Quality of Service (IWQoS)10.1109/IWQoS54832.2022.9812905(1-10)Online publication date: 10-Jun-2022
https://doi.org/10.1109/IWQoS54832.2022.9812905
Show More Cited By

Index Terms

POLLA: Enhancing the Local Structure Awareness in Long Sequence Spatial-temporal Modeling

Recommendations

Multi-view Cascading Spatial-Temporal Graph Neural Network for Traffic Flow Forecasting
Artificial Neural Networks and Machine Learning – ICANN 2022
Abstract
Spatial-temporal patterns have been applied in many areas, such as traffic forecasting, skeleton-based recognition, and so on. In such areas, researchers can convert the prior knowledge into graphs and combine the latent graph dependencies into ...
An encoder framework for taxi-demand prediction using spatio-temporal function approximation
Abstract
Predicting taxi demands in large cities can help in better traffic management as well as ensure better commuter satisfaction for an intelligent transportation system. However, the traffic demands across different locations have varying spatio-...
Spatio-Temporal Data Types: An Approach to Modeling and Querying Moving Objects in Databases

Spatio-temporal databases deal with geometries changing over time. In general, geometries cannot only change in discrete steps, but continuously, and we are talking about moving objects. If only the position in space of an object is relevant, then ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Intelligent Systems and Technology

ACM Transactions on Intelligent Systems and Technology Volume 12, Issue 6

December 2021

356 pages

ISSN:2157-6904

EISSN:2157-6912

DOI:10.1145/3501281

Editor:
Huan Liu
Arizona State University, USA

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 November 2021

Accepted: 01 January 2021

Revised: 01 December 2020

Received: 01 October 2020

Published in TIST Volume 12, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

Natural Science Foundation of China
State Key Laboratory of Software Development Environment
CAAI-Huawei MindSpore Open Fund

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
929
Total Downloads

Downloads (Last 12 months)100
Downloads (Last 6 weeks)11

Reflects downloads up to 26 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li ZGao ZZhang XZhang GXu L(2024)Time-aware personalized graph convolutional network for multivariate time series forecastingExpert Systems with Applications10.1016/j.eswa.2023.122471240(122471)Online publication date: May-2024
https://doi.org/10.1016/j.eswa.2023.122471
Li ZGao ZZhang GLiu JXu L(2024)Dynamic personalized graph neural network with linear complexity for multivariate time series forecastingEngineering Applications of Artificial Intelligence10.1016/j.engappai.2023.107291127:PAOnline publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1016/j.engappai.2023.107291
Li JZhu TZhou HSun QJiang CZhang SHu C(2022)AIQoSer: Building the efficient Inference-QoS for AI Services2022 IEEE/ACM 30th International Symposium on Quality of Service (IWQoS)10.1109/IWQoS54832.2022.9812905(1-10)Online publication date: 10-Jun-2022
https://doi.org/10.1109/IWQoS54832.2022.9812905
Papachary BAmru MRama Kishore Reddy S(2021)An Effective Segmentation of Tissues from MR Brain ImagesJournal of Physics: Conference Series10.1088/1742-6596/1964/6/0620291964:6(062029)Online publication date: 1-Jul-2021
https://doi.org/10.1088/1742-6596/1964/6/062029

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents