Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3583780.3615097acmconferencesArticle/Chapter ViewAbstractPublication PagescikmConference Proceedingsconference-collections
research-article

TriD-MAE: A Generic Pre-trained Model for Multivariate Time Series with Missing Values

Published: 21 October 2023 Publication History
  • Get Citation Alerts
  • Abstract

    Multivariate time series(MTS) is a universal data type related to various real-world applications. Data imputation methods are widely used in MTS applications to deal with the frequent data missing problem. However, these methods inevitably introduce biased imputation and training-redundancy problems in downstream training. To address these challenges, we propose TriD-MAE, a generic pre-trained model for MTS data with missing values. Firstly, we introduce TriD-TCN, an end-to-end module based on TCN that effectively extracts temporal features by integrating dynamic kernel mechanisms and a time-flipping trick. Building upon that, we designed an MAE-based pre-trained model as the precursor of specialized downstream models. Our model cooperates with a dynamic positional embedding mechanism to represent the missing information and generate transferable representation through our proposed encoder units. The overall mixed data feed-in strategy and weighted loss function are established to ensure adequate training of the whole model. Comparative experiment results in time series prediction and classification manifest that our TriD-MAE model outperforms the other state-of-the-art methods within six real-world datasets. Moreover, ablation and interpretability experiments are delivered to verify the validity of TriD-MAE's

    References

    [1]
    Arthur Asuncion and David Newman. 2007. UCI machine learning repository.
    [2]
    Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. 2018. The UEA multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075 (2018).
    [3]
    Shaojie Bai, J Zico Kolter, and Vladlen Koltun. 2018. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 (2018).
    [4]
    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, and Amanda Askell. 2020. Language models are few-shot learners. Advances in neural information processing systems, Vol. 33 (2020), 1877--1901.
    [5]
    Luis Candanedo. 2017. Appliances energy prediction. UCI Machine Learning Repository.
    [6]
    Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. 2018. Brits: Bidirectional recurrent imputation for time series. Advances in neural information processing systems, Vol. 31 (2018).
    [7]
    Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. 2018. Recurrent neural networks for multivariate time series with missing values. Scientific reports, Vol. 8, 1 (2018), 1--12.
    [8]
    Song Chen. 2019. Beijing Multi-Site Air-Quality Data. UCI Machine Learning Repository.
    [9]
    Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. 2020. Dynamic convolution: Attention over convolution kernels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11030--11039.
    [10]
    Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 (2014).
    [11]
    Andrea Cini, Ivan Marisca, and Cesare Alippi. 2021. Filling the G_ap_s: Multivariate Time Series Imputation by Graph Neural Networks. In International Conference on Learning Representations.
    [12]
    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
    [13]
    A Rogier T Donders, Geert JMG Van Der Heijden, Theo Stijnen, and Karel GM Moons. 2006. A gentle introduction to imputation of missing values. Journal of clinical epidemiology, Vol. 59, 10 (2006), 1087--1091.
    [14]
    Wei Fan, Shun Zheng, Xiaohan Yi, Wei Cao, Yanjie Fu, Jiang Bian, and Tie-Yan Liu. 2022. DEPTS: deep expansion learning for periodic time series forecasting. arXiv preprint arXiv:2203.07681 (2022).
    [15]
    Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. 2019. Unsupervised Scalable Representation Learning for Multivariate Time Series. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2019/file/53c6de78244e9f528eb3e1cda69699bb-Paper.pdf
    [16]
    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2021. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377 (2021).
    [17]
    Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, Vol. 9, 8 (1997), 1735--1780.
    [18]
    Kathan Kashiparekh, Jyoti Narwariya, Pankaj Malhotra, Lovekesh Vig, and Gautam Shroff. 2019. ConvTimeNet: A Pre-trained Deep Convolutional Neural Network for Time Series Classification. In 2019 International Joint Conference on Neural Networks (IJCNN). 1--8. https://doi.org/10.1109/IJCNN.2019.8852105
    [19]
    Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 (2020).
    [20]
    .Ibrahim Kök and Suat Özdemir. 2020. Deepmdp: A novel deep-learning-based missing data prediction protocol for iot. IEEE Internet of Things Journal, Vol. 8, 1 (2020), 232--243.
    [21]
    Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. 2018. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 95--104.
    [22]
    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. https://doi.org/10.48550/arXiv.1907.11692 arXiv:1907.11692 [cs].
    [23]
    Yonghong Luo, Ying Zhang, Xiangrui Cai, and Xiaojie Yuan. 2019. E2gan: End-to-end generative adversarial network for multivariate time series imputation. In Proceedings of the 28th international joint conference on artificial intelligence. AAAI Press, 3094--3100.
    [24]
    Pankaj Malhotra, Vishnu TV, Lovekesh Vig, Puneet Agarwal, and Gautam Shroff. 2017. TimeNet: Pre-trained deep recurrent neural network for time series classification. arXiv preprint arXiv:1706.08838 (2017).
    [25]
    Jingci Ming, Le Zhang, Wei Fan, Weijia Zhang, Yu Mei, Weicen Ling, and Hui Xiong. 2022. Multi-Graph Convolutional Recurrent Network for Fine-Grained Lane-Level Traffic Flow Imputation. In 2022 IEEE International Conference on Data Mining (ICDM). IEEE, 348--357.
    [26]
    Vladimiro Miranda, Jakov Krstulovic, Hrvoje Keko, Cristiano Moreira, and Jorge Pereira. 2011. Reconstructing missing data in state estimation with autoencoders. IEEE Transactions on power systems, Vol. 27, 2 (2011), 604--611.
    [27]
    Vladimiro Miranda, Jakov Krstulovic, Hrvoje Keko, Cristiano Moreira, and Jorge Pereira. 2012. Reconstructing Missing Data in State Estimation With Autoencoders. IEEE Transactions on Power Systems, Vol. 27, 2 (2012), 604--611. https://doi.org/10.1109/TPWRS.2011.2174810
    [28]
    Zhuofu Pan, Yalin Wang, Kai Wang, Hongtian Chen, Chunhua Yang, and Weihua Gui. 2022. Imputation of Missing Values in Time Series Using an Adaptive-Learned Median-Filled Deep Autoencoder. IEEE Transactions on Cybernetics (2022), 1--12. https://doi.org/10.1109/TCYB.2022.3167995
    [29]
    Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and Garrison Cottrell. 2017. A dual-stage attention-based recurrent neural network for time series prediction. arXiv preprint arXiv:1704.02971 (2017).
    [30]
    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. (2018).
    [31]
    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, Vol. 1, 8 (2019), 9.
    [32]
    Francisco Romeu-Guallart, Pablo Zamora-Martinez. 2014. SML2010. UCI Machine Learning Repository.
    [33]
    Donald B Rubin. 1976. Inference and missing data. Biometrika, Vol. 63, 3 (1976), 581--592.
    [34]
    Joseph L Schafer. 1999. Multiple imputation: a primer. Statistical methods in medical research, Vol. 8, 1 (1999), 3--15.
    [35]
    Shun-Yao Shih, Fan-Keng Sun, and Hung-yi Lee. 2019. Temporal pattern attention for multivariate time series forecasting. Machine Learning, Vol. 108, 8 (2019), 1421--1441.
    [36]
    Xianfeng Tang, Huaxiu Yao, Yiwei Sun, Charu Aggarwal, Prasenjit Mitra, and Suhang Wang. 2020. Joint modeling of local and global temporal dynamics for multivariate time series forecasting with missing values. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 5956--5963.
    [37]
    Roman Tkachenko, Ivan Izonin, Natalia Kryvinska, Ivanna Dronyuk, and Khrystyna Zub. 2020. An approach towards increasing prediction accuracy for the recovery of missing IoT data based on the GRNN-SGTM ensemble. Sensors, Vol. 20, 9 (2020), 2625.
    [38]
    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, Vol. 30 (2017).
    [39]
    Wang Weihan. 2020. MAGAN: A masked autoencoder generative adversarial network for processing missing IoT sequence data. Pattern Recognition Letters, Vol. 138 (2020), 211--216.
    [40]
    Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. 2021. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, Vol. 34 (2021), 22419--22430.
    [41]
    Jinsung Yoon, James Jordon, and Mihaela Schaar. 2018. Gain: Missing data imputation using generative adversarial nets. In International conference on machine learning. PMLR, 5689--5698.
    [42]
    George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. 2021. A Transformer-Based Framework for Multivariate Time Series Representation Learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (Virtual Event, Singapore) (KDD '21). Association for Computing Machinery, New York, NY, USA, 2114--2124. https://doi.org/10.1145/3447548.3467401
    [43]
    Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, Vol. 35. 11106--11115.

    Cited By

    View all
    • (2024)Label-Free Multivariate Time Series Anomaly DetectionIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.334961336:7(3166-3179)Online publication date: Jul-2024
    • (2024)WindTrans: Transformer-Based Wind Speed Forecasting Method for High-Speed RailwayIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2023.333715025:6(4947-4963)Online publication date: Jun-2024
    • (2024)Deep learning-based clustering method for single-cell RNA data2024 7th International Symposium on Autonomous Systems (ISAS)10.1109/ISAS61044.2024.10552501(1-6)Online publication date: 7-May-2024
    • Show More Cited By

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CIKM '23: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management
    October 2023
    5508 pages
    ISBN:9798400701245
    DOI:10.1145/3583780
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 21 October 2023

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. missing data
    2. pre-trained model
    3. time series

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    CIKM '23
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

    Upcoming Conference

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)251
    • Downloads (Last 6 weeks)23
    Reflects downloads up to 09 Aug 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Label-Free Multivariate Time Series Anomaly DetectionIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.334961336:7(3166-3179)Online publication date: Jul-2024
    • (2024)WindTrans: Transformer-Based Wind Speed Forecasting Method for High-Speed RailwayIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2023.333715025:6(4947-4963)Online publication date: Jun-2024
    • (2024)Deep learning-based clustering method for single-cell RNA data2024 7th International Symposium on Autonomous Systems (ISAS)10.1109/ISAS61044.2024.10552501(1-6)Online publication date: 7-May-2024
    • (2024)Multidimensional information fusion and broad learning system-based condition recognition for energy pipeline safetyKnowledge-Based Systems10.1016/j.knosys.2024.112259300(112259)Online publication date: Sep-2024
    • (2024)Localizing and tracking of in-pipe inspection robots based on distributed optical fiber sensingAdvanced Engineering Informatics10.1016/j.aei.2024.10242460(102424)Online publication date: Apr-2024
    • (2023)NTDformer: A Multi-Scale Forecasting Model for Non-Stationary Multilevel Time Series2023 5th International Conference on Smart Power & Internet Energy Systems (SPIES)10.1109/SPIES60658.2023.10474846(105-110)Online publication date: 1-Dec-2023

    View Options

    Get Access

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media