Abstract
We present a theoretical analysis of the approximation properties of convolutional architectures when applied to the modeling of temporal sequences. Specifically, we prove an approximation rate estimate (Jackson-type result) and an inverse approximation theorem (Bernstein-type result), which together provide a comprehensive characterization of the types of sequential relationships that can be efficiently captured by a temporal convolutional architecture. The rate estimate improves upon a previous result via the introduction of a refined complexity measure, whereas the inverse approximation theorem is new.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Achieser, N.I.: Theory of Approximation. Courier Corporation (2013)
Bai, S., Kolter, J.Z., Koltun, V.: An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018). https://doi.org/10.48550/arXiv.1803.01271
Chu, B.S., Lee, C.R.: Low-rank Tensor Decomposition for Compression of Convolutional Neural Networks Using Funnel Regularization (2021). https://doi.org/10.48550/arXiv.2112.03690
De Lathauwer, L., De Moor, B., Vandewalle, J.: A multilinear singular value decomposition. SIAM J. Matrix Anal. Appl. 21(4), 1253–1278 (2000). https://doi.org/10.1137/S0895479896305696
Hayashi, K., Yamaguchi, T., Sugawara, Y., Maeda, S.I.: Exploring unexplored tensor network decompositions for convolutional neural networks. In: Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc. (2019)
Jiang, H., Li, Q., Li, Z., Wang, S.: A Brief Survey on the Approximation Theory for Sequence Modelling (2023). https://doi.org/10.48550/arXiv.2302.13752
Jiang, H., Li, Z., Li, Q.: Approximation theory of convolutional architectures for time series modelling. In: Proceedings of the 38th International Conference on Machine Learning, pp. 4961–4970. PMLR (2021)
Kim, Y.D., Park, E., Yoo, S., Choi, T., Yang, L., Shin, D.: Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications (2016). https://doi.org/10.48550/arXiv.1511.06530
Kreyszig, E.: Introductory Functional Analysis with Applications. Wiley Classics Library, Wiley, New York, wiley classics library ed edn. (1989)
Li, Z., Han, J., E, W., Li, Q.: Approximation and optimization theory for linear continuous-time recurrent neural networks. J. Mach. Learn. Res. 23(42), 1–85 (2022)
van den Oord, A., et al.: WaveNet: A Generative Model for Raw Audio. arXiv:1609.03499 (2016)
Phan, A.H., et al.: Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Network (2020). https://doi.org/10.48550/arXiv.2008.05441
Yin, W., Kann, K., Yu, M., Schütze, H.: Comparative Study of CNN and RNN for Natural Language Processing (2017). https://doi.org/10.48550/arXiv.1702.01923
Yu, F., Koltun, V.: Multi-Scale Context Aggregation by Dilated Convolutions (2016). https://doi.org/10.48550/arXiv.1511.07122
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Jiang, H., Li, Q. (2023). Forward and Inverse Approximation Theory for Linear Temporal Convolutional Networks. In: Nielsen, F., Barbaresco, F. (eds) Geometric Science of Information. GSI 2023. Lecture Notes in Computer Science, vol 14072. Springer, Cham. https://doi.org/10.1007/978-3-031-38299-4_36
Download citation
DOI: https://doi.org/10.1007/978-3-031-38299-4_36
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-38298-7
Online ISBN: 978-3-031-38299-4
eBook Packages: Computer ScienceComputer Science (R0)