research-article

Free access

From Fourier to Koopman: spectral methods for long-term time series prediction

AUTHORs:

Steven L. Brunton,

J. Nathan KutzAuthors Info & Claims

The Journal of Machine Learning Research, Volume 22, Issue 1

Article No.: 41, Pages 1881 - 1918

Published: 01 January 2021 Publication History

PDF eReader Publisher Site

Abstract

We propose spectral methods for long-term forecasting of temporal signals stemming from linear and nonlinear quasi-periodic dynamical systems. For linear signals, we introduce an algorithm with similarities to the Fourier transform but which does not rely on periodicity assumptions, allowing for forecasting given potentially arbitrary sampling intervals. We then extend this algorithm to handle nonlinearities by leveraging Koopman theory. The resulting algorithm performs a spectral decomposition in a nonlinear, data-dependent basis. The optimization objective for both algorithms is highly non-convex. However, expressing the objective in the frequency domain allows us to compute global optima of the error surface in a scalable and efficient manner, partially by exploiting the computational properties of the Fast Fourier Transform. Because of their close relation to Bayesian Spectral Analysis, uncertainty quantification metrics are a natural byproduct of the spectral forecasting methods. We extensively benchmark these algorithms against other leading forecasting methods on a range of synthetic experiments as well as in the context of real-world power systems and fluid flows.

References

[1]

J. S. Armstrong. Long-range forecasting. Wiley New York ETC., 1985.

[2]

T. Askham and J. N. Kutz. Variable projection methods for an optimized dynamic mode decomposition. SIAM Journal on Applied Dynamical Systems, 17(1):380-416, 2018.

[3]

S. Bai, J. Z. Kolter, and V. Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv:1803.01271, 2018.

[4]

A. Beck and L. Tetruashvili. On the convergence of block coordinate descent type methods. SIAM journal on Optimization, 23(4):2037-2060, 2013.

[5]

G. D. Birkhoff. Proof of the ergodic theorem. Proceedings of the National Academy of Sciences, 17(12):656-660, 1931.

[6]

G. D. Birkhoff and B. Koopman. Recent contributions to the ergodic theory. Proceedings of the National Academy of Sciences of the United States of America, 18(3):279, 1932.

[7]

G. E. Box, G. M. Jenkins, G. C. Reinsel, and G. M. Ljung. Time series analysis: forecasting and control. John Wiley & Sons, 2015.

[8]

G. L. Bretthorst. Bayesian spectrum analysis and parameter estimation, volume 48. Springer Science & Business Media, 2013.

[9]

B. W. Brunton, L. A. Johnson, J. G. Ojemann, and J. N. Kutz. Extracting spatial-temporal coherent patterns in large-scale neural recordings using dynamic mode decomposition. Journal of neuroscience methods, 258:1-15, 2016a.

[10]

S. L. Brunton, B. W. Brunton, J. L. Proctor, and J. N. Kutz. Koopman invariant subspaces and finite linear representations of nonlinear dynamical systems for control. PLoS ONE, 11(2):e0150171, 2016b.

[11]

S. L. Brunton, B. W. Brunton, J. L. Proctor, E. Kaiser, and J. N. Kutz. Chaos as an intermittently forced linear system. Nature Communications, 8(1):1-9, 2017.

[12]

M. Budišić, R. Mohr, and I. Mezić. Applied Koopmanism. Chaos: An Interdisciplinary Journal of Nonlinear Science, 22(4):047510, 2012.

[13]

K. Champion, B. Lusch, J. N. Kutz, and S. L. Brunton. Data-driven discovery of coordinates and governing equations. Proceedings of the National Academy of Sciences, 116(45): 22445-22451, 2019.

[14]

S.-C. Chan, K. M. Tsui, H. Wu, Y. Hou, Y.-C. Wu, and F. F. Wu. Load/price forecasting and managing demand response for smart grids: Methodologies and challenges. IEEE signal processing magazine, 29(5):68-85, 2012.

[15]

K. K. Chen, J. H. Tu, and C. W. Rowley. Variants of dynamic mode decomposition: boundary condition, Koopman, and Fourier analyses. Journal of nonlinear science, 22 (6):887-915, 2012.

[16]

J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.

[17]

J. W. Cooley and J. W. Tukey. An algorithm for the machine calculation of complex Fourier series. Mathematics of computation, 19(90):297-301, 1965.

[18]

S. T. Dawson, M. S. Hemati, M. O. Williams, and C. W. Rowley. Characterizing and correcting for the effect of sensor noise in the dynamic mode decomposition. Experiments in Fluids, 57(3):42, 2016.

[19]

N. B. Erichson, S. L. Brunton, and J. N. Kutz. Compressed dynamic mode decomposition for real-time object detection. Journal of Real-Time Image Processing, 16(5):1479-1492, 2019a.

[20]

N. B. Erichson, M. Muehlebach, and M. W. Mahoney. Physics-informed autoencoders for Lyapunov-stable fluid flow prediction. arXiv preprint arXiv:1905.10866, 2019b.

[21]

J. R. Fienup. Phase retrieval algorithms: a comparison. Applied optics, 21(15):2758-2769, 1982.

[22]

J. Fourier. Theorie analytique de la chaleur, par M. Fourier. Chez Firmin Didot, père et fils, 1822.

[23]

M. S. Gashler and S. C. Ashmore. Modeling time series data with deep Fourier neural networks. Neurocomputing, 188:3-11, 2016.

[24]

J. D. Hamilton. Time series analysis, volume 2. Princeton New Jersey, 1994.

[25]

M. S. Hemati, C. W. Rowley, E. A. Deem, and L. N. Cattafesta. De-biasing the dynamic mode decomposition for applied Koopman spectral analysis of noisy datasets. Theoretical and Computational Fluid Dynamics, 31(4):349-368, 2017.

[26]

S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.

[27]

R. J. Hyndman, Y. Khandakar, et al. Automatic time series for forecasting: the forecast package for R. Number 6/07. Monash University, Department of Econometrics and Business Statistics, 2007.

[28]

H. Jaeger. Echo state network. scholarpedia, 2(9):2330, 2007.

[29]

E. Jaynes. Bayesian spectrum and chirp analysis. In Maximum-Entropy and Bayesian Spectral Analysis and Estimation Problems, pages 1-37. Springer, 1987.

[30]

T. V. Jensen and P. Pinson. Re-europe, a large-scale dataset for modeling a highly renewable european electricity system. Scientific data, 4:170175, 2017.

[31]

X. Jiang and H. Adeli. Dynamic wavelet neural network model for traffic flow forecasting. Journal of transportation engineering, 131(10):771-779, 2005.

[32]

M. Kamb, E. Kaiser, S. L. Brunton, and J. N. Kutz. Time-delay observables for Koopman: Theory and applications. arXiv preprint arXiv:1810.01479, 2018.

[33]

B. O. Koopman. Hamiltonian systems and transformation in Hilbert space. Proceedings of the National Academy of Sciences of the United States of America, 17(5):315, 1931.

[34]

B. O. Koopman and J. v. Neumann. Dynamical systems of continuous spectra. Proceedings of the National Academy of Sciences of the United States of America, 18(3):255, 1932.

[35]

J. Koutnik, K. Greff, F. Gomez, and J. Schmidhuber. A clockwork rnn. arXiv preprint arXiv:1402.3511, 2014.

[36]

J. N. Kutz, S. L. Brunton, B. W. Brunton, and J. L. Proctor. Dynamic mode decomposition: data-driven modeling of complex systems. SIAM, 2016.

[37]

Y. Lan and I. Mezić. Linearization in the large of nonlinear systems and Koopman operator spectrum. Physica D: Nonlinear Phenomena, 242(1):42-53, 2013.

[38]

Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521(7553):436, 2015.

[39]

B. Lusch, J. N. Kutz, and S. L. Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature Communications, 9(1):4950, 2018.

[40]

D. A. Lyon. The discrete Fourier transform, part 4: spectral leakage. Journal of object technology, 8(7), 2009.

[41]

C. A. Metzler, P. Schniter, A. Veeraraghavan, and R. G. Baraniuk. prdeep: Robust phase retrieval with a flexible deep network. arXiv preprint arXiv:1803.00212, 2018.

[42]

I. Mezić. Spectral properties of dynamical systems, model reduction and decompositions. Nonlinear Dynamics, 41(1-3):309-325, 2005.

[43]

I. Mezic. Analysis of fluid flows via spectral properties of the Koopman operator. Annual Review of Fluid Mechanics, 45:357-378, 2013.

[44]

I. Mezić. On applications of the spectral theory of the Koopman operator in dynamical systems and control theory. In 2015 54th IEEE Conference on Decision and Control (CDC), pages 7034-7041. IEEE, 2015.

[45]

J. P. Moeck, J.-F. Bourgouin, D. Durox, T. Schuller, and S. Candel. Tomographic reconstruction of heat release rate perturbations induced by helical modes in turbulent swirl flames. Experiments in fluids, 54(4):1498, 2013.

[46]

C. C. Moore. Ergodic theorem, ergodic theory, and statistical mechanics. Proceedings of the National Academy of Sciences, 112(7):1907-1911, 2015.

[47]

J. v. Neumann. Physical applications of the ergodic hypothesis. Proceedings of the National Academy of Sciences of the United States of America, 18(3):263, 1932a.

[48]

J. v. Neumann. Proof of the quasi-ergodic hypothesis. Proceedings of the National Academy of Sciences, 18(1):70-82, 1932b.

[49]

S. E. Otto and C. W. Rowley. Linearly recurrent autoencoder networks for learning dynamics. SIAM Journal on Applied Dynamical Systems, 18(1):558-593, 2019.

[50]

S. Pan and K. Duraisamy. Physics-informed probabilistic learning of linear embeddings of non-linear dynamics with guaranteed stability. arXiv preprint arXiv:1906.03663, 2019a.

[51]

S. Pan and K. Duraisamy. On the structure of time-delay embedding in linear models of non-linear dynamical systems. arXiv preprint arXiv:1902.05198, 2019b.

[52]

A. Pankratz. Forecasting with univariate Box-Jenkins models: Concepts and cases, volume 224. John Wiley & Sons, 2009.

[53]

Y. Paquot, F. Duport, A. Smerieri, J. Dambre, B. Schrauwen, M. Haelterman, and S. Massar. Optoelectronic reservoir computing. Scientific reports, 2:287, 2012.

[54]

J. L. Proctor and P. A. Eckhoff. Discovering dynamic patterns from infectious disease data using dynamic mode decomposition. International health, 7(2):139-145, 2015.

[55]

J. L. Proctor, S. L. Brunton, and J. N. Kutz. Dynamic mode decomposition with control. SIAM Journal on Applied Dynamical Systems, 15(1):142-161, 2016.

[56]

J. L. Proctor, S. L. Brunton, and J. N. Kutz. Generalizing Koopman theory to allow for inputs and control. SIAM Journal on Applied Dynamical Systems, 17(1):909-930, 2018.

[57]

C. W. Rowley, I. Mezić, S. Bagheri, P. Schlatter, and D. S. Henningson. Spectral analysis of nonlinear flows. Journal of fluid mechanics, 641:115-127, 2009.

[58]

Y. Sakamoto, M. Ishiguro, and G. Kitagawa. Akaike information criterion statistics. Dordrecht, The Netherlands: D. Reidel, 81, 1986.

[59]

W. Saxton. Computer techniques for image processing in electron microscopy, volume 10. Academic Press, 2013.

[60]

P. J. Schmid. Dynamic mode decomposition of numerical and experimental data. Journal of fluid mechanics, 656:5-28, 2010.

[61]

B. Scholkopf and A. J. Smola. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, 2001.

[62]

A. Silvescu. Fourier neural networks. In IJCNN'99. International Joint Conference on Neural Networks. Proceedings, volume 1, pages 488-491. IEEE, 1999.

[63]

G. Song, F. Alizard, J.-C. Robinet, and X. Gloerfelt. Global and Koopman modes analysis of sound generation in mixing layers. Physics of Fluids, 25(12):124101, 2013.

[64]

J. C. Spall. Cyclic seesaw process for optimization and identification. Journal of optimization theory and applications, 154(1):187-208, 2012.

[65]

N. Takeishi, Y. Kawahara, and T. Yairi. Learning Koopman invariant subspaces for dynamic mode decomposition. In Advances in Neural Information Processing Systems, pages 1130-1140, 2017.

[66]

J. Tithof, B. Suri, R. K. Pallantla, R. O. Grigoriev, and M. F. Schatz. Bifurcations in a quasi-two-dimensional kolmogorov-like flow. Journal of Fluid Mechanics, 828:837-866, 2017.

[67]

P. Tseng. Convergence of a block coordinate descent method for nondifferentiable minimization. Journal of optimization theory and applications, 109(3):475-494, 2001.

[68]

J. H. Tu, C. W. Rowley, D. M. Luchtenburg, S. L. Brunton, and J. N. Kutz. On dynamic mode decomposition: theory and applications. Journal of Computational Dynamics, 1 (2):391-421, 2014.

[69]

A. Vajdi, M. R. Zaghian, S. Farahmand, E. Rastegar, K. Maroofi, S. Jia, M. Pomplun, N. Haspel, and A. Bayat. Human gait database for normal walk collected by smart phone accelerometer. arXiv preprint arXiv:1905.03109, 2019.

[70]

C. Wehmeyer and F. Noé. Time-lagged autoencoders: Deep learning of slow collective variables for molecular kinetics. The Journal of chemical physics, 148(24):241703, 2018.

[71]

E. Yeung, S. Kundu, and N. Hodas. Learning deep neural network representations for Koopman operators of nonlinear dynamical systems. In 2019 American Control Conference (ACC), pages 4832-4839. IEEE, 2019.

[72]

T. Zielińska. Coupled oscillators utilised as gait rhythm generators of a two-legged walking machine. Biological Cybernetics, 74(3):263-273, 1996.

Cited By

Liu JMa TSu YRong HKhalil AWahab MOsibo B(2024)Temporal patterns decomposition and Legendre projection for long-term time series forecastingThe Journal of Supercomputing10.1007/s11227-024-06313-480:16(23407-23441)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1007/s11227-024-06313-4
Liu YLi CWang JLong MOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)KoopaProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3666660(12271-12290)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3666660
Zhang ZWang XZhang ZQin ZWen WXue HLi HZhu WOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)Spectral invariant learning for dynamic graphs under distribution shiftsProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3666412(6619-6633)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3666412

Index Terms

From Fourier to Koopman: spectral methods for long-term time series prediction

Index terms have been assigned to the content through auto-classification.

Recommendations

The fast generalized discrete Fourier transforms: a unified approach to the discrete sinusoidal transforms computation
Split vector-radix fast Fourier transform

The split-radix approach for computing the discrete Fourier transform (DFT) is extended for the vector-radix fast Fourier transform (FFT) to two and higher dimensions. It is obtained by further splitting the ( N /2× N /2) transforms with twiddle factors ...
Sequency-ordered generalized Walsh-Fourier transform

A new transform family, called the sequency-ordered generalized Walsh-Fourier transform (SGWFT), is proposed in this paper. Using the kernel matrix generation process and the controllable phase quantization parameter, the Walsh-Hadamard transform (WHT), ...

Comments

Information & Contributors

Information

Published In

cover image The Journal of Machine Learning Research

The Journal of Machine Learning Research Volume 22, Issue 1

January 2021

13310 pages

ISSN:1532-4435

EISSN:1533-7928

Editors:
Francis Bach
INRIA
,
David Blei
Columbia University

Issue’s Table of Contents

Copyright © 2021.

CC-BY 4.0

Publisher

JMLR.org

Publication History

Published: 01 January 2021

Published in JMLR Volume 22, Issue 1

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
365
Total Downloads

Downloads (Last 12 months)198
Downloads (Last 6 weeks)5

Reflects downloads up to 17 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liu JMa TSu YRong HKhalil AWahab MOsibo B(2024)Temporal patterns decomposition and Legendre projection for long-term time series forecastingThe Journal of Supercomputing10.1007/s11227-024-06313-480:16(23407-23441)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1007/s11227-024-06313-4
Liu YLi CWang JLong MOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)KoopaProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3666660(12271-12290)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3666660
Zhang ZWang XZhang ZQin ZWen WXue HLi HZhu WOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)Spectral invariant learning for dynamic graphs under distribution shiftsProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3666412(6619-6633)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3666412

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents