article

Reservoir computing and extreme learning machines for non-linear time-series data analysis

Authors:

D. Verstraeten,

P. W. HaycockAuthors Info & Claims

Neural Networks, Volume 38

Pages 76 - 89

https://doi.org/10.1016/j.neunet.2012.11.011

Published: 01 February 2013 Publication History

Abstract

Random projection architectures such as Echo state networks (ESNs) and Extreme Learning Machines (ELMs) use a network containing a randomly connected hidden layer and train only the output weights, overcoming the problems associated with the complex and computationally demanding training algorithms traditionally used to train neural networks, particularly recurrent neural networks. In this study an ESN is shown to contain an antagonistic trade-off between the amount of non-linear mapping and short-term memory it can exhibit when applied to time-series data which are highly non-linear. To overcome this trade-off a new architecture, Reservoir with Random Static Projections (R^2SP) is investigated, that is shown to offer a significant improvement in performance. A similar approach using an ELM whose input is presented through a time delay (TD-ELM) is shown to further enhance performance where it significantly outperformed the ESN and R^2SP as well other architectures when applied to a novel task which allows the short-term memory and non-linearity to be varied. The hard-limiting memory of the TD-ELM appears to be best suited for the data investigated in this study, although ESN-based approaches may offer improved performance when processing data which require a longer fading memory.

References

[1]

Antonelo, E., Schrauwen, B., & Stroobandt, D. (2008). Mobile robot control in the road sign problem using reservoir computing networks. In Proceedings of the IEEE international conference on robotics and automation (pp. 911-916).

[2]

Connectivity, dynamics and memory in reservoir computing with binary and analog neurons. Neural Computation. v22 i5. 1272-1311.

[3]

Butcher, J.B. (2012). Reservoir computing with high non-linear separation and long-term memory for time-series data analysis, Ph.D. Thesis, Institute for the Environment, Physical Sciences and Applied Mathematics (EPSAM), Keele University, Staffordshire, ST5 5BG, UK.

[4]

Butcher, J.B., Verstraeten, D., Schrauwen, B., Day, C.R., & Haycock, P.W. (2010a). Extending reservoir computing with random static projections: a hybrid between OP-ELM and RC. In European symposium on artificial neural networks, ESANN (pp. 303-308).

[5]

Butcher, J.B., Verstraeten, D., Schrauwen, B., Day, C.R., & Haycock, P.W. (2010b). Pruning reservoirs with random static projections. In IEEE International workshop on machine learning for signal processing, MLSP (pp. 250-255). http://dx.doi.org/10.1109/MLSP.2010.5589251.

[6]

Butcher, J.B., Verstraeten, D., Schrauwen, B., Day, C.R., & Haycock, P.W. (2012). Defect detection in reinforced concrete using random neural architectures. In Computer aided civil and infrastructure engineering (submitted for publication).

[7]

Sales forecasting system based on Gray extreme learning machine with Taguchi method in retail industry. Expert Systems with Applications. v38 i3. 1336-1345.

[8]

Speech recognition: turning theory to practice. IEEE Spectrum. v18 i9. 26-32.

[9]

Architectural and Markovian factors of echo state networks. Neural Networks. v24 i5. 440-456.

[10]

Holzmann, G. (2009). Reservoir computing: a powerful black-box framework for nonlinear audio processing. In 12th international conference on digital audio effects, DAFx-09. Como, Italy.

[11]

Advances in extreme learning machines (elm2010). Neurocomputing. v74 i16. 2411-2412.

[12]

Extreme learning machines: theory and applications. Neurocomputing. v70 i1-3. 489-501.

[13]

Jaeger, H. (2001). The "echo state" approach to analysing and training recurrent neural networks, Tech. rep., German National Research Institute for Computer Science.

[14]

Jaeger, H. (2002a). Adaptive nonlinear system identification with echo state networks. In Neural information processing systems, NIPS.

[15]

Jaeger, H. (2002b). A tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the "echo state network" approach, Tech. Rep. GMD Report 159, German National Research Center for Information Technology.

[16]

Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science. v304 i5667. 78-80.

[17]

Optimisation and applications of echo state networks with leaky-integrator neurons. Neural Networks. v20 i3. 335-352.

[18]

Reservoir computing approaches to recurrent neural network training. Computer Science Review. v3 i3. 127-149.

[19]

. In: Probability and mathematical statistics, Wiley, USA.

[20]

On the reciprocal of the general algebraic matrix. Bulletin of The American Mathematical Society. v26. 394-395.

[21]

A generalised inverse for matrices. In: Cambridge philosophical society, Vol. 51, pp. 406-413).

[22]

Essential statistics. 2000. 4th Ed. Chapman and Hall, London, UK.

[23]

Application of extreme learning machine method for time series analysis. International Journal of Intelligent Technology. v2 i4. 256-262.

[24]

Adaptive ensemble models of extreme learning machines for time series prediction. In: Proceedings of the 19th international conference on artificial neural networks, Springer-Verlag, Berlin, Heidelberg. pp. 305-314.

[25]

Verstraeten, D. (2009). Reservoir computing: computation with dynamical systems, Ph.D. Thesis, Electronics and Information Systems department, University of Ghent, Belgium.

[26]

Verstraeten, D., Dambre, J., Dutoit, X., & Schrauwen, B. (2010). Memory versus non-linearity in reservoirs. In International joint conference on neural networks, IJCNN (pp. 2669-2676).

[27]

An experimental unification of reservoir computing methods. Neural Networks. v20 i3. 391-403.

[28]

Verstraeten, D., Schrauwen, B., & Stroobandt, D. (2006). Reservoir-based techniques for speech recognition. In Proceedings of the international joint conference on neural networks, IJCNN (pp. 1050-1053).

[29]

Isolated word recognition with a liquid state machine: a case study. Information Processing Letters. v95 i6. 521-528.

Cited By

Liu QLi FWang W(2024)Memory augmented echo state network for time series predictionNeural Computing and Applications10.1007/s00521-023-09276-436:7(3761-3776)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1007/s00521-023-09276-4
Basterrech SRubino G(2023)Evolutionary Echo State NetworkApplied Soft Computing10.1016/j.asoc.2023.110463144:COnline publication date: 1-Sep-2023
https://dl.acm.org/doi/10.1016/j.asoc.2023.110463
Yıldırım HRevan Özkale M(2021)LL-ELM: A regularized extreme learning machine based on -norm and Liu estimatorNeural Computing and Applications10.1007/s00521-021-05806-033:16(10469-10484)Online publication date: 1-Aug-2021
https://dl.acm.org/doi/10.1007/s00521-021-05806-0
Show More Cited By

Reservoir computing and extreme learning machines for non-linear time-series data analysis
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches

Recommendations

Direct model of memory properties and the linear reservoir topologies in echo state networks

Echo state networks (ESN), with a large number of randomly connected neurons (called ''reservoir'') and a simple linear output neuron, are a kind of novel recurrent neural network (RNN). One of the most important properties of ESN is short-term memory (...
The Performance Analysis of Extreme Learning Machines on Odour Recognition
ICCBDC '18: Proceedings of the 2018 2nd International Conference on Cloud and Big Data Computing

Extreme Learning Machine (ELM) is a single hidden layer feed-forward neural network learning method, which has a high generalization performance as well as faster. In this paper, odour data is discriminated based on the sensor response curve by using ...
Online time-series forecasting using spiking reservoir
Abstract
IoT-based automated systems require efficient online time-series analysis and forecasting and there is a growing requirement to enable such processing at the low-cost constrained edge devices. Classical approaches such as Online ...

Comments

Information & Contributors

Information

Published In

Copyright © Elsevier Ltd © 2012.

Publisher

Elsevier Science Ltd.

United Kingdom

Publication History

Published: 01 February 2013

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

31
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 17 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liu QLi FWang W(2024)Memory augmented echo state network for time series predictionNeural Computing and Applications10.1007/s00521-023-09276-436:7(3761-3776)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1007/s00521-023-09276-4
Basterrech SRubino G(2023)Evolutionary Echo State NetworkApplied Soft Computing10.1016/j.asoc.2023.110463144:COnline publication date: 1-Sep-2023
https://dl.acm.org/doi/10.1016/j.asoc.2023.110463
Yıldırım HRevan Özkale M(2021)LL-ELM: A regularized extreme learning machine based on -norm and Liu estimatorNeural Computing and Applications10.1007/s00521-021-05806-033:16(10469-10484)Online publication date: 1-Aug-2021
https://dl.acm.org/doi/10.1007/s00521-021-05806-0
Ma QShen LCottrell G(2020)DeePr-ESNInformation Sciences: an International Journal10.1016/j.ins.2019.09.049511:C(152-171)Online publication date: 1-Feb-2020
https://dl.acm.org/doi/10.1016/j.ins.2019.09.049
Basterrech SKrömer P(2020)A nature-inspired biomarker for mental concentration using a single-channel EEGNeural Computing and Applications10.1007/s00521-019-04574-232:12(7941-7956)Online publication date: 1-Jun-2020
https://dl.acm.org/doi/10.1007/s00521-019-04574-2
Carmichael ZSyed HKudithipudi DOkandan MAimone J(2019)Analysis of Wide and Deep Echo State Networks for Multiscale Spatiotemporal Time Series ForecastingProceedings of the 7th Annual Neuro-inspired Computational Elements Workshop10.1145/3320288.3320303(1-10)Online publication date: 26-Mar-2019
https://dl.acm.org/doi/10.1145/3320288.3320303
Kraus MFeuerriegel S(2019)Forecasting remaining useful lifeDecision Support Systems10.1016/j.dss.2019.113100125:COnline publication date: 1-Oct-2019
https://dl.acm.org/doi/10.1016/j.dss.2019.113100
Serrano W(2019)Genetic and deep learning clusters based on neural networks for management decision structuresNeural Computing and Applications10.1007/s00521-019-04231-832:9(4187-4211)Online publication date: 20-May-2019
https://dl.acm.org/doi/10.1007/s00521-019-04231-8
Ramamurthy RStenzel RSifa RLadi ABauckhage C(2019)Echo State Networks for Named Entity RecognitionArtificial Neural Networks and Machine Learning – ICANN 2019: Workshop and Special Sessions10.1007/978-3-030-30493-5_11(110-120)Online publication date: 17-Sep-2019
https://dl.acm.org/doi/10.1007/978-3-030-30493-5_11
Popov AKoprinkova-Hristova PSimov KOsenova P(2019)Echo State vs. LSTM Networks for Word Sense DisambiguationArtificial Neural Networks and Machine Learning – ICANN 2019: Workshop and Special Sessions10.1007/978-3-030-30493-5_10(94-109)Online publication date: 17-Sep-2019
https://dl.acm.org/doi/10.1007/978-3-030-30493-5_10
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents