On Chaos and Neural Networks: The Backpropagation Paradigm

Abstract

In training feed-forward neural networks using the backpropagation algorithm, a sensitivity to the values of the parameters of the algorithm hasbeen observed. In particular, it has been observed that this sensitivity with respect to the values of the parameters, such as thelearning rate, plays an important role in the final outcome. In thistutorial paper, we will look at neural networks from a dynamical systemspoint of view andexamine its properties. To this purpose, we collect results regarding chaostheory as well as the backpropagation algorithmand establish a relationship between them. We study in detail as an example the learning of the exclusive OR,an elementary Boolean function. The following conclusions hold for our XOR neural network: no chaos appears for learning rates lower than 5, when chaosoccurs, it disappears as learning progresses. For non-chaotic learning rates, the network learns faster than for other learning rates for which chaos occurs.