Computing Partial Functions and Fractal Decision Boundaries with Efficient Spiking Neural Nets

By Forrest Briggs 12/5/03 Harvey Mudd College

Abstract In this paper, I introduce Spiking Neural Nets (SNN), which are both more biologically plausible and powerful than multilayer perceptrons. I present an efficient way of simulating SNNs that does not require numerical integration and show that this model can compute partial functions and decision boundaries that reseble fractals.

Biological Neural Networks The brains of animals are comprised of neurons. The most pertinent structures in neurons are axons, dendrites, the cell body and synapses. The axons carry signals away from the cell body to other neurons. The dendrites receive stimuli and carry it to the cell body. The cell body is separated from the surrounding medium by a selectively permeable membrane. There is an electric potential associated with the concentration of charged ions inside the cell. When the cell receives a signal, the signal may cause it to either increase or decrease the potential. If the potential exceeds a certain threshold, the neuron fires, sending signals to each other neuron to which it connects through a synapse. Immediately after a neuron fires, its potential is lowered, which prevents it from repeatedly firing in some circumstances. The recording below shows the neural activity from 3 neurons over a period of 4 seconds

in the visual cortex of a monkey. Each vertical bar represents a potential spike and neuron firing. The neural dynamics occur on widely varying time scales in a somewhat fractal like manner. There are many suggested explanations for how neurons perform computation and represent information using pulses of neural firings. One idea that many early artificial neural network designs are based on is that information is encoded in neural firing frequency. However, recent evidence indicates that neural computations take advantage of the information stored in the precise ordering and delays between neuron firings.

Image from "Spiking neural networks, an introduction" [1]

Multilayer Feed forward Perceptron The most commonly used artificial neural network is the multilayer feed forward perceptron (MLP) model. MLPs do not accurately model the potential-threshold-fire dynamic of biological neurons. Instead each neuron has a value which is in some sense analogous to a firing frequency. The model loses a significant amount of computation power by overlooking the individual neuron firings. Information propagates linearly through layers of neurons in this model, unlike biological neural networks and more advanced artificial recurrent neural nets where neurons are densely and mutually connected in ways that create feedback and chaos. Finally, all of the dynamics of the network occur in discrete time intervals, which makes it ill suited to dealing with real time applications and again reduces its computational power compared to continuous time systems like biological neural nets. Designing an MLP requires a human choice about what activation functions to use in each layer.

Pulsed Neural Networks by Wolfgang and Maass Pulsed or spiking neural nets (SNNs) are recurrent, meaning that any neuron can be connected to any other neuron. They send pulses of energy that arrive in continuous time. Wolfgang and Maass published Pulsed Neural Networks [2] in 1999, which is at this time, the most comprehensive source on the subject. It primarily covers implementing SNNs in software and VLSI and the theory of computing with SNNs. The book shows that SNNs are able to do any computation that multilayer feed forward perceptrons can, generally with orders fewer neurons, including universal approximation of continuous functions. SNNs are also more clearly analogous to what we know about how real neurons work. All of the software implementations of SNNs described in Pulsed Neural Networks require numerical integration of the neuron potential function, which is computationally intensive.

Spiking Neural Networks We will now explore my independently devised implementation of spiking neural networks (SNNs) which is conceptually similar to the models described in Pulsed Neural Networks, but does not use numerical integration. SNNs are fully recurrent, meaning that each neuron is connected to all neurons (including itself.) Each connection from one neuron to another has an associated weight and delay (wi,j and di,j denote the weight and delay from neuron i to neuron j, respectively.) Neurons have a certain amount of activation energy, which decays as follows: dx = kx or x(t) = x 0e kt where x = activation energy, k = energy decay rate and t = time. dt

If at any time, a neuron's activation energy exceeds its activation threshold a, then it fires, sending a signal to each neuron in the net. Signals propagate such that if neuron i fires at time tfi then the signal will arrival at neuron j at time taj where taj = tfi + di,j. Since the only time a neuron's

activation energy could exceed its threshold is when a signal adds new energy to the neuron, energy decay only needs to be calculated when this occurs. When a signal sent from neuron i to neuron j arrives, neuron j's activation energy the instant before the signal arrives must be calculated with x(t) = x 0e k (t t 0) where t = the arrival time of the signal and t0 = the time when the neuron's activation energy was last updated. Then the signal energy is added to the neuron's energy such that xj = xj0 + wi,j where xj = the neuron's new energy and xj0 = the neuron's energy immediately before the signal arrives (including its decay.) Finally, if xj > a, then the neuron fires and its energy is set to xr, the neuron's recoil energy. The firing neuron sends a signal to every other neuron in the net, according to taj = tfi + di,j. Signals are stored in the signal queue, which keeps all of the unhandled signals sorted by arrival time and lets us pop the signal with the earliest arrival time. The signal queue is most efficiently implemented as a heap based priority queue data structure. For simplicity, in most cases we set the threshold a to 1, the energy decay rate k to 1 and the recoil energy xr to -1 for all neurons. Running a network of these neurons is as simple repeatedly processing the signal with the earliest arrival time. This model uses the signal queue and the fact that activation energy is always decaying when signals are not arriving to bypass numerical integration. This has the consequence that a neuron can only fire when a signal arrives, and a signal can only arrive when a neuron fires plus the delay for the signal to propagate. In the numerically integrated model, the energy from a spike does not arrive all at once, thus a neuron could fire at a time other than the firing time of another neuron and a delay. We thus cannot assume that the two models have equivalent power. See the

above figure for a comparison of activation potential in integrated and nonintegrated SNNs.

Stability and Behavior Let us now look at the behavior of some randomly generated SNNs. Using the SNN model described above, most randomly generated SNNs rapidly get out of control. Usually each time a neuron fires it causes several other neurons to fire. This results in a chain reaction, which makes the neurons fire exponentially more as time increases. Randomly generated SNNs are very unstable. The easiest solution to this is to dynamically adjust the weights of the net as it is running to prevent it from firing too much. In order to do this, we simply divide the weight by the number of unhanded signals in the signal queue plus one. Another interesting phenomenon is the complexity of the neural firing patterns that SNNs produce. Even SNNs with only 2 or 3 neurons can produce very complex behavior. Firing patterns tend to look very similar on a wide variety of time scales, much like a fractal.

Input and Output for Functions and Decision Boundaries In the neuron fires vs. time graph above from a randomly generated SNN, which looks suggestively similar to the neural activity from the monkey, each line represents the activity of one neuron in a randomly generated network of 10 neurons. I will refer to the top line as neuron 0, the 2nd line down as neuron 1 and so on. The position of each tick on the x-axis indicates the time when a neuron fired. Notice that the entire neural firing pattern arises from neuron 0 initially being forced to fire. The time scale of the graph is irrelevant since it can be dilated arbitrarily by

multiplying all of the delays in the network by a constant. We can now model the behavior of SNNs, but in order to use them to compute functions, we must consider how to represent input and output. One bit of information can be conveyed by a single neuron either firing or not firing. A real number is encoded in the time when the neuron fires. Encoding schemes based on either of these properties or neuron firing frequency over a time interval can be devised, but generally it is more efficient to work with firing times, which encode the most information. Suppose we want a neural network that computes a function f from positive real numbers to positive real numbers. This network needs one special neuron for input and one for output. Let neuron 0 be the input neuron and neuron 1 be the output neuron. The difference in the first two firing times of neuron 0 is the input, which we force to occur. The difference between the first two firing times of neuron 1 is the output. There may be other neurons besides 0 and 1 that send signals between themselves, calculating f until they cause the output neuron to fire. There are many variations on this general scheme. For example, we could force neuron 0 to fire at time 0 and neuron 1 to fire at time x, rather than firing neuron 0 at both times 0 and x. Sometimes the output neuron does not fire at all (or within a specified maximum number of firings of the entire network,) which can be interpreted as the function being undefined for that particular x value. This may or may not be desirable. If our functions are undefined in too many places, we may wish to use multiple neurons for a single output, and accept the output of whichever fires first.

Let us now examine some of the functions from the postive reals to the postive reals that can be computed in this way. Beyond 8 neurons (3 for input / output, 5 hidden,) all of the functions exhibit comparable complexity. They appear to be composed of piece wise sub domains of either y = k or y = k + x for k > 0. Clearly this is not good enough for universal approximation of continuous functions. However, if we have three outputs a, b and c, such that y = f (x) = (a(x) b(x)) c( x) , we can output functions composed of piece wise quadratic sub domains (see figure below). Since these sub domains can be very small, this may be enough for universal approximation, not just of continuos functions, but of any partial function. More testing is necessary to determine if this model and encoding are completely univeral.

We can use SNNs as classifiers as well as to compute numerical functions. Classification is encoded in exactly the same way as we encoded numerical functions, but now if the output value is greater than 0, the input is in the class, otherwise it is not (no output means the input is not in the class.) Using two input neurons instead of one allows us to classify on the cartesian plane. The decision boundaries below (blue is in the class) are from randomly generated SNNs using this scheme. Very complex decision boundaries can be generated by relatively few neurons. Decision boundaries exhibit detail on a range of scales, like a fractal. I have not found any decision boundaries that have detail on infinitely many scales, but it is possible that they are very rare. If true fractal decision boundaries can be computed by SNNs, it should be possible to find them with evolutionary algorithms and other optimization techniques. Computing any kind of function with detail on multiple scales is a difficult task for MLPs.

In an MLP network, the number of neurons determines the complexity of the decision boundaries that can be recognized. In SNNs, after a low minimum number of neurons, this does not seem to be the case. Instead, its seems that the number of times neurons are allowed to fire is the determining factor in decision boundary complexity. To recognize any fractal decision boundary such as the julia set, an MLP would require an infinite number of neurons. SNNs may be able to do this with a finite number of neurons and an increasing number of firings. This is advantageous, since it would be impossible to train an MLP with an infinite number of neurons.

Conclusion SNNs offer the next level in power and biological realism after MLPs. I have presented a model for SNNs that can be simulated efficiently in software and shown that this model is capable of computing certain partial functions and decision boundaries that resemble fractals. More testing is necessary to determine if this model is completely universal in its capabilities. It is possible to encode partial functions in the output of MLPs, but this has not been thoroughly investigated. MLPs have difficulty computing functions with detail on a large range of scales, while SNNs do not. The complexity of the functions a SNN can compute is mostly independent of the number of neurons in the net.

References 1. Jilles Vreeken, Spiking neural networks, an introduction, http://www.aiathome.com/articles/snnintro.pdf 2. Maass, Wolfgang, Pulsed Neural Networks, 1999 Cambridge, Mass. MIT Press