Radial Basis Function Networks

Radial basis function (RBF) networks are feed-forward networks trained using a supervised training
algorithm. They are typically configured with a single hidden layer of units whose activation function is
selected from a class of functions called basis functions. The major difference between RBF networks
and back propagation networks (that is, multi-layer perceptron trained by Back Propagation algorithm)
is the behavior of the single hidden layer.
The Structure of the RBF Networks
RBF is first introduced in the solution of the real multivariable interpolation problems. The structure of an
RBF networks in its most basic form involves three entirely different layers:

Structure of the Standard RBF network

Input layer
The input layer is made up of source nodes (sensory units) whose number is equal to the dimension p of the
input vector u.
Hidden layer
The second layer is the hidden layer which is composed of nonlinear units that are connected directly to all
of the nodes in the input layer.
The hidden units contain a basis function, which has the parameters center and width. The center of the
basis function for a node i at the hidden layer is a vector ci whose size is the as the input vector u and there is
normally a different center for each unit in the network.
i
First, the radial distance di, between the input vector u and the center of the basis function c is computed for
each unit i in the hidden layer as:

20
 Using the Euclidean distance.
The output hi of each hidden unit i is then computed by applying the basis function G to this distance

The basis function is a curve (typically a Gaussian function, the width corresponding to the variance,σi )
which has a peak at zero distance and it decreases as the distance from the center increases.

Base function
ange of theoretical and empirical studies have indicated that many properties of the interpolating
function are relatively insensitive to the precise form of the basis functions .Some of the most
commonly used basis functions are: , for the Gaussian Functions:

( ) with parameter ơ >0

( )
Output layer
th
The transformation to the hidden unit space to the output space is linear. The j output is computed as

Mathematical model
In summary, the mathematical model of the RBF network can be expressed as:

∑ | |

Where is the Euclidean distance between u and ci

Function approximation
Let y=g(u) be a given function of u, y∈R, u∈R, g:R→R, and let Gi i=1..L, be a finite set of basis
functions. The function g can be written in terms of the given basis functions as
∑

where r(u) is the residual.

The function y can be approximated as
∑
The aim is to minimize the error by setting the parameters of Gi appropriately. A possible choice for the
error definition is the L2 norm of the residual function r(u) which is defined as

21
 ∫
Approximation by RBFNN
Now, consider the single input single output RBF network shown in Figure 4. Then y can be
written as:

By the use of such a network, y can be written as

∑ | |

Where f(u) is the output of the RBFNN given in Figure and r(u) is the residual. By setting the center c i, the
variance σi , and the weight wi the error appropriately, the error can be minimized.

Figure 4: Single input, single output RBF network

N M
Whatever we discussed here for g:R→R, can be generalized to g:R →R easily by using an N input, M
output RBFNN given in figure 9.1 previously.

Data Interpolation
k k
Given input output training patterns (u ,y ), k=1,2, ..K, the aim of data interpolation is to
approximate the function y from which the data is generated. Since the function y is unknown, the
problem can be stated as a minimization problem which takes only the sample points into
consideration:
Choose wi,j and ci, i=1,2...L, j=1,2...M so as to minimize:

As an example, the output of an RBF network trained to fit the data points given in Table is given
in Figure 95.

TABLE I: 13 data points generated by using sum of three gaussians with c1=0.2000
c2=0.6000 c3=0.9000 w1=0.2000 w2=0.5000 w3=0.3000 σ=0.1000

22
 data no 1 2 3 4 5 6 7 9 10 11 12 13
x 0.0500 0.2000 0.2500 0.3000 0.4000 0.4300 0.4800 0.6000 0.7000 0.8000 0.9000 0.9500
f(x) 0.0863 0.2662 0.2362 0.1687 0.1260 0.1756 0.3290 0.6694 0.4573 0.3320 0.4063 0.3535

Figure 5 Output of the RBF network trained to fit the datapoints given in Table

Training RBF Networks

The training of a RBF network can be formulated as a nonlinear unconstrained optimization problem given
below:
Given input output training patterns (uk,yk), k=1,2, ..K, choose wi,j and ci,, i=1,2...L, j=1,2...M so as to
minimize

Note that the training problem becomes quadratic once if ci‟s (radial basis function centers) are known.

Adjusting the widths

In its simplest form, all hidden units in the RBF network have the same width or degree of sensitivity to inputs.
However, in portions of the input space where there are few patterns, it is sometime desirable to have hidden
units with a wide area of reception. Likewise, in portions of the input space, which are crowded, it might be
desirable to have very highly tuned processors with narrow reception fields. Computing these individual widths
increases the performance of the RBF network at the expense of a more complicated training process.
Adjusting the centers
Remember that in a back propagation network, all weights in all of the layers are adjusted at the same time. In
radial basis function networks, however, the weights into the hidden layer basis units are usually set before the
second layer of weights is adjusted. As the input moves away from the connection weights, the activation value
falls off. This behavior leads to the use of the term “center” for the first-layer weights. These center weights can
be computed using Kohonen feature maps, statistical methods such as K-Means clustering, or some other
means. In any case, they are then used to set the areas of sensitivity for the RBF network‟s hidden units, which
then remain fixed.
3 Adjusting the weights
Once the hidden layer weights are set, a second phase of training is used to adjust the output weights. This
process typically uses the standard steepest descent algorithm. Note that the training problem becomes
quadratic once if ci‟s (radial basis function centers) are known.

23
 Self-Organizing Maps
Topology Preserving Maps
 A topology is a system of subsets called neighborhoods.
 In a metric space (X, d), the neighborhood Oε(x) of point x can be defined as a set of points a
such that
d(x, a) < ε
 If X and Y are two topological spaces, then functions f : X → Y that preserve the topology of
X in Y are continuous functions.
 Continuous functions map some neighborhood O(x) ⊆ X into any neighborhood O(y) ⊆ Y :
f(x) = y and f(O(x)) ⊆ O(y)
 Thus, to visualise data we need a continuous function from m-dimensional space into a 2-
dimensional space:
f : Rm → R2
 Remark 1. If the data „lives‟ in a m-dimensional space with m > 3, then there A topology is a
system of subsets O X, called neighborhoods.

Self-Organising Maps
1 SOM Architecture

• A Self-Organizing Map (SOM) is an unsupervised neural network algorithm that learns a

topology preserving map, and it is used to visualize high-dimensional data.
• It uses a single layer network of artificial neurons (or nodes), which are arranged into an k × l grid
and are connected to the input space (data).
• The SOM algorithm is designed to establish a correspondence between topologies of the input
space (data) and the output lattice (aka Kohonen‟s topology preserving map).
• Each node has m weights, and together the weights represent some m-dimensional vector:
wj = (w1j, . . . , wmj) ∈ Rm .
• The weight vector w of each neuron can be compared to an m-dimensional vector
x = (x1, . . . , xm) ∈ Rm from the input space (data)
• This comparison is usually done using the Euclidean metric in the input space Rm

( ) √

24
• The nodes are arranged into an k × l grid (output space).
• The topology in the grid is defined by another metric, such as the taxi-cab
distance:

• The algorithm involves three phases: competition, adaptation and cooperation.

SOM Algorithm
1. Define the size of the output lattice (i.e. k × l).
2. Initialize the weights of the nodes (e.g. randomly)
3. Define metrics din, dout and other parameters (α, h)
4. Repeat
- Select an input vector x € Rm from data
- Find the winning node by minimizing din(x,wj)
- Adapt the weights of the winner and its neighbors
5. Until the network stabilizes

Competition
• An input vector x = (x1, . . . , xm) is compared with the weight vector wj = (w1j, . . . , wmj) of
each node by computing the distance d(x, wj):
√
.
.
.
√

• The winner is the node with the weight wj closest to the input x (i.e. shortest d(x, wj)).
• Thus, nodes „compete‟ in the sense which of the nodes wj is more „similar‟ to a given input
pattern x.
Example 1. Consider SOM with three inputs and two output nodes (A and B).
Let
wA = (2, −1, 3) , wB = (−2, 0, 1)

Find which node is the winner for the input

x = (1, −2, 2)

25
 • Node A is the winner because it is „closer‟ ( √ √ )
What if x = (−1, −2, 0)>

Adaptation
After the input x has been presented to SOM, the weights of all nodes are adapted, so that they
become more „similar‟ to the input x vector.
• The adaptation formula for node j is:
[ ]

where
– wj is the weight vector of node j ∈ [1, . . . , k × l];
– α is the learning rate coefficient;
– hij is the neighborhood of node j with respect to the winner i.

Adaptation (cont.)
To understand better the adaptation formula, let us check how the weights change for
different values of α and hij.
[ ]

- Suppose α =0 or hij= 0

[ ]

The weight does not change ( )

- Suppose α =1 or hij= 1
[ ]
The new weight is equal to the input (

Cooperation
• Although weights of all nodes are adapted, they do not adapt equally. Adaptation
depends on how close the nodes are from the winner in the output lattice.
• If the winner is node i, then the level of adaptation for node j is defined by the
neighborhood function hij = h(d(i, j)), where d(i, j) is the distance in the lattice.
• The neighborhood is defines in such a way that it is smaller as the distance d(i, j) gets
larger. For example, the Gaussian bell function

26
 • The winner „helps‟ mostly its neighbours to adapt. Note also that the winner is
adapted more than any other node (i.e. because d(i, i) = 0).
Example 2. Let α = 0.5 and h = 1, and let us adapt the winning node A from previous
example:
wA = (2, −1, 3) , x = (1, -2 ,2)
the adaptation formula:
[ ]

27