10/2/2001

Soft Computing: Neural Networks

Soft Computing: Neural Networks

Outline
Introduction Categories Hopfield Net Perceptron Single Layer Multi Layer

Neural Networks
(Chapter 9)

Kai Goebel, Bill Cheetham GE Corporate Research & Development

goebel@cs.rpi.edu cheetham@cs.rpi.edu

Soft Computing: Neural Networks

Soft Computing: Neural Networks

Introduction
Human brain is superior to digital computer at many tasks + e.g., processing of visual information + robust and fault tolerant (nerve cells in the brain die every day) + flexible; adjusts to new environment + can deal with information that is sparse, imprecise, noisy, inconsistent + highly parallel + small, compact, dissipates very little power - slower in primarily (simple) arithmetic operations
3 4

Neurons

McCulloch & Pitts (1943) - simple model of neuron as a binary threshold unit

- uses step function to fire when threshold is surpassed x1 w1

x2 x3 w2 w3

Soft Computing: Neural Networks

Soft Computing: Neural Networks

Real Neurons
Real Neurons - use not even approximately threshold devices - it is assumed they use a non-linear summation method - produce a sequence of pulses (not a single output level) - do not have the same fixed delay (t-> t+1) - are not updated synchronously - amount of transmitter substance varies unpredictably
5 6

Issues
What does that leave us with? What is the best architecture? (layers, connections, activation functions, updating, # units?) How can it be programmed? (can it learn, # examples needed, time to learn, amount of supervision, real-time learning) What can it do? (how many tasks, how well, how fast, how robust, level of generalization)

Page 1

10/2/2001

Soft Computing: Neural Networks

Soft Computing: Neural Networks

Neural Nets: Categorization

Supervised Learning
Multilayer perceptrons Radial basis function networks Modular neural networks LVQ (learning vector quantization)

Supervised Neural Networks

Requirement: known input-output relations

Reinforcement Learning
Temporal Difference Learning Q-Learning

Unsupervised Learning
Competitive learning networks Kohonen self-organizing networks ART (adaptive resonant theory)
7 8

input pattern

output

Soft Computing: Neural Networks

Soft Computing: Neural Networks

Hopfield Model
Associative Memory is considered the fruit fly of this field. It illustrates in the simplest possible manner the way that collective computation can work. Store a set of patterns in such a way that when presented with a new pattern, the network responds by producing the closest stored pattern. Conventional approach: store a list of patterns, compute the Hamming distance, find the smallest, et voila!
9 10

Hopfield Network Operation

Picture is pattern; stored as attractor in the configuration space. From arbitrary starting points, one attractor will be found

Soft Computing: Neural Networks

Soft Computing: Neural Networks

Hopfield Network Operation

Picture is pattern; stored as attractor in the configuration space. From arbitrary starting points, one attractor will be found

Hopfield Architecture
- Recurrent Network - Symmetric Architecture - Evaluation until no more changes are observed i.e., network settles into local minimum config.
wij

- local minimum corresponds to energy function

11 12

E=

1 w x x 2 i , j ij i j

Page 2

10/2/2001

Soft Computing: Neural Networks

Soft Computing: Neural Networks

Hopfield Network Equations

The operative equation, i.e., the network output at each step is
yi = sgn wij x j i

Learning in Hopfield Models

Learning Rule:
( ( wijn + 1) = wijn ) + xi x j

wii = 0

where
1 if x 0 sgn( x ) = - 1 if x < 0

13

14

Soft Computing: Neural Networks

Soft Computing: Neural Networks

Hopfield Example
Learn x=[1 1 -1 -1] which gives us the weight matrix w=[0 1 -1 -1 1 -1 -1 -1 1 0]
1 wij w14=-1 w13=-1 w12=-1 w34=-1 2 3 w23=-1 w24=-1 4

Hopfield Example
Learn second pattern x=[-1 -1 1 1] which gives us the new weight matrix w=[0 2 -2 -2 2 0 -2 -2 -2 -2 0 2 -2 -2 2 0] Now lets check the slightly corrupted pattern p=[-1 -1 -1 1] which will restore the pattern y=[-1 -1 1 1] with an energy level of E=-12
16

0 -1 -1 0 -1 1

Now lets check the slightly corrupted pattern p=[1 1 -1 1] which will restore the pattern found close y=[1 1 -1 -1]
15

with an energy level of E=-6

Soft Computing: Neural Networks

Soft Computing: Neural Networks

More Complex Hopfield Examples

Reconstruction of Images

Hopfield Book Example

Character Recognition

eight exemplar patterns

17

binary images are 130x180 pixels

18

output pattern for noisy 3 input

Page 3

10/2/2001

Soft Computing: Neural Networks

Soft Computing: Neural Networks

Hopfield: Issues
- Other memories can get lost - Memories are created that were not supposed to be there - crosstalk: if there are many memories, they might interfere - no emphasis on learning; rather handcrafting to get desired properties - goes towards optimization

Perceptrons
-Rosenblatt: 1950s -Input patterns represented is binary -Single layer network can be trained easily -Output o is computed by
n o = f wi xi i =1

where wi is a (modifiable) weight xi is the input signal

20

19

is some threshold (weight of constant input) 1 if x > 0 f(.) is the activation function f ( x ) = sgn( x ) = 0 otherwise

Soft Computing: Neural Networks

Soft Computing: Neural Networks

Single-Layer Perceptrons
Network architecture
x1 x2 x3 w1 w2 w3 y = signum(wi xi + w0) w0

Single-Layer Perceptron
Example: Gender classification (according to Jang)
Network Arch. w1 w2 v (voice freq.) h w0 y v Training data

y = signum(hw1+vw2+w0) =
21 22

-1 if female 1 if male h (hair length)

Soft Computing: Neural Networks

Soft Computing: Neural Networks

Perceptron
Learning: select an input vector if the response is incorrect, modify all weights
wi = t i x i

ADALINE
single layer network (conceived by Widrow and Hoff) output is weighted linear combination of weights
o = wi xi w0
i =1 n

where ti is a target output is the learning rate

error is described as (for pattern p) E = (t o )

2 p p p

where If a set of weights for converged state exists, then a method for tuning towards convergence exists (Rosenblatt, 1962) tp is the target output op is the actual output

23

24

Page 4

10/2/2001

Soft Computing: Neural Networks

Soft Computing: Neural Networks

ADALINE
To decrease the error, the derivative wrt the weights is taken
E p = 2 t p o p xi wi

ADALINE and MADALINE

+ Simplicity of learning procedure + Distributed learning; can be performed locally at node level + on-line (pattern by pattern) learning + connect several ADALINEs to MADALINEs to deal with XOR problem + were used for noise cancellation, adaptive inverse control - only one layer; no suitable training method for multi-layer perceptron why?
26

The delta rule is:

p wi = t p o p x i

Intuitive appeal: if tp>op, boost op by increasing wixi increase wi if xi is positive

25

decrease wi is xi is negative

Soft Computing: Neural Networks

Soft Computing: Neural Networks

XOR
Minsky and Papert reported a severe shortcoming of single layer perceptrons, the XOR problem x1 x2 output 1 x O 0 0 0 0 1 1 1 0 1 0 O x 1 1 0 0 1 not linearly separable

XOR
Minsky and Papert reported a severe shortcoming of single layer perceptrons, the XOR problem x1 x2 output 1 x O 0 0 0 0 1 1 1 0 1 0 O x 1 1 0 0 1 not linearly separable
0 w1 + 0 w 2 + w0 0 w0 0 0 w1 + 1w 2 + w0 > 0 w 2 > w0 1w1 + 0 w 2 + w0 > 0 w1 > w0

27

28

1w1 + 1w 2 + w0 0 w1 + w2 w0

Soft Computing: Neural Networks

Soft Computing: Neural Networks

Enter the Dark Ages of NNs

which (together with a lack or proper training techniques for multi-layer perceptrons) all but killed interest in neural nets in the 70s and early 80s.

29

30

Page 5

10/2/2001

Soft Computing: Neural Networks

Soft Computing: Neural Networks

Multilayer Perceptrons
Two-layer perceptron
1.5

Two-Layer Perceptron: XOR

Node output as surface of their two inputs

1.5

x1 x2

+1 +1 +1 +1 0.5

x3

-1 +1 0.5

x1

+1 +1

x3 -1 +1 +1 x5

x5

x2

+1 0.5

x4

0.5

x4

31

32

note location of o and x

Soft Computing: Neural Networks

Soft Computing: Neural Networks

Multilayer Perceptrons (MLPs)

Network architecture
x1 x2 y1 y2 hyperbolic tangent or logistic function

Multi-Layer Perceptrons
-Recall the output
n o = f wi xi i =1

-and the squared error measure E = (t o ) which is amended to

2 p p p

Ek =

(
n k =1

t k p ok )

-and the activation function

f (x ) = 1 1 + ex

Learning rule:

33

or

f (x ) =

Steepest descent (Backprop) Conjugate gradient method All optim. methods using first derivative Derivative-free optim.
34

1 e x x = tanh 2 1 + e x

or

f (x ) = x

then the learning rule for each node can be derived using the chain rule...

Soft Computing: Neural Networks

Soft Computing: Neural Networks

Multi-Layer Perceptrons
-Recall the output
n o = f wi xi i =1

Backpropagation
make incremental change in the direction dE/dw to decrease the error.
(
n k =1

-and the squared error measure E = (t o ) which is amended to

2 p p p

Ek =

t k p ok )

The learning rule for each node can be derived using the chain rule... to propagate the error back through a multilayer perceptron. E
parameters

-and the activation function

f (x ) =
1 0.5 0 y

1 1 + ex

or

f (x ) =

1 e x = tanh 2 1 + e x
1 0.5 0 -0.5 -1 -10 -5 0 5 10

or

f (x ) = x

-0.5 -1 -10 0 x 10

squashing functions

35

36

w ki =
p

E p w ki

Page 6

error

10/2/2001

Soft Computing: Neural Networks

Soft Computing: Neural Networks

Back-prop procedure
1. Initialize weights to small random values 2. Choose a pattern and apply it to input layer 3. Propagate the signal forward through the network 4. Compute the deltas for the output layer 5. Compute the deltas for the preceding layers by propagating the error backwards 6. Update all weights 7. Go back to step 2 and repeat for next pattern 8. Repeat until error rate is acceptable
37 38

Step 3
Propagate signal forward through the network
Oim = g

w
j

m m 1 ij O j

until all outputs have been calculated For m=0 (input layer), the output is the pattern.

Soft Computing: Neural Networks

Soft Computing: Neural Networks

Step 4
Compute the deltas for the output layer
iM
= g hiM tip OiM

Step 5

( )[

Compute the deltas for the preceding layers by propagating the errors backwards
im1 = g him1

( )w
n j

m m ij j

by comparing the actual output O with the target output t for the pattern p considered

for m=M, M-1, M-2, until a delta has been calculated for every unit

39

40

Soft Computing: Neural Networks

Soft Computing: Neural Networks

Step 6
Use
m wij = imOm1 j

Step Size, Initial Weights

Step size: too big variable: compute error backpropagate compute error again if error bigger, reduce step size (0.5) otherwise, increase a little (1.1) Initial weights: randomize (= 0)
error

too small

to update all connections to

new wij old = wij

+ wij

41

42

Page 7

10/2/2001

Soft Computing: Neural Networks

Soft Computing: Neural Networks

Momentum
If error minimum in long narrow valley, then updating can happen to zig-zag down the valley

Overfitting
Error on learning cases
error

epoch

Error on validation cases

error

smoothes weight updating can speed learning up

43

w = w E + w prev
44

epoch

trained things that are accidental and unimportant

Soft Computing: Neural Networks

Soft Computing: Neural Networks

Local Minima
There is no guarantee that the algorithm converges to a global minimum
error

Architectures and other Techniques

Normalize weights move weights same Euclidean distance each epoch Data scaling Input scaling: allows weights to have same order of magnitude Output scaling: let target go between +- 0.9 to avoid saturation What number of nodes per layer? How many layers?

epoch

- check with different initial conditions (different weights, etc.)

45

- perturb the system (data) with noise to improve result

46

Soft Computing: Neural Networks

Soft Computing: Neural Networks

MLP Decision Boundaries

XOR 1-layer: Half planes
A B B A A B

Radial Basis Function (RBF) Networks

General

Interwined

Network architecture
x1 x2 y1 y2

2-layer: Convex

A B

B A A B

Each node is described by a bell shaped function

xc i oi = exp 2 i2
2

3-layer: Arbitrary

A B

B A A B

47

48

where ci is the center of the curve

Page 8

10/2/2001

Soft Computing: Neural Networks

Soft Computing: Neural Networks

RBF
Output: weighted sum weighted average linear combination Location of Center: Use (fuzzy) k-means clustering Size of Variance: Use knn-classifier and take average distance
x 0 0 1 1

XOR, revisited
y 0 1 0 1 output 0 1 1 0

49

50

Soft Computing: Neural Networks

Soft Computing: Neural Networks

RBF and FIS

Consider the radial basis functions:
1.0

Modular Networks
Task decomposition Local Experts Fuse information
local expert 1
O1

c1

c2

c3

Input x

and a linear combination of the output variables

y i = a i o + bi
51

local expert 2 ... local expert n

O2

weigh experts opinions by gi

fuse

On

then the response is equivalent to ...

Soft Computing: Neural Networks

53

52

last slide

Page 9