BEE4333 INTELLIGENT

CONTROL
Chapter 4 :
Artificial Neural Network (ANN)

Dr. Dwi Pebrianti

Ext: 6127
Contents
4.1 Basic Concept
4.2 ANN Applications
4.3 ANN Model
4.4 ANN Learning
4.5 Simple ANN
4.6 Multilayer Neural Networks &
Backpropagation Algorithm
4.1
BASIC CONCEPT
Basic Concept
ANN born from the demand of machine
learning; computer learns from experience,
examples and analogy.
Simple concept : computer attempts to
model the human brain.
Also known as parallel distributed
processors.
Why we need an intelligent processor or
computer to replace current technology?
To decide intelligently and interact accordingly.
Human Brain; biological NN
Synapses;
connection between neutrons

Axon;
sends
information
SOMA

Dendrites; SOMA
Received information

NEURON
Plasticity : Neurons heading to right answer are Learning from
strengthen and for the wrong answer is weakened. experience!
Learning

OUTPUT SIGNALS
INPUT SIGNALS

INPUT LAYER MIDDLE LAYER OUTPUT LAYER

ANN Architecture
Learning
Synapses has their own weight to express the
importance of input.
The output of a neuron might be the final
solution or the input to other networks.
ANN learns through iterated adjustment from
synapses weight.
Weight is adjusted to cope with the output
environment regarding about its network
input/output behavior.
Each neutron computes its activation level based
on the I/O numerical weights.
How to design ANN?
Decide how many neurons to be used.
How the connections between neurons
are constructed? How many layers
needed?
Which learning algorithm to be apply?
Train the ANN by initialize the weight and
update the weights from training sets.
ANN characteristics
Advantages:
A neural network can perform tasks that a linear program can not.

When an element of the neural network fails, it can continue

without any problem by their parallel nature.

A neural network learns and does not need to be reprogrammed.

It can be implemented in any application.

It can be implemented without any problem.

Disadvantages:
The neural network needs training to operate.

The architecture of a neural network is different from the

architecture of microprocessors therefore needs to be emulated.

Requires high processing time for large neural networks.
4.2
ANN APPLICATION
ANN Application
Character recognition
Image compression
Stock market prediction
Travelling salesmans problem
Medicine, electronic nose, security and
loan application
Etc.
Application : Fingerprint Recognition
INTRODUCTION
Features of finger prints
Finger print recognition system
Why neural networks?
Goal of the system
Preprocessing system
Feature extraction using neural networks
Classification
Result
Features of finger prints

Finger prints are the

unique pattern of
ridges and valleys in
every persons
fingers.
Their patterns are
permanent and
unchangeable for
whole life of a person.
They are unique and
the probability that two
fingerprints are alike is
only 1 in 1.9x10^15.
Their uniqueness is
used for identification
of a person.
 Finger print recognition
system

Image edge Ridge Thinin Feature classifi

acquisiti detecti extractio g extractio cation
on on n n

Image acquisition: the acquired image is digitalized

into 512x512 image with each pixel assigned a
particular gray scale value (raster image).
edge detection and thinning: these are preprocessing
of the image , remove noise and enhance the image.
 Finger print recognition
system
Feature extraction:
this the step where
we point out the
features such as
ridge bifurcation and
ridge endings of the
finger print with the
help of neural
network.
Classification: here a
class label is
assigned to the
image depending on
Image edge Ridge Thinin Feature classifi
the extracted
acquisiti detecti extractio g extractio cation
features.on
on n n
Why using neural networks?
Neural networks enable us to find solution
where algorithmic methods are
computationally intensive or do not exist.
There is no need to program neural
networks they learn with examples.
Neural networks offer significant speed
advantage over conventional techniques.
Preprocessing system
The first phase of finger print recognition
is to capture a image .
The image is captured using total internal
reflection of light (TIR).
The image is stored as a two dimensional array of
512x512 size, each element of array representing
a pixel and assigned a gray scale value from 256
gray scale levels.

Image edge Ridge Thinin Feature classifi

acquisiti detecti extractio g extractio cation
on on n n
 Preprocessing system
After image is captured ,noise is
removed using edge detection, ridge
extraction and thinning.
Edge detection: the edge of the image is
defined where the gray scale levels
changes greatly. also, orientation of ridges
is determined for each 32x32 block of
pixels using gray scale gradient.
Ridge extraction: ridges are extracted using
the fact that gray scale value of pixels are
maximum along the direction normal to the
ridge orientation.

Image edge Ridge Thinin Feature classifi

acquisiti detecti extractio g extractio cation
on on n n
 Preprocessing system
Thinning: the extracted ridges are
converted into skeletal structure
in which ridges are only one pixel
wide. thinning should not-
Remove isolated as well as
surrounded pixel.
Break connectedness.
Make the image shorter.

Image edge Ridge Thinin Feature classifi

acquisiti detecti extractio g extractio cation
on on n n
 Feature extraction using neural networks

Multilayer perceptron network of

three layers is trained to detect
minutiae in the thinned image.
The first layer has nine perceptrons
The hidden layer has five perceptrons
The output layer has one perceptron.
The network is trained to output 1
when the input window is centered
at the minutiae and it outputs 0
when minutiae are not present.
Feature Extraction Using
Neural Networks
Trained neural networks
are used to analyze the
image by scanning the
image with a 3x3
window.
To avoid falsely reported
features which are due
to noise
The size of scanning
window is increased to 5x5
If the minutiae are too
close to each other than
we ignore all of them.
Classification

finger prints can be

classified mainly in
four classes
depending upon their
general pattern-
Arch
Tented arch
Right loop
Left loop
Applications of Fingerprint Recognition

As finger print recognition system can be

easily embedded in any system. It is
used in-
Recognition of criminals in law enforcement bodies.
Used to provide security to cars, lockers, banks
,shops.
To differentiate between a person who has voted
and those who have not voted in govt. elections.
To count individuals.
Assignment 0
Find a system that Artificial Neural
Network can be applied.
Explain the working principle of the
system.
In which part, ANN can be applied.
Define a preliminary design of your
chosen system.
Define the preliminary step to build the
system, as shown in the fingerprint
recognition system
4.3
ANN MODEL
ANN Models
Feed-forward networks
Radial basis function networks
Recurrent networks
Echo state networks
Hopfield networks
Competitive model
Self-organizing maps
ART model
Boltzmann machine
Committee of machines
Etc.
 Feed-forward Network

x1 w1 y1

x2
w2 y2

x3
w3 y3

http://en.wikibooks.org/wiki/Artificial_Neural_Networks/Feed-Forward_Networks
 Radial Basis Function
Networks
Each hidden layer neuron represents
a basis function of the output space,
with respect to a particular center in
the input space.
The activation function chosen is
commonly a Gaussian kernel:

This kernel is centered at the point in

http://bio.felk.cvut.cz/biocmsms/index.php?page=neural-networks

3 layers the input space specified by the

weight vector. The closer the input
Hidden layer : Radial basis activation
signal is to the current weight vector,
function (RBF)
the higher the output of the neuron
Output layer of linear summation unit(s). will be. Radial basis function
Only the tap weights between the hidden networks are used commonly in
layer and the output layer are modified function approximation and series
during training. prediction.
Recurrent Networks
In contrast to feed-forward networks, it has feedback
elements that enable signals from one layer to be fed back
to a previous layer.
A simple recurrent network has three layers, an input, an
output, and a hidden layer.
A set of additional context units are added to the input
layer that receive input from the hidden layer neurons.
The feedback paths from the hidden layer to the context
units have a fixed weight of unity.
A fully recurrent network is one where every neuron
receives input from all other neurons in the system. Such
networks cannot be easily arranged into layers.
A small subset of neurons receives external input, and
another small subset produce system output.

http://en.wikibooks.org/wiki/Artificial_Neural_Networks/Recurrent_Networks
 Echo State Networks
Recurrent networks where the hidden layer
neurons are not completely connected to all
input neurons.
Known as sparsely connected networks.
Only the weights from the hidden layer to the
output layer may be altered during training.
Echo state networks are useful for matching
and reproducing specific input patterns.
Because the only tap weights modified during
training are the output layer tap weights.
Training is typically quick and computationally
efficient in comparison to other multi-layer
networks that are not sparsely connected.

http://www.scholarpedia.org/article/Echo_state_network
Hopfield Networks

 Competitive Networks

- Competitive networks are

networks where neurons
compete with one another.
- The weight vector is treated as a
"prototype", and is matched
against the input vector.
- The "winner" of each training
session is the neuron whose
weight vector is most similar to
the input vector.
http://en.wikibooks.org/wiki/Artificial_Neural_Networks/Competitive_Models
4.4
ANN LEARNING
ANN Learning
In all of the neural paradigms, the
application of
an ANN involves two phases:
(1) Learning phase (pengajaran)
(2) Recall phase (penggunaan)
In the learning phase (usually offline) the
ANN is trained until it has learned its tasks
(through the adaptation of its weights)
The recall phase is used to solve the task.
ANN Learning
An ANN solves a task when its weights are adapted
through a learning phase.
All neural networks have to be trained before they can be
used.
They are given training patterns and their weights are
adjusted iteratively until an error function is
minimized.
Once the ANN has been trained no more training is
needed.
Two types of learning prevailed in ANNs:
Supervised learning:- learning with teacher signals or targets
Unsupervised learning:- learning without the use of teacher
signals
 Supervised Learning
In supervised learning the training patterns are provided
to the ANN together with a teaching signal or target.
The difference between the ANN output and the target is
the error signal.
Initially the output of the ANN gives a large error during
the learning phase.
The error is then minimized through continuous
adaptation of the weights to solve the problem
through a learning algorithm.
In the end when the error becomes very small, the ANN is
assumed to have learned the task and training is stopped.
It can then be used to solve the task in the recall phase.
Supervised Learning

Matching the I/O pattern

Unsupervised Learning
In unsupervised learning, the ANN is trained without
teaching signals or targets.
It is only supplied with examples of the input patterns that it
will solve eventually.
The ANN usually has an auxilliary cost function which
needs to be minimized like an energy function, distance,
etc.
Usually a neuron is designated as a winner from similarities
in the input patterns through competition.
The weights of the ANN are modified where a cost
function is minimized.
At the end of the learning phase, the weights would have been
adapted in such a manner such that similar patterns are
clustered into a particular node.
ANN paradigm
There are a number of ANN paradigms developed over the
past few decades.
These ANN paradigms are mainly distinguished through
their different learning algorithms rather than their
models.
Some ANN paradigms are named after their proposer such as
Hopfield, Kohonen, etc.
Most ANNs are named after their learning algorithm such as
Backpropagation, Competitive learning, Counter propagation,
ART, etc. and some are named after their model such as BAM.

Basically a particular ANN can be divided into either a

feedforward or a feedback model and into either a supervised
or unsupervised learning mode.
ANN Classifications
ANN Performance
The performance of an ANN is described
by the figure of merit, which expresses
the number of recalled patterns when
input patterns are applied, that could be
complete, partially complete, or even
noisy.
A 100% performance in recalled patterns
means that for every trained input
stimulus signal, the ANN always
produces the desired output pattern.
ANN Performance
 Basis of ANN computing
idea
Neuron computes the input signals and compares the
result with a threshold value, .
If the input is less than , then the neuron output is -1,
otherwise +1.
Hence, the following activation function(sign function) is
used,

where X is the net weighted input to neuron, xi is the i

input value, wi is the weight of input i . n is the number of
neuron input and Y is the neuron output.
 Other types of activation function
Sign Sigmoid
Y
function function Y
+1 +1

0 X 0 X

-1 -1

Step Y Linear Y
function function
+1 +1

0 X 0 X
-1 -1
4.5
SIMPLE ANN
Simple ANN: A Perceptron
Perceptron is used to classify input
in two classes; e.g class A1 or A2.
x2
A linear separable function is
used to divide the n-dimensional
space as follows; 1

0 x1
Say, 2 inputs, then we have a
characteristics as shown on left 2
figure. is used to shift the bound.
Three dimensional states is also
possible to be view.
Simple Perceptron
Must be boolean!
Inputs
x1
w1 Linear Hard
Combiner limiter Output
/bias

x2 w2

Threshold
Learning: Classification
Learning is done by adjusting the actual output Y to
meet the desired output Yd.
Usually, the initial weight is adjust between -0.5 to
0.5. At iteration k of the training example, we have
the error
e as

If the error is positive, the weight must be decrease

and otherwise must be increase.
Perceptron
learning rule also can be obtained where

is the learning rate and 0< <1.

Training algorithm

Example
Consider truth table of AND operation
Input x1 Input x2 AND (x1
x2)
Threshold,
0 0 0 = 0.2
0 1 0 Learning rate,
= 0.1
1 0 0
1 1 1

How ANN of a single perceptron can be trained?

Consider a step activation function in this example.
Epoc Inpu Inpu Desire Initia Initia Actua Error Final Final
h t x1 t x2 d l l l ,e weig weig
output weig weig Outp ht w1 ht w2
yd ht ht w2 ut Y
w1
1 0 0 0 0.3 -0.1 0 0 0.3 -0.1
0 1 0 0.3 -0.1 0 0 0.3 -0.1
1 0 0 0.3 -0.1 1 -1 0.2 -0.1
1 1 1 0.2 -0.1 0 1 0.3 0.0

The epoch continues until the weights are converging to a steady state
values.
4.6
Multilayer Neural Networks &
Backpropagation Algorithm
Multilayer neural networks
Multilayer NN-feedforward neural
network with one or more hidden layer.
Model consists of input layer, middle or
hidden layer and an output layer.
Why hidden layer is important?
Input layer only receives input signal
Output layer only display the output
patterns.
Hidden layer process the input signals;
weight represents feature of inputs.
Multilayer NN model
Inputs Output
x1

x2

x3

1st 2nd
hidden hidden
layer layer
Multilayer Neural Network
Learning
Multilayer NN learns through a learning
algorithm; the popular one is BACK-
PROPAGATION.
The computations are similar to a simple
perceptron.
Back-propagation has two phases;
Input layer demonstrates the training input
pattern and then propagates from layer to
layer to output.
The calculation for error will notify the
system to modified the weights
Back Propagation NN
Back propagation is the learning or
training algorithm.
Each neuron must be connected to each
other.
Sigmoid function
Y is used for the network.
+1

0 X
-1

Ideal for simple Pattern Recognition and

Mapping Tasks.
Lets look for a case
xi wij wjk yi

n m l

inputs i j k output
Input signals
Error signals
Case Study

Black pixel : 1
White Pixel : 0
Network
Training Process

Set up the weight, all in the range of
Apply input pattern calculate output.
(FORWARD PASS)
Calculated output will be different with
the TARGET.
Differences between CALCULATED
OUTPUT and TARGET is equal to error.
Error will be used for updating weight.
Step of Back Propagation
Method
Step of Back Propagation
Method (ctd)
 Case Study
A

C

i : input 1, 2, 3, 4
Initial weight all 0.1
j : hidden layer A, B, C
( can be choosen -1<weight<1
k:
Training Process
1)
Find the neuron output in hidden layer

=
=

2) Find the output of and

 Training Process
3)

4)

5)

6)

is learning rate, usually 1.

When does the training
process stop?
Training process stop until the sum of
squared error for the output y is less than
a prescribed value; 0.001.
Sum of squared error : performance
indicator of the system.
The smaller, the better the system
performance.
More about back-
propagation
Different initial weights and threshold
may have different solution, but we still
have the solutions.
The decisions boundaries can be view if
we use the sign activation function.
Drawbacks of back-propagation
Not suitable for biological neurons; to
adjust the neurons weight.
Computational expensive
Slow training
XOR PROBLEM
In this example,
is not available!




Wji(t+1) = 0.1 X (-0.0035) X 0
+ (0.9 X 0)
=0

Wj01(t+1) = 0.55 + (0)

= 0.55
Wj02(t+1) = 0.15 + (0)
= 0.15
Final results of Logical Operation
XOR
Input Desired Actual Error Sum of
x1 x2 Output Output squared
errors

1 1 0 0.0155 -0.0155 0.0010

0 1 1 0.9849 0.0151
1 0 1 0.9849 0.0151
0 0 0 0.0175 -0.0175
 END
OF
CHAPTER 4