Chapter 3

Perceptron
and
Multilayer Perceptron

Assoc. Prof. Dr. Haza Nuzly Abdull Hamed

Assoc. Prof. Dr. Siti Zaiton Mohd Hashim
 Introduction to Perceptron
• The aim of the perceptron is to classify a set of inputs
into one of two categories (usually 1 or 0).

• If the inputs are in the form of a grid, a perceptron can be

used to recognize visual images of shapes.

• The perceptron usually uses a step function, which

returns 1 if the weighted sum of inputs exceeds a
threshold, and 0 otherwise.
Perceptron structure
 Learning Rules
1. Perceptron Learning Rule
• The perceptron training algorithm for classification
tasks.

2. Delta Rules
• For continuous activation function
• The aim of the delta rule is to minimize the error
over all training patterns
Perceptron Training Algorithm
1. Set initial weights w1, w2,…, wn and threshold θ to
random numbers. Normally [-0.5,0.5]

2. Calculate output:
n
X   wi xi
i 1

3. Apply activation functions

1 for X  
Y 
0 for X  
 Perceptron Training Algorithm
4. An item of training data is presented. If the perceptron
mis-classifies it, the weights are modified according to
the following:
wi  wi  a  xi  t  o 
where t is the target output for the training example,
o is the output generated by the preceptron and a is
the learning rate, between 0 and 1 (usually small
such as 0.1)

5. Cycle through training examples until successfully

classify all examples
Linear and Non Linear Separable

OR, AND and NOT are linearly separable

Boolean Functions

XOR is not linearly separable

Perceptron learning example: AND

Input
1 1 0 0 W1 = 0.3 Step function
Target output
n 1 for X  
X   wi xi Y 
0 for X   1 0 0 0
i 1
  0.2
1 0 1 0
W2 = -0.1

Pattern 1: (0x0.3)+(0x-0.1) = 0, Less than 0.2, output = 0 (target 0)

Pattern 2: (0x0.3)+(1x-0.1) = -0.1, Less than 0.2, output = 0 (target 0)
Pattern 3: (1x0.3)+(0x-0.1) = 0.3, Greater than 0.2, output = 1 (target 0)
i i 
Weight update: w  w  a  x  t  o
w1 = 0.3 + (0.1 x 1 x (0-1) = 0.2
i  
w2 = -0.1 + (0.1 x 0 x (0-1) = -0.1
Pattern 4: (1x0.2)+(1x-0.1) = 0.1, Less than 0.2, output = 0 (target 1)
Weight update:…..
Perceptron learning example: AND
Inputs Desired Initial Actual Error Final
Epoch output weights output weights
x1 x2 Yd w1 w2 Y e w1 w2
1 0 0 0 0.3  0.1 0 0 0.3  0.1
0 1 0 0.3  0.1 0 0 0.3  0.1
1 0 0 0.3  0.1 1 1 0.2  0.1
1 1 1 0.2  0.1 0 1 0.3 0.0
2 0 0 0 0.3 0.0 0 0 0.3 0.0
0 1 0 0.3 0.0 0 0 0.3 0.0
1 0 0 0.3 0.0 1 1 0.2 0.0
1 1 1 0.2 0.0 1 0 0.2 0.0
3 0 0 0 0.2 0.0 0 0 0.2 0.0
0 1 0 0.2 0.0 0 0 0.2 0.0
1 0 0 0.2 0.0 1 1 0.1 0.0
1 1 1 0.1 0.0 0 1 0.2 0.1
4 0 0 0 0.2 0.1 0 0 0.2 0.1
0 1 0 0.2 0.1 0 0 0.2 0.1
1 0 0 0.2 0.1 1 1 0.1 0.1
1 1 1 0.1 0.1 1 0 0.1 0.1
5 0 0 0 0.1 0.1 0 0 0.1 0.1
0 1 0 0.1 0.1 0 0 0.1 0.1
1 0 0 0.1 0.1 0 0 0.1 0.1
1 1 1 0.1 0.1 1 0 0.1 0.1
Threshold:  = 0.2; learning rate: = 0.1
 Lab Exercise 1
Implementation of Perceptron using C/C++/Python

10
Perceptron Assignment:

Using Lab Exercise 1 code, solve OR and NOT

problem using perceptron. Select your own
threshold and learning rate values.
 Introduction to
Multilayer Perceptron(MLP)
A multilayer perceptron is a feedforward neural network with
one or more hidden layers.
The network consists of:
-An input layer of source neurons,
-At least one middle or hidden layer of computational neurons,
-An output layer of computational neurons.
The input signals are propagated in a forward direction on a
layer-by-layer basis.
Introduction to MLP
 Introduction to MLP
Learning in MLP: Backpropagation
• Learning in a multilayer network proceeds the same way
as for a perceptron.
• A training set of input patterns is presented to the
network.
• The network computes its output pattern, and if there is
an error - or in other words a difference between actual
and desired output patterns - the weights are adjusted to
reduce this error.
 Introduction to MLP
• In a back-propagation neural network, the learning
algorithm has two phases.
1. A training input pattern is presented to the network
input layer. The network propagates the input pattern
from layer to layer until the output pattern is generated
by the output layer.
2. If this pattern is different from the desired output, an
error is calculated and then propagated backwards
through the network from the output layer to the input
layer. The weights are modified as the error is
propagated.
 Introduction to MLP
Input signals
1
x1
1 y1
1
2
x2 2 y2
2

i wij j wjk
xi k yk

m
n l yl
xn
Input Hidden Output
layer layer layer

Error signals
 Activation Function

1
Sigmoid activation function: f ( x) 
1  ex
 Delta Rules
Often utilized by backpropagation

The network sees how far its answer was from the actual
one and makes an appropriate adjustment to its connection
weights.

Input

Desired
Output
 Delta Rules
Backpropagation performs a gradient descent within the
solution's vector space towards a global minimum and
avoiding local minimum
 Backpropagation Algorithm
1. Calculate the outputs of all neurons in the hidden layer:
n
x   xi  wi  bias
i 1
1
O j  f ( x) 
1  ex

2. Calculate the outputs of (all) neuron(s) in the output layer:

n
x   xi  wi  bias
i 1

1
Ok  f ( x) 
1  ex
 Backpropagation Algorithm
3. Calculate output error:

 k  Ok  (1  Ok )  (t  Ok )

4. Update weight between hidden-output layer:

w( jk )    O j   k
w( jk ) (t 1)  w( jk ) t  w( jk )

5. Calculate hidden error:

n
 j  O j  (1  O j )  (  k  w( jk ) )
k 1
 Backpropagation Algorithm
6. Update weight between input-hidden layer:

w( ij )    xi   j
w( ij ) ( t 1)  w( ij ) t  w( ij )

i = Input
j = Hidden
k = Output
 Example:
Input (X) Weight (Input to Hidden) Weight (Hidden to Output)
X1 = 0.8 X1 to H1 = 0.3 H1 to O1 = 0.6
X2 = 0.5 X1 to H2 = 0.4 H2 to O1 = 0.9
X2 to H1 = 0.7
X2 to H2 = 0.9
Learning rate α = 0.6
Target Output = 1.0

0.3
0.8 X1 H1
0.6
0.7
O1 Target = 1.0
0.4
0.9
0.5 X2 H1
0.9
 Example
1. Calculate the outputs of all neurons in the hidden layer:

H1 = (0.8 x 0.3) + (0.5 x 0.7) = 0.59, Sigmoid = 1/1+2.71828^-0.59 = 0.6434

H2 = (0.8 x 0.4) + (0.5 x 0.9) = 0.77, Sigmoid = 0.6835

2. Calculate the outputs of (all) neuron(s) in the output layer:

O1 = (0.6434 x 0.6) + (0.6835 x 0.9) = 1.0012, Sigmoid = 0.7313

3. Calculate output error

Error = 0.7313 x (1-0.7313) x (1-0.7313) = 0.0528

 Example
4. Update weight between hidden-output layer :

WH1-O1 = 0.6 + (0.6 x 0.6434 x 0.0528) = 0.6204

WH2-O1 = 0.9 + (0.6 x 0.6835 x 0.0528) = 0.9217

5. Calculate hidden error

6. Update weight between input-hidden layer:
Error gradient H1 = (0.0528 x 0.6) x ((1-0.6434) x 0.6434) = 0.0073
WX1-H1 = 0.3 + (0.6 x 0.8 x 0.0073) = 0.3035
WX2-H1 = 0.7 + (0.6 x 0.5 x 0.0073) = 0.7021

Error gradient H2 = (0.0528 x 0.9) x ((1-0.6835) x 0.6835) = 0.0103

WX1-H2 = 0.4 + (0.6 x 0.8 x 0.0103) = 0.4049
WX2-H2 = 0.9 + (0.6 x 0.5 x 0.0103) = 0.9031
 Quiz
New weights:

0.3035
0.8 X1 H1
0.6204
0.7021
O1 Target = 1.0
0.4049
0.9217
0.5 X2 H1
0.9031

Execute the feed-forward network to calculate the new

output using the updated weights. Discuss the current
output compared to the previous output. Perform 2nd
iteration backpropagation and feedforward and
compare the new result with previous result.
 Example: XOR Problem
1

3
w13 1
x1 1 3 w35
w23 5

5 y5
w
w24
14 w45
x2 2 4
w24
Input 4 Output
layer layer
1
Hiddenlayer
The initial weights and threshold levels are set randomly as follows:
w13 = 0.5, w14 = 0.9, w23 = 0.4, w24 = 1.0, w35 = -1.2, w45 = 1.1
θ3 = 0.8, θ 4 = -0.1 and θ 5 = 0.3.
 Example: XOR Problem
Sum-Squared Network Error for 224 Epochs
1
10

10 0
Sum-Squared Error

10 -1

10 -2

10 -3

10 -4
0 50 100 150 200
Epoch
 Example: XOR Problem

Inputs Desired Actual Sum of

output output squared
x1 x2 yd y5 e errors
1 1 0 0.0155 0.0010
0 1 1 0.9849
1 0 1 0.9849
0 0 0 0.0175
 Applications
Abdominal Pain Prediction

Intensity Duration
Male Age Temp WBC Pain Pain
adjustable
1 20 37 10 1 1
weights

AppendicitisDiverticulitis Ulcer Pain Cholecystitis Obstruction Pancreatitis

Duodenal Non-specific
Small Bowel
Perforated
0 0 0 0 0 0
1
Applications
Voice Recognition
 Applications
Educational Loan Forecasting System
Applications
Stock Market Prediction
 Applications
Automated Industrial Inspection
Traditional vs ANN
 Lab Exercise 2
Download and install WEKA
https://www.cs.waikato.ac.nz/ml/weka/

Download dataset from UCI Machine Learning and prepare

ARFF format
https://www.cs.waikato.ac.nz/ml/weka/arff.html

36
 Lab Exercise 2
Data visualization

37
 Lab Exercise 2
Classification using MLP

38
Lab Exercise 2

39
 Lab Exercise 3
Implementation of MLP using C/C++/Python

40
 MLP Assignment 1:

From Lab Exercise 3 code, classify Iris dataset

using MLP. Explore best MLP parameter values
for Iris classification problem.
MLP Assignment 2:
Discuss other ANN architectures:

1. Hopfield Network
2. Kohonen Network
3. Self-Organizing Map (SOM)

And others….