Introduction To Neural Networks
Introduction To Neural Networks
Introduction To Neural Networks
Introduction To
Neural Networks
Prof. George Papadourakis,
Ph.D.
Part I
Introduction and
Architectures
Introduction To
Neural Networks
Slide 2
Neural Network
Techniques
Slide 3
NNs vs Computers
Digital Computers
Deductive Reasoning.
We apply known rules
to input data to
produce output.
Computation is
centralized,
synchronous, and
serial.
Memory is packetted,
literally stored, and
location addressable.
Exact.
Static connectivity.
Applicable if well
defined rules with
precise input data.
Neural Networks
Inductive Reasoning.
Given input and output
data (training
examples), we construct
the rules.
Computation is
collective,
asynchronous, and
parallel.
Memory is distributed,
internalized, short term
and content
addressable.
Fault tolerant,
redundancy, and sharing
of responsibilities.
Inexact.
Dynamic connectivity.
Slide 4
Applications off
NNs
classification
in marketing: consumer spending pattern
classification
In defence: radar and sonar image classification
In agriculture & fishing: fruit and catch grading
In medicine: ultrasound and electrocardiogram image
classification, EEGs, medical diagnosis
assessment
In engineering: product inspection monitoring and control
In defence: target tracking
In security: motion detection, surveillance image analysis
and fingerprint matching
Slide 5
Slide 6
Who is concerned
with NNs?
Slide 7
The Biological
Neuron
Slide 8
W e ig h t s
w1
w
w
2
3
O u tp u t
B ia s
a f p 1w 1 p 2 w 2 p 3w 3 b f
p iw i b
Slide 9
Activation
functions
The activation function is generally nonlinear. Linear functions are limited because
the output is simply proportional to the input.
Slide 10
Training
methods
Supervised learning
In supervised training, both the inputs and the
outputs are provided. The network then
processes the inputs and compares its
resulting outputs against the desired outputs.
Errors are then propagated back through the
system, causing the system to adjust the
weights which control the network. This
process occurs over and over as the weights
are continually tweaked. The set of data which
enables the training is called the training set.
During the training of a network the same set
of data is processed many times as the
connection weights are ever refined.
Example architectures : Multilayer perceptrons
Unsupervised learning
Slide 11
Perceptrons
Neuron Model
Architecture
Decision boundaries
Slide 12
Error Surface
Error Contour
Error surface
Bias
Weight
Slide 13
Feedforword NNs
Slide 14
An overview of the
backpropagation
1. A set of examples for training the network is assembled. Each
case consists of a problem statement (which represents the
input into the network) and the corresponding solution (which
represents the desired output from the network).
2. The input data is entered into the network via the input layer.
3. Each neuron in the network processes the input data with the
resultant values steadily "percolating" through the network,
layer by layer, until a result is generated by the output layer.
4.
Slide 15
In p u t
D e s ir e d
O u tp u t
Slide 16
Slide 17
Early stopping
Training data
Validation
data
Test data
Slide 18
Other architectures
Slide 19
Design
Conciderations
Slide 20
Slide 21
TD NNs
applications
Adaptive Filter
Prediction example
Slide 22
Auto-associative
NNs
The auto-associative neural network is a special kind of MLP in fact, it normally consists of two MLP networks connected
"back to back. The other distinguishing feature of autoassociative networks is that they are trained with a target
data set that is identical to the input data set.
Slide 23
Recurrent
Networks
Elman Networks
Hopfield
Slide 24
Self Organising
Maps (Kohonen)
Slide 25
Normalization
Normalization
x xi 2 1
m Kohonen Neurons
Connect
Connect
Connect
Inputs must be in a
hyperdimension sphere
The dimension shinks
from n to n-1. (-2,1,3)
and (-4,2,6) becomes the
same.
n Inputs +1 Synthetic
Normalization
Composite inputs
The classical
method
xi 2
i
n actual Inputs
x
xi i 1,1
z-Axis x f x
1
i
i
f
,
ormalization
2
n s f n
Slide 26
Learning procedure
wc t 1 wc t hci t x t wc t
d j oi
i
wij
Slide 27
Neighborhood
kernel function
hci t e
A
t
Bt
rc ri
2 2 t
Slide 28
Self Organizing
Maps
Slide 29
Introduction To
Neural Networks
Prof. George Papadourakis,
Ph.D.
Part II
Application
Development
And Portofolio
30
Characteristics of
NNs
Slide 31
Neural Networks
Projects Are
Different
Project planning
Project management
Project documentation
Slide 32
Developmen
t and
validation of
prototype
Optimize prototype
Data Collection
Design Prototype
Validate prototype
Implement System
Validate System
Slide 33
NNs in real
problems
Rest of System
Raw data
Pre-processing
Feature vector
Input encode
Network inputs
Neural Network
Network outputs
Output encode
Decoded outputs
Post-processing
Rest of System
Slide 34
Pre-processing
Applying a mathematical or
statistical function
Encoding textual data from a
database
Feature extraction
Principal components analysis
Waveform / Image analysis
Fibre Optic
Image
Transmission
Transmitting image without the
distortion
In addition to
transmitting data fiber
optics, they also offer
a potential for
transmitting images.
Unfortunately images
transmitted over long
distance fibre optic
cables are more
susceptible to
distortion due to
A large Japanese telecommunications company decided to
noise.
use neural computing to tackle this problem. Rather than
trying to make the transmission line as perfect and noisefree as possible, they used a neural network at the receiving
end to reconstruct the distorted image back into its original
form.
Slide 36
TV Picture
Quality Control
Assessing picture quality
One of the main quality controls in television manufacture is, a test of
picture quality when interference is present. Manufacturers have tried to
automate the tests, firstly by analysing the pictures for the different
factors that affect picture quality as seen by a customer, and then by
combining the different factors measured into an overall quality
assessment. Although the various factors can be measured accurately, it
has proved very difficult to combine them into a single measure of quality
because they interact in very complex ways.
Neural networks are well suited to problems where many factors combine
in ways that are difficult to analyse. ERA Technology Ltd, working for the
UK Radio Communications Agency, trained a neural network with the
results from a range of human assessments. A simple network proved
easy to train and achieved excellent results on new tests. The neural
network was also very fast and reported immediately
The neural system is able to
carry out the range of
required testing far more
quickly than a human
assessor, and at far lower
cost. This enables
manufacturers to increase
the sampling rate and
achieve higher quality, as
well as reducing the cost of
their current level of quality
Related Applications : Signal Analysis
control.
Testing equipment for electromagnetic compatibility (EMC)
Testing faulty equipment
Switching car radios between alternative transmitters
Slide 37
Adaptive Inverse
Control
NNs can be used in adaptive control applications.
The top block diagram shows the training of the
inverse model. Essentially, the neural network is
learning to recreate the input that created the
current output of the plant. Once properly trained,
the inverse model (which is another NN) can be used
to control the plant since it can create the necessary
control signals to create the desired system output.
Slide 38
Chemical
Manufacture
Getting the right mix
Slide 39
Stock Market
Prediction
Improving portfolio returns
A major Japanese securities company decided to user neural
computing in order to develop better prediction models. A neural
network was trained on 33 months' worth of historical data. This
data contained a variety of economic indicators such as turnover,
previous share values, interest rates and exchange rates. The
network was able to learn the complex relations between the
indicators and how they contribute to the overall prediction. Once
trained it was then in a position to make predictions based on "live"
economic indicators.
Slide 40
Oil Exploration
Getting the right signal
The vast quantities of
seismic data involved
are cluttered with
noise and are highly
dependent on the
location being
investigated. Classical
statistical analysis
techniques lose their
effectiveness when the
data is noisy and
comes from an
environment not
A neural network was trained on a
previously
set of traces selected from a
encountered. Even a
representative set of seismic records,
small improvement in
each of which had their first break
correctly identifying
signals highlighted by an expert.
first break signals
could result in a
The neural network achieves better than 95
% accuracy,
easily
considerable
return
on
outperforming existing manual and computer-based
methods.
As
investment.
well as being more accurate, the system also achieves an 88%
improvement in the time taken to identify first break signals.
Considerable cost savings have been made as a result.
Slide 41
Automated
Industrial
Inspection
Making better pizza
Slide 42
A Brief
Introduction To
Neural Networks
Prof. George Papadourakis Phd
Part III
Neural Networks
Hardware
43
Hardware vs
Software
Slide 44
Applications
of Hardware NNWs
While not yet as successful as NNWs in
software, there are in fact hardware NNW's
hard at work in the real world. For example:
OCR (Optical Character Recognition)
Voice Recognition
Traffic Monitoring
Slide 45
NNets in VLSI
Neural networks are parallel devices, but usually
is implement in traditional Von Neuman
architectures. There is also exist Hardware
implementations of NNs.Such hardware includes
digital and analog hardware chips, PC
accelerator boards, and multi-board
neurocomputers.
Digital
Slice Architectures
Multi-processor Chips
Radial Basis Functions
Other Digital Designs
Analog
Hybrid
Optical hardware
Slide 46
NNW Features
Slide 47
NeuroComputers
Examples:
VME System
VME boards in a custom cabinet run
from a UNIX host via an ethernet link.
Boards come with 1 to 4 chips and up
to two boards to give a total of 512
PE's.
Software includes a C-language
library, assembler, compiler, and a
package of NN algorithms.
Slide 48
Analog advantages:
Analog disadvantages
Slide 49
NNW Accelerator
Cards
ZISC implements a RBF architecture with RCE learning (more ZISC discussion later.)
ISA card holds to 16 ZISC036 chips, giving 576 prototype neurons.
PCI card holds up to 19 chips for 684 prototypes.
PCI card can process 165,000 patterns/sec, where patterns are 64 8-bit element
vectors.
Runs with CalSci's popular BrainMaker NNW software.
With either 4 or 8 chips (16-PE/chip) to give 64 or 128 total PEs.
Up to 2.27GCPS. See their Benchmarks
Speeds can vary depending on transfer speeds of particular machines.
Hardware and software included
Slide 50
OCNNs inVLSI
1.
2.
A Typical n-by-m
Rectangular Cellular Neural
Network contains cells that
are connected to their nearest
neighbors only.
Slide 51