07: Regularization

6/15/2019 07_Regularization

Previous Next Index

The problem of overfitting

So far we've seen a few algorithms - work well for many applications, but can suffer from the problem
of overfitting
What is overfitting?
What is regularization and how does it help

Overfitting with linear regression

Using our house pricing example again

Fit a linear function to the data - not a great model
This is underfitting - also known as high bias
Bias is a historic/technical one - if we're fitting a straight line to the data we have a strong
preconception that there should be a linear fit
In this case, this is not correct, but a straight line can't help being straight!
Fit a quadratic function
Works well
Fit a 4th order polynomial
Now curve fit's through all five examples
Seems to do a good job fitting the training set
But, despite fitting the data we've provided very well, this is actually not such a
good model
This is overfitting - also known as high variance
Algorithm has high variance
High variance - if fitting high order polynomial then the hypothesis can basically fit
any data
Space of hypothesis is too large

To recap, if we have too many features then the learned hypothesis may give a cost function of
exactly zero
But this tries too hard to fit the training set
Fails to provide a general solution - unable to generalize (apply to new examples)
www.holehouse.org/mlclass/07_Regularization.html 1/7
 Overfitting with logistic regression
6/15/2019 07_Regularization

Same thing can happen to logistic regression

Sigmoidal function is an underfit
But a high order polynomial gives and overfitting (high variance hypothesis)

Addressing overfitting

Later we'll look at identifying when overfitting and underfitting is occurring

Earlier we just plotted a higher order function - saw that it looks "too curvy"
Plotting hypothesis is one way to decide, but doesn't always work
Often have lots of a features - here it's not just a case of selecting a degree polynomial,
but also harder to plot the data and visualize to decide what features to keep and which
to drop
If you have lots of features and little data - overfitting can be a problem
How do we deal with this?
1) Reduce number of features
Manually select which features to keep
Model selection algorithms are discussed later (good for reducing number of
features)
But, in reducing the number of features we lose some information
Ideally select those features which minimize data loss, but even so, some info
is lost
2) Regularization
Keep all features, but reduce magnitude of parameters θ
Works well when we have a lot of features, each of which contributes a bit to
predicting y

Cost function optimization for regularization

Penalize and make some of the θ parameters really small
e.g. here θ3 and θ4
www.holehouse.org/mlclass/07_Regularization.html 2/7
6/15/2019 07_Regularization

The addition in blue is a modification of our cost function to help penalize θ3 and θ4
So here we end up with θ3 and θ4 being close to zero (because the constants are massive)
So we're basically left with a quadratic function

In this example, we penalized two of the parameter values

More generally, regularization is as follows

Regularization
Small values for parameters corresponds to a simpler hypothesis (you effectively get rid
of some of the terms)
A simpler hypothesis is less prone to overfitting
Another example
Have 100 features x1, x2, ..., x100
Unlike the polynomial example, we don't know what are the high order terms
How do we pick the ones to pick to shrink?
With regularization, take cost function and modify it to shrink all the parameters
Add a term at the end
This regularization term shrinks every parameter
By convention you don't penalize θ0 - minimization is from θ1 onwards

In practice, if you include θ0 has little impact

λ is the regularization parameter
Controls a trade off between our two goals
1) Want to fit the training set well
2) Want to keep parameters small
With our example, using the regularized objective (i.e. the cost function with the
regularization term) you get a much smoother curve which fits the data and gives a much3/7
www.holehouse.org/mlclass/07_Regularization.html
 better hypothesis
6/15/2019 07_Regularization
If λ is very large we end up penalizing ALL the parameters (θ1, θ2 etc.) so all the
parameters end up being close to zero
If this happens, it's like we got rid of all the terms in the hypothesis
This results here is then underfitting
So this hypothesis is too biased because of the absence of any parameters
(effectively)
So, λ should be chosen carefully - not too big...
We look at some automatic ways to select λ later in the course

Regularized linear regression

Previously, we looked at two algorithms for linear regression
Gradient descent
Normal equation
Our linear regression with regularization is shown below

Previously, gradient descent would repeatedly update the parameters θj, where j = 0,1,2...n
simultaneously
Shown below

We've got the θ0 update here shown explicitly

This is because for regularization we don't penalize θ0 so treat it slightly differently
How do we regularize these two rules?
Take the term and add λ/m * θj
Sum for every θ (i.e. j = 0 to n)
This gives regularization for gradient descent
We can show using calculus that the equation given below is the partial derivative of the regularized
J(θ)

www.holehouse.org/mlclass/07_Regularization.html 4/7
6/15/2019 07_Regularization

The update for θj

θj gets updated to
θj - α * [a big term which also depends on θj]
So if you group the θj terms together

The term

Is going to be a number less than 1 usually

Usually learning rate is small and m is large
So this typically evaluates to (1 - a small number)
So the term is often around 0.99 to 0.95
This in effect means θj gets multiplied by 0.99
Means the squared norm of θj a little smaller
The second term is exactly the same as the original gradient descent

Regularization with the normal equation

Normal equation is the other linear regression model
Minimize the J(θ) using the normal equation
To use regularization we add a term (+ λ [n+1 x n+1]) to the equation
[n+1 x n+1] is the n+1 identity matrix

Regularization for logistic regression

We saw earlier that logistic regression can be prone to overfitting with lots of features
Logistic regression cost function is as follows;
www.holehouse.org/mlclass/07_Regularization.html 5/7
6/15/2019 07_Regularization

To modify it we have to add an extra term

This has the effect of penalizing the parameters θ1, θ2 up to θn

Means, like with linear regression, we can get what appears to be a better fitting lower order
hypothesis
How do we implement this?
Original logistic regression with gradient descent function was as follows

Again, to modify the algorithm we simply need to modify the update rule for θ1, onwards
Looks cosmetically the same as linear regression, except obviously the hypothesis is very
different

Advanced optimization of regularized linear regression

As before, define a costFunction which takes a θ parameter and gives jVal and gradient back

www.holehouse.org/mlclass/07_Regularization.html 6/7
 use fminunc
6/15/2019
Pass it an @costfunction argument07_Regularization
Minimizes in an optimized manner using the cost function
jVal
Need code to compute J(θ)
Need to include regularization term
Gradient
Needs to be the partial derivative of J(θ) with respect to θi
Adding the appropriate term here is also necessary

Ensure summation doesn't extend to to the lambda term!

It doesn't, but, you know, don't be daft!

www.holehouse.org/mlclass/07_Regularization.html 7/7
 08: Neural Networks - Representation
6/15/2019 08_Neural_Networks_Representation

Previous Next Index

Neural networks - Overview and summary

Why do we need neural networks?

Say we have a complex supervised learning classification problem

Can use logistic regression with many polynomial terms
Works well when you have 1-2 features
If you have 100 features

e.g. our housing example

100 house features, predict odds of a house being sold in the next 6 months
Here, if you included all the quadratic terms (second order)
There are lots of them (x12 ,x1x2, x1x4 ..., x1x100)
For the case of n = 100, you have about 5000 features
Number of features grows O(n2)
This would be computationally expensive to work with as a feature set
A way around this to only include a subset of features
However, if you don't have enough features, often a model won't let you fit a complex
dataset
If you include the cubic terms
e.g. (x12x2, x1x2x3, x1x4x23 etc)
There are even more features grows O(n3)
About 170 000 features for n = 100
Not a good way to build classifiers when n is large

Example: Problems where n is large - computer vision

Computer vision sees a matrix of pixel intensity values

Look at matrix - explain what those numbers represent
To build a car detector
Build a training set of
Not cars
Cars
Then test against a car
www.holehouse.org/mlclass/08_Neural_Networks_Representation.html 1/14
 How can we do this
6/15/2019 08_Neural_Networks_Representation
Plot two pixels (two pixel locations)
Plot car or not car on the graph

Need a non-linear hypothesis to separate the classes

Feature space
If we used 50 x 50 pixels --> 2500 pixels, so n = 2500
If RGB then 7500
If 100 x 100 RB then --> 50 000 000 features
Too big - wayyy too big
So - simple logistic regression here is not appropriate for large complex systems
Neural networks are much better for a complex nonlinear hypothesis even when feature
space is huge

Neurons and the brain

Neural networks (NNs) were originally motivated by looking at machines which replicate
the brain's functionality
Looked at here as a machine learning technique
Origins
To build learning systems, why not mimic the brain?
Used a lot in the 80s and 90s
Popularity diminished in late 90s
Recent major resurgence
NNs are computationally expensive, so only recently large scale neural networks
became computationally feasible
Brain
Does loads of crazy things
Hypothesis is that the brain has a single learning algorithm
Evidence for hypothesis
Auditory cortex --> takes sound signals
If you cut the wiring from the ear to the auditory cortex
Re-route optic nerve to the auditory cortex
Auditory cortex learns to see
Somatosensory context (touch processing)
If you rewrite optic nerve to somatosensory cortex then it learns to see
With different tissue learning to see, maybe they all learn in the same way
Brain learns by itself how to learn
Other examples
www.holehouse.org/mlclass/08_Neural_Networks_Representation.html 2/14
 Seeing with your tongue
6/15/2019 08_Neural_Networks_Representation
Brainport
Grayscale camera on head
Run wire to array of electrodes on tongue
Pulses onto tongue represent image signal
Lets people see with their tongue
Human echolocation
Blind people being trained in schools to interpret sound and echo
Lets them move around
Haptic belt direction sense
Belt which buzzes towards north
Gives you a sense of direction
Brain can process and learn from data from any source

Model representation 1
How do we represent neural networks (NNs)?
Neural networks were developed as a way to simulate networks of neurones
What does a neurone look like

Three things to notice

Cell body
Number of input wires (dendrites)
Output wire (axon)
Simple level
Neurone gets one or more inputs through dendrites
Does processing
Sends output down axon
Neurons communicate through electric spikes
Pulse of electricity via axon to another neurone

Artificial neural network - representation of a neurone

In an artificial neural network, a neurone is a logistic unit

Feed input via input wires
Logistic unit does computation
Sends output down output wires
That logistic computation is just like our previous logistic regression hypothesis calculation
www.holehouse.org/mlclass/08_Neural_Networks_Representation.html 3/14
6/15/2019 08_Neural_Networks_Representation

Very simple model of a neuron's computation

Often good to include an x0 input - the bias unit
This is equal to 1
This is an artificial neurone with a sigmoid (logistic) activation function
Ɵ vector may also be called the weights of a model
The above diagram is a single neurone
Below we have a group of neurones strung together

Here, input is x1, x2 and x3

We could also call input activation on the first layer - i.e. (a11, a21 and a31 )
Three neurones in layer 2 (a12, a22 and a32 )
Final fourth neurone which produces the output
Which again we *could* call a13
First layer is the input layer
Final layer is the output layer - produces value computed by a hypothesis
Middle layer(s) are called the hidden layers
You don't observe the values processed in the hidden layer
Not a great name
Can have many hidden layers

Neural networks - notation

www.holehouse.org/mlclass/08_Neural_Networks_Representation.html 4/14
 ai(j) - activation of unit i in layer j
6/15/2019 08_Neural_Networks_Representation
So, a12 - is the activation of the 1st unit in the second layer
By activation, we mean the value which is computed and output by that node
(j)
Ɵ - matrix of parameters controlling the function mapping from layer j to layer j + 1
Parameters for controlling mapping from one layer to the next
If network has
sj units in layer j and
sj+1 units in layer j + 1
Then Ɵj will be of dimensions [sj+1 X sj + 1]
Because
sj+1 is equal to the number of units in layer (j + 1)
is equal to the number of units in layer j, plus an additional unit
Looking at the Ɵ matrix
Column length is the number of units in the following layer
Row length is the number of units in the current layer + 1 (because we have to map
the bias unit)
So, if we had two layers - 101 and 21 units in each
Then Ɵj would be = [21 x 102]
What are the computations which occur?
We have to calculate the activation for each node
That activation depends on
The input(s) to the node
The parameter associated with that node (from the Ɵ vector associated with that
layer)
Below we have an example of a network, with the associated calculations for the four nodes
below

As you can see

www.holehouse.org/mlclass/08_Neural_Networks_Representation.html 5/14
 We calculate each of the layer-2 activations based on the input values with the bias term
6/15/2019 08_Neural_Networks_Representation
(which is equal to 1)
i.e. x0 to x3
We then calculate the final hypothesis (i.e. the single node in layer 3) using exactly the
same logic, except in input is not x values, but the activation values from the preceding
layer
The activation value on each hidden unit (e.g. a12 ) is equal to the sigmoid function applied to
the linear combination of inputs
Three input units
So Ɵ(1) is the matrix of parameters governing the mapping of the input units to
hidden units
Ɵ(1) here is a [3 x 4] dimensional matrix
Three hidden units
Then Ɵ(2) is the matrix of parameters governing the mapping of the hidden layer to
the output layer
Ɵ(2) here is a [1 x 4] dimensional matrix (i.e. a row vector)
One output unit
Something conceptually important (that I hadn't really grasped the first time) is that
Every input/activation goes to every node in following layer
Which means each "layer transition" uses a matrix of parameters with the following
significance
For the sake of consistency with later nomenclature, we're using j,i and l as our
variables here (although later in this section we use j to show the layer we're
on)
Ɵjil
j (first of two subscript numbers)= ranges from 1 to the number of units in
layer l+1
i (second of two subscript numbers) = ranges from 0 to the number of
units in layer l
l is the layer you're moving FROM
This is perhaps more clearly shown in my slightly over the top example below

www.holehouse.org/mlclass/08_Neural_Networks_Representation.html 6/14
6/15/2019 08_Neural_Networks_Representation

For example
Ɵ131 = means
1 - we're mapping to node 1 in layer l+1
3 - we're mapping from node 3 in layer l
1 - we're mapping from layer 1

Model representation II
Here we'll look at how to carry out the computation efficiently through a vectorized
implementation. We'll also consider
why NNs are good and how we can use them to learn complex non-linear things

Below is our original problem from before

Sequence of steps to compute output of hypothesis are the equations below

www.holehouse.org/mlclass/08_Neural_Networks_Representation.html 7/14
6/15/2019 08_Neural_Networks_Representation

Define some additional terms

z12 = Ɵ101x0 + Ɵ111x1 + Ɵ121x2 + Ɵ131x3
Which means that
a12 = g(z12)
NB, superscript numbers are the layer associated
Similarly, we define the others as
z22 and z32
These values are just a linear combination of the values
If we look at the block we just redefined
We can vectorize the neural network computation
So lets define
x as the feature vector x
z2 as the vector of z values from the second layer

z2 is a 3x1 vector
We can vectorize the computation of the neural network as as follows in two steps
z2 = Ɵ(1)x
i.e. Ɵ(1) is the matrix defined above
x is the feature vector
a = g(z(2))
2

To be clear, z2 is a 3x1 vecor

a2 is also a 3x1 vector
www.holehouse.org/mlclass/08_Neural_Networks_Representation.html 8/14
 g() applies the sigmoid (logistic) function element wise to each member of
6/15/2019 08_Neural_Networks_Representation
the z2 vector
To make the notation with input layer make sense;
a1 = x
a1 is the activations in the input layer
Obviously the "activation" for the input layer is just the input!
So we define x as a1 for clarity
So
a1 is the vector of inputs
a2 is the vector of values calculated by the g(z2) function
Having calculated then z2 vector, we need to calculate a02 for the final hypothesis calculation

To take care of the extra bias unit add a02 = 1

So add a02 to a2 making it a 4x1 vector
So,
z3 = Ɵ2a2
This is the inner term of the above equation
hƟ(x) = a3 = g(z3)
This process is also called forward propagation
Start off with activations of input unit
i.e. the x vector as input
Forward propagate and calculate the activation of each layer sequentially
This is a vectorized version of this implementation

Neural networks learning its own features

Diagram below looks a lot like logistic regression

Layer 3 is a logistic regression node

The hypothesis output = g(Ɵ102 a02 + Ɵ112 a12 + Ɵ122 a22 + Ɵ132 a32)
www.holehouse.org/mlclass/08_Neural_Networks_Representation.html 9/14
 This is just logistic regression
6/15/2019 08_Neural_Networks_Representation
The only difference is, instead of input a feature vector, the features are just
values calculated by the hidden layer
The features a12, a22, and a32 are calculated/learned - not original features
So the mapping from layer 1 to layer 2 (i.e. the calculations which generate the a2 features) is
determined by another set of parameters - Ɵ1
So instead of being constrained by the original input features, a neural network can learn
its own features to feed into logistic regression
Depending on the Ɵ1 parameters you can learn some interesting things
Flexibility to learn whatever features it wants to feed into the final logistic
regression calculation
So, if we compare this to previous logistic regression, you would have to
calculate your own exciting features to define the best way to classify or
describe something
Here, we're letting the hidden layers do that, so we feed the hidden layers our
input values, and let them learn whatever gives the best final result to feed into
the final output layer
As well as the networks already seen, other architectures (topology) are possible
More/less nodes per layer
More layers
Once again, layer 2 has three hidden units, layer 3 has 2 hidden units by the time you get
to the output layer you get very interesting non-linear hypothesis

Some of the intuitions here are complicated and hard to understand

In the following lectures we're going to go though a detailed example to understand how
to do non-linear analysis

Neural network example - computing a complex,

nonlinear function of the input
Non-linear classification: XOR/XNOR
x1, x2 are binary

www.holehouse.org/mlclass/08_Neural_Networks_Representation.html 10/14
6/15/2019 08_Neural_Networks_Representation

Example on the right shows a simplified version of the more complex problem we're dealing
with (on the left)
We want to learn a non-linear decision boundary to separate the positive and negative
examples

y = x1 XOR x2
x1 XNOR x2

Where XNOR = NOT (x1 XOR x2)

Positive examples when both are true and both are false
Let's start with something a little more straight forward...
Don't worry about how we're determining the weights (Ɵ values) for now - just get
a flavor of how NNs work

Neural Network example 1: AND function

Simple first example

Can we get a one-unit neural network to compute this logical AND function? (probably...)
Add a bias unit
Add some weights for the networks
What are weights?
Weights are the parameter values which multiply into the input nodes (i.e. Ɵ)

www.holehouse.org/mlclass/08_Neural_Networks_Representation.html 11/14
6/15/2019 08_Neural_Networks_Representation

Sometimes it's convenient to add the weights into the diagram

These values are in fact just the Ɵ parameters so
Ɵ101 = -30
Ɵ111 = 20
Ɵ121 = 20
To use our original notation
Look at the four input values

So, as we can see, when we evaluate each of the four possible input, only (1,1) gives a positive
output

Neural Network example 2: NOT function

How about negation?

www.holehouse.org/mlclass/08_Neural_Networks_Representation.html 12/14
 Negation is achieved by putting a large negative weight in front of the variable you want to
6/15/2019 08_Neural_Networks_Representation
negative

Neural Network example 3: XNOR function

So how do we make the XNOR function work?

XNOR is short for NOT XOR
i.e. NOT an exclusive or, so either go big (1,1) or go home (0,0)
So we want to structure this so the input which produce a positive output are
AND (i.e. both true)
OR
Neither (which we can shortcut by saying not only one being true)
So we combine these into a neural network as shown below;

Simplez!

Neural network intuition - handwritten digit classification

Yann LeCun = machine learning pioneer

Early machine learning system was postcode reading
Hilarious music, impressive demonstration!

Multiclass classification
Multiclass classification is, unsurprisingly, when you distinguish between more than two
categories (i.e. more than 1 or 0)
With handwritten digital recognition problem - 10 possible categories (0-9)
How do you do that?
Done using an extension of one vs. all classification
Recognizing pedestrian, car, motorbike or truck
Build a neural network with four output units
Output a vector of four numbers
www.holehouse.org/mlclass/08_Neural_Networks_Representation.html 13/14
 1 is 0/1 pedestrian
6/15/2019 08_Neural_Networks_Representation
2 is 0/1 car
3 is 0/1 motorcycle
4 is 0/1 truck
When image is a pedestrian get [1,0,0,0] and so on
Just like one vs. all described earlier
Here we have four logistic regression classifiers

Training set here is images of our four classifications

While previously we'd written y as an integer {1,2,3,4}
Now represent y as

www.holehouse.org/mlclass/08_Neural_Networks_Representation.html 14/14
 09: Neural Networks - Learning
6/15/2019 09_Neural_Networks_Learning

Previous Next Index

Neural network cost function

NNs - one of the most powerful learning algorithms
Is a learning algorithm for fitting the derived parameters given a training set
Let's have a first look at a neural network cost function
Focus on application of NNs for classification problems
Here's the set up
Training set is {(x1, y1), (x2, y2), (x3, y3) ... (xn, ym)
L = number of layers in the network
In our example below L = 4
sl = number of units (not counting bias unit) in layer l

So here
l =4
s1 = 3
s2 = 5
s3 = 5
s4 = 4

Types of classification problems with NNs

Two types of classification, as we've previously seen

Binary classification
1 output (0 or 1)
So single output node - value is going to be a real number
k=1
NB k is number of units in output layer
sL = 1
Multi-class classification
k distinct classifications
Typically k is greater than or equal to three
If only two just go for binary
sL = k
So y is a k-dimensional vector of real numbers

Cost function for neural networks

The (regularized) logistic regression cost function is as follows;

For neural networks our cost function is a generalization of this equation above, so instead of one output we generate k outputs

Our cost function now outputs a k dimensional vector

www.holehouse.org/mlclass/09_Neural_Networks_Learning.html 1/11
 hƟ(x) is a k dimensional vector, so hƟ(x)i refers to the ith value in that vector
6/15/2019 Costfunction J(Ɵ) is 09_Neural_Networks_Learning
[-1/m] times a sum of a similar term to which we had for logic regression
But now this is also a sum from k = 1 through to K (K is number of output nodes)
Summation is a sum over the k output units - i.e. for each of the possible classes
So if we had 4 output units then the sum is k = 1 to 4 of the logistic regression over each of the four output units in turn
This looks really complicated, but it's not so difficult
We don't sum over the bias terms (hence starting at 1 for the summation)
Even if you do and end up regularizing the bias term this is not a big problem
Is just summation over the terms

Woah there - lets take a second to try and understand this!

There are basically two halves to the neural network logistic regression cost function

First half

This is just saying

For each training data example (i.e. 1 to m - the first summation)
Sum for each position in the output vector
This is an average sum of logistic regression

Second half

This is a massive regularization summation term, which I'm not going to walk through, but it's a
fairly straightforward triple nested summation
This is also called a weight decay term
As before, the lambda value determines the important of the two halves

The regularization term is similar to that in logistic regression

So, we have a cost function, but how do we minimize this bad boy?!

Summary of what's about to go down

The following section is, I think, the most complicated thing in the course, so I'm going to take a second to explain the general idea of
what we're going to do;

We've already described forward propagation

This is the algorithm which takes your neural network and the initial input into that network and pushes the input through the
network
It leads to the generation of an output hypothesis, which may be a single real number, but can also be a vector
We're now going to describe back propagation
Back propagation basically takes the output you got from your network, compares it to the real value (y) and calculates how
wrong the network was (i.e. how wrong the parameters were)
It then, using the error you've just calculated, back-calculates the error associated with each unit from the preceding layer (i.e.
layer L - 1)
This goes on until you reach the input layer (where obviously there is no error, as the activation is the input)
These "error" measurements for each unit can be used to calculate the partial derivatives
Partial derivatives are the bomb, because gradient descent needs them to minimize the cost function
We use the partial derivatives with gradient descent to try minimize the cost function and update all the Ɵ values
This repeats until gradient descent reports convergence

A few things which are good to realize from the get go

There is a Ɵ matrix for each layer in the network
This has each node in layer l as one dimension and each node in l+1 as the other dimension
Similarly, there is going to be a Δ matrix for each layer
This has each node as one dimension and each training data example as the other

Back propagation algorithm

We previously spoke about the neural network cost function
Now we're going to deal with back propagation
www.holehouse.org/mlclass/09_Neural_Networks_Learning.html 2/11
 Algorithm used to minimize the cost function, as it allows us to calculate partial derivatives!
6/15/2019 09_Neural_Networks_Learning

The cost function used is shown above

We want to find parameters Ɵ which minimize J(Ɵ)
To do so we can use one of the algorithms already described such as
Gradient descent
Advanced optimization algorithms
To minimize a cost function we just write code which computes the following
J(Ɵ)
i.e. the cost function itself!
Use the formula above to calculate this value, so we've done that
Partial derivative terms
So now we need some way to do that
This is not trivial! Ɵ is indexed in three dimensions because we have separate parameter values for each node in each
layer going to each node in the following layer
i.e. each layer has a Ɵ matrix associated with it!
We want to calculate the partial derivative Ɵ with respect to a single parameter

Remember that the partial derivative term we calculate above is a REAL number (not a vector or a matrix)
Ɵ is the input parameters
Ɵ1 is the matrix of weights which define the function mapping from layer 1 to layer 2
Ɵ101 is the real number parameter which you multiply the bias unit (i.e. 1) with for the bias unit input into the
first unit in the second layer
Ɵ111 is the real number parameter which you multiply the first (real) unit with for the first input into the first
unit in the second layer
Ɵ211 is the real number parameter which you multiply the first (real) unit with for the first input into the second
unit in the second layer
As discussed, Ɵijl i
i here represents the unit in layer l+1 you're mapping to (destination node)
j is the unit in layer l you're mapping from (origin node)
l is the layer your mapping from (to layer l+1) (origin layer)
NB
The terms destination node, origin node and origin layer are terms I've made up!
So - this partial derivative term is
The partial derivative of a 3-way indexed dataset with respect to a real number (which is one of the values in that
dataset)
Gradient computation
One training example
Imagine we just have a single pair (x,y) - entire training set
How would we deal with this example?
The forward propagation algorithm operates as follows
Layer 1
a1 = x
z2 = Ɵ1a1
Layer 2
a2 = g(z2) (add a02)
z3 = Ɵ2a2
Layer 3
a3 = g(z3) (add a03)
z4 = Ɵ3a3
Output
a4 = hƟ(x) = g(z4)

www.holehouse.org/mlclass/09_Neural_Networks_Learning.html 3/11
6/15/2019 09_Neural_Networks_Learning

This is the vectorized implementation of forward propagation

Lets compute activation values sequentially (below just re-iterates what we had above!)

What is back propagation?

Use it to compute the partial derivatives

Before we dive into the mechanics, let's get an idea regarding the intuition of the algorithm
For each node we can calculate (δjl) - this is the error of node j in layer l
If we remember, ajl is the activation of node j in layer l
Remember the activation is a totally calculated value, so we'd expect there to be some error compared to the "real" value
The delta term captures this error
But the problem here is, "what is this 'real' value, and how do we calculate it?!"
The NN is a totally artificial construct
The only "real" value we have is our actual classification (our y value) - so that's where we start
If we use our example and look at the fourth (output) layer, we can first calculate
δj4 = aj4 - yj
[Activation of the unit] - [the actual value observed in the training example]
We could also write aj4 as hƟ(x)j
Although I'm not sure why we would?
This is an individual example implementation
Instead of focussing on each node, let's think about this as a vectorized problem
δ4 = a4 - y
So here δ4 is the vector of errors for the 4th layer
a4 is the vector of activation values for the 4th layer
4
With δ calculated, we can determine the error terms for the other layers as follows;

Taking a second to break this down

Ɵ3 is the vector of parameters for the 3->4 layer mapping
δ4 is (as calculated) the error vector for the 4th layer
g'(z3) is the first derivative of the activation function g evaluated by the input values given by z3
You can do the calculus if you want (...), but when you calculate this derivative you get
g'(z3) = a3 . * (1 - a3)
So, more easily
δ3 = (Ɵ3)T δ4 . *(a3 . * (1 - a3))
. * is the element wise multiplication between the two vectors
Why element wise? Because this is essentially an extension of individual values in a vectorized implementation,
so element wise multiplication gives that effect
We highlighted it just in case you think it's a typo!
www.holehouse.org/mlclass/09_Neural_Networks_Learning.html 4/11
 Analyzing the mathematics
6/15/2019 09_Neural_Networks_Learning

And if we take a second to consider the vector dimensionality (with our example above [3-5-5-4])
Ɵ3 = is a matrix which is [4 X 5] (if we don't include the bias term, 4 X 6 if we do)
(Ɵ3)T = therefore, is a [5 X 4] matrix
4
δ = is a 4x1 vector
So when we multiply a [5 X 4] matrix with a [4 X 1] vector we get a [5 X 1] vector
Which, low and behold, is the same dimensionality as the a3 vector, meaning we can run our pairwise multiplication

For δ3 when you calculate the derivative terms you get

a3 . * (1 - a3)
Similarly For δ2 when you calculate the derivative terms you get
a2 . * (1 - a2)
So to calculate δ2 we do
δ2 = (Ɵ2)T δ3 . *(a2 . * (1 - a2))
There's no δ1 term
Because that was the input!

Why do we do this?

We do all this to get all the δ terms, and we want the δ terms because through a very complicated derivation you can use δ to get the
partial derivative of Ɵ with respect to individual parameters (if you ignore regularization, or regularization is 0, which we deal with
later)

= ajl δi(l+1)

By doing back propagation and computing the delta terms you can then compute the partial derivative terms
We need the partial derivatives to minimize the cost function!

Putting it all together to get the partial derivatives!

What is really happening - lets look at a more complex example

Training set of m examples

First, set the delta values

Set equal to 0 for all values

Eventually these Δ values will be used to compute the partial derivative
Will be used as accumulators for computing the partial derivatives
Next, loop through the training set

i.e. for each example in the training set (dealing with each example as (x,y)
Set a1 (activation of input layer) = xi
Perform forward propagation to compute al for each layer (l = 1,2, ... L)
i.e. run forward propagation
Then, use the output label for the specific example we're looking at to calculate δL where δL = aL - yi
So we initially calculate the delta value for the output layer
Then, using back propagation we move back through the network from layer L-1 down to layer
Finally, use Δ to accumulate the partial derivative terms

Note here
l = layer
j = node in that layer
i = the error of the affected node in the target layer
www.holehouse.org/mlclass/09_Neural_Networks_Learning.html 5/11
 You can vectorize the Δ expression too, as
6/15/2019 09_Neural_Networks_Learning

Finally
After executing the body of the loop, exit the for loop and compute

When j = 0 we have no regularization term

At the end of ALL this
You've calculated all the D terms above using Δ
NB - each D term above is a real number!
We can show that each D is equal to the following

We have calculated the partial derivative for each parameter

We can then use these in gradient descent or one of the advanced optimization algorithms
Phew!
What a load of hassle!

Back propagation intuition

Some additionally back propagation notes
In case you found the preceding unclear, which it shouldn't be as it's fairly heavily modified with my own explanatory notes
Back propagation is hard(ish...)
But don't let that discourage you
It's hard in as much as it's confusing - it's not difficult, just complex
Looking at mechanical steps of back propagation

Forward propagation with pictures!

Feeding input into the input layer (xi, yi)

Note that x and y here are vectors from 1 to n where n is the number of features
So above, our data has two features (hence x1 and x2)
With out input data present we use forward propagation

www.holehouse.org/mlclass/09_Neural_Networks_Learning.html 6/11
6/15/2019 09_Neural_Networks_Learning

The sigmoid function applied to the z values gives the activation values
Below we show exactly how the z value is calculated for an example

Back propagation

With forwardprop done we move on to do back propagation

Back propagation is doing something very similar to forward propagation, but backwards
Very similar though
Let's look at the cost function again...
Below we have the cost function if there is a single output (i.e. binary classification)

This function cycles over each example, so the cost for one example really boils down to this

Which, we can think of as a sigmoidal version of the squared difference (check out the derivation if you don't believe me)
So, basically saying, "how well is the network doing on example i "?
We can think about a δ term on a unit as the "error" of cost for the activation value associated with a unit
More formally (don't worry about this...), δ is

Where cost is as defined above

Cost function is a function of y value and the hypothesis function
So - for the output layer, back propagation sets the δ value as [a - y]
Difference between activation and actual value
We then propagate these values backwards;

www.holehouse.org/mlclass/09_Neural_Networks_Learning.html 7/11
6/15/2019 09_Neural_Networks_Learning

Looking at another example to see how we actually calculate the delta value;

So, in effect,
Back propagation calculates the δ, and those δ values are the weighted sum of the next layer's delta values, weighted by
the parameter associated with the links
Forward propagation calculates the activation (a) values, which
Depending on how you implement you may compute the delta values of the bias values
However, these aren't actually used, so it's a bit inefficient, but not a lot more!

Implementation notes - unrolling parameters (matrices)

Needed for using advanced optimization routines

Is the MATLAB/octave code

But theta is going to be matrices
fminunc takes the costfunction and initial theta values
These routines assume theta is a parameter vector
Also assumes the gradient created by costFunction is a vector
For NNs, our parameters are matrices
e.g.

Example

www.holehouse.org/mlclass/09_Neural_Networks_Learning.html 8/11
6/15/2019 09_Neural_Networks_Learning

Use the thetaVec = [ Theta1(:); Theta2(:); Theta3(:)]; notation to unroll the matrices into a long vector
To go back you use
Theta1 = resape(thetaVec(1:110), 10, 11)

Gradient checking
Backpropagation has a lot of details, small bugs can be present and ruin it :-(
This may mean it looks like J(Ɵ) is decreasing, but in reality it may not be decreasing by as much as it should
So using a numeric method to check the gradient can help diagnose a bug
Gradient checking helps make sure an implementation is working correctly
Example
Have an function J(Ɵ)
Estimate derivative of function at point Ɵ (where Ɵ is a real number)
How?
Numerically
Compute Ɵ + ε
Compute Ɵ - ε
Join them by a straight line
Use the slope of that line as an approximation to the derivative

Usually, epsilon is pretty small (0.0001)

If epsilon becomes REALLY small then the term BECOMES the slopes derivative
The is the two sided difference (as opposed to one sided difference, which would be J(Ɵ + ε) - J(Ɵ) /ε
If Ɵ is a vector with n elements we can use a similar approach to look at the partial derivatives

So, in octave we use the following code the numerically compute the derivatives

www.holehouse.org/mlclass/09_Neural_Networks_Learning.html 9/11
 So on each loop thetaPlus = theta except for thetaPlus(i)
6/15/2019 Resets thetaPlus on each loop 09_Neural_Networks_Learning
Create a vector of partial derivative approximations
Using the vector of gradients from backprop (DVec)
Check that gradApprox is basically equal to DVec
Gives confidence that the Backproc implementation is correc
Implementation note
Implement back propagation to compute DVec
Implement numerical gradient checking to compute gradApprox
Check they're basically the same (up to a few decimal places)
Before using the code for learning turn off gradient checking
Why?
GradAprox stuff is very computationally expensive
In contrast backprop is much more efficient (just more fiddly)

Random initialization
Pick random small initial values for all the theta values
If you start them on zero (which does work for linear regression) then the algorithm fails - all activation values for each layer
are the same
So chose random values!
Between 0 and 1, then scale by epsilon (where epsilon is a constant)

Putting it all together

1) - pick a network architecture
Number of
Input units - number of dimensions x (dimensions of feature vector)
Output units - number of classes in classification problem
Hidden units
Default might be
1 hidden layer
Should probably have
Same number of units in each layer
Or 1.5-2 x number of input features
Normally
More hidden units is better
But more is more computational expensive
We'll discuss architecture more later

2) - Training a neural network

2.1) Randomly initialize the weights
Small values near 0
2.2) Implement forward propagation to get hƟ(x)i for any xi
2.3) Implement code to compute the cost function J(Ɵ)
2.4) Implement back propagation to compute the partial derivatives

General implementation below

for i = 1:m {
Forward propagation on (xi, yi) --> get activation (a) terms
Back propagation on (xi, yi) --> get delta (δ) terms
Compute Δ := Δl + δl+1(al)T
}

With this done compute the partial derivative terms

Notes on implementation
Usually done with a for loop over training examples (for forward and back propagation)
Can be done without a for loop, but this is a much more complicated way of doing things
Be careful

www.holehouse.org/mlclass/09_Neural_Networks_Learning.html 10/11
 2.5) Use gradient checking to compare the partial derivatives computed using the above algorithm and numerical estimation of
6/15/2019 gradient of J(Ɵ) 09_Neural_Networks_Learning
Disable the gradient checking code for when you actually run it
2.6) Use gradient descent or an advanced optimization method with back propagation to try to minimize J(Ɵ) as a function of
parameters Ɵ
Here J(Ɵ) is non-convex
Can be susceptible to local minimum
In practice this is not usually a huge problem
Can't guarantee programs with find global optimum should find good local optimum at least

e.g. above pretending data only has two features to easily display what's going on
Our minimum here represents a hypothesis output which is pretty close to y
If you took one of the peaks hypothesis is far from y
Gradient descent will start from some random point and move downhill
Back propagation calculates gradient down that hill

www.holehouse.org/mlclass/09_Neural_Networks_Learning.html 11/11