8/27/2019 How to Code a Neural Network with Backpropagation In Python

 Navigation

Want help with machine learning? Take the FREE Crash-Course.

Search... 

How to Code a Neural Network with Backpropagation

In Python
by Jason Brownlee on November 7, 2016 in Code Machine Learning Algorithms From Scratch

Tweet Share Share

The backpropagation algorithm is used in the classical feed-forward artificial neural network.

It is the technique still used to train large deep learning networks.

In this tutorial, you will discover how to implement the backpropagation algorithm for a neural network from
scratch with Python.

After completing this tutorial, you will know:

How to forward-propagate an input to calculate an output.

How to back-propagate error and train a network.
How to apply the backpropagation algorithm to a real-world predictive modeling problem.

Discover how to code ML algorithms from scratch including kNN, decision trees, neural nets, ensembles
and much more in my new book, with full Python code and no fancy libraries.

Let’s get started.

Update Nov/2016: Fixed a bug in the activate() function. Thanks Alex!

Update Jan/2017: Fixes issues with Python 3.
Update Jan/2017: Updated small bug in update_weights(). Thanks Tomasz!
Update Apr/2018: Added direct link to CSV dataset.
Update Aug/2018: Tested and updated to work with Python 3.6.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 1/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Your Start in Machine ×

Learning
You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.

Email Address

START MY EMAIL COURSE

How to Implement the Backpropagation Algorithm From Scratch In Python

Photo by NICHD, some rights reserved.

Description
This section provides a brief introduction to the Backpropagation Algorithm and the Wheat Seeds dataset
that we will be using in this tutorial.

Backpropagation Algorithm
The Backpropagation algorithm is a supervised learning method for multilayer feed-forward networks from
the field of Artificial Neural Networks.
Your Start in Machine Learning
Feed-forward neural networks are inspired by the information processing of one or more neural cells, called
a neuron. A neuron accepts input signals via its dendrites, which pass the electrical signal down to the cell
body. The axon carries the signal out to synapses, which are the connections of a cell’s axon to other cell’s
dendrites.

The principle of the backpropagation approach is to model a given function by modifying internal
weightings of input signals to produce an expected output signal. The system is trained using a supervised

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 2/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

learning method, where the error between the system’s output and a known expected output is presented
to the system and used to modify its internal state.

Technically, the backpropagation algorithm is a method for training the weights in a multilayer feed-forward
neural network. As such, it requires a network structure to be defined of one or more layers where one
layer is fully connected to the next layer. A standard network structure is one input layer, one hidden layer,
and one output layer.

Backpropagation can be used for both classification and regression problems, but we will focus on
classification in this tutorial.

In classification problems, best results are achieved when the network has one neuron in the output layer
for each class value. For example, a 2-class or binary classification problem with the class values of A and
B. These expected outputs would have to be transformedYour Start
into binary in Machine
vectors with one column for each ×
class value. Such as [1, 0] and [0, 1] for A and B respectively. This is called a one hot encoding.
Learning
Wheat Seeds Dataset You can master applied Machine Learning
without math or fancy degrees.
The seeds dataset involves the prediction of species given measurements
Find seeds
out how in this free andfrom different
practical course.varieties of
wheat.

Email Address
There are 201 records and 7 numerical input variables. It is a classification problem with 3 output classes.
The scale for each numeric input value vary, so some data normalization may be required for use with
algorithms that weight inputs like the backpropagation algorithm.
START MY EMAIL COURSE

Below is a sample of the first 5 rows of the dataset.

1 15.26,14.84,0.871,5.763,3.312,2.221,5.22,1
2 14.88,14.57,0.8811,5.554,3.333,1.018,4.956,1
3 14.29,14.09,0.905,5.291,3.337,2.699,4.825,1
4 13.84,13.94,0.8955,5.324,3.379,2.259,4.805,1
5 16.14,14.99,0.9034,5.658,3.562,1.355,5.175,1

Using the Zero Rule algorithm that predicts the most common class value, the baseline accuracy for the
problem is 28.095%.

You can learn more and download the seeds dataset from the UCI Machine Learning Repository.

Download the seeds dataset and place it into your current working directory with the filename
Your Start in Machine Learning
seeds_dataset.csv.

The dataset is in tab-separated format, so you must convert it to CSV using a text editor or a spreadsheet
program.

Update, download the dataset in CSV format directly:

Download Wheat Seeds Dataset

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 3/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Tutorial
This tutorial is broken down into 6 parts:

1. Initialize Network.
2. Forward Propagate.
3. Back Propagate Error.
4. Train Network.
5. Predict.
6. Seeds Dataset Case Study.

These steps will provide the foundation that you need to implement the backpropagation algorithm from
scratch and apply it to your own predictive modeling problems.
Your Start in Machine ×
1. Initialize Network Learning
Let’s start with something easy, the creation of a new network ready
You can forapplied
master training.
Machine Learning
without math or fancy degrees.
Each neuron has a set of weights that need to be maintained. One
Find out howweight for each
in this free input connection
and practical course. and an
additional weight for the bias. We will need to store additional properties for a neuron during training,
therefore we will use a dictionary to represent each neuron and store properties by names such as
Email Address
‘weights‘ for the weights.

A network is organized into layers. The input layer is really START

just a MY
rowEMAIL
from COURSE
our training dataset. The first
real layer is the hidden layer. This is followed by the output layer that has one neuron for each class value.

We will organize layers as arrays of dictionaries and treat the whole network as an array of layers.

It is good practice to initialize the network weights to small random numbers. In this case, will we use
random numbers in the range of 0 to 1.

Below is a function named initialize_network() that creates a new neural network ready for training. It
accepts three parameters, the number of inputs, the number of neurons to have in the hidden layer and the
number of outputs.

You can see that for the hidden layer we create n_hidden neurons and each neuron in the hidden layer
has n_inputs + 1 weights, one for each input column in a dataset and an additional one for the bias.
Your Start in Machine Learning
You can also see that the output layer that connects to the hidden layer has n_outputs neurons, each with
n_hidden + 1 weights. This means that each neuron in the output layer connects to (has a weight for) each
neuron in the hidden layer.

1 # Initialize a network
2 def initialize_network(n_inputs, n_hidden, n_outputs):
3 network = list()
4 hidden_layer = [{'weights':[random() for i in range(n_inputs + 1)]} for i in range(n_hidden)
5 network.append(hidden_layer)
6 output_layer = [{'weights':[random() for i in range(n_hidden + 1)]} for i in range(n_outputs

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 4/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python
7 network.append(output_layer)
8 return network

Let’s test out this function. Below is a complete example that creates a small network.

1 from random import seed

2 from random import random
3  
4 # Initialize a network
5 def initialize_network(n_inputs, n_hidden, n_outputs):
6 network = list()
7 hidden_layer = [{'weights':[random() for i in range(n_inputs + 1)]} for i in range(n_hidden
8 network.append(hidden_layer)
9 output_layer = [{'weights':[random() for i in range(n_hidden + 1)]} for i in range(n_output
10 network.append(output_layer)
11 return network
12  
13
14
seed(1)
network = initialize_network(2, 1, 2) Your Start in Machine ×
15 for layer in network:
16 print(layer) Learning
Running the example, you can see that the code prints outYoueach layer one
can master by one.
applied YouLearning
Machine can see the hidden
layer has one neuron with 2 input weights plus the bias. The output
without layer
math has 2degrees.
or fancy neurons, each with 1
weight plus the bias. Find out how in this free and practical course.

1 [{'weights': [0.13436424411240122, 0.8474337369372327, 0.763774618976614]}]

Email Address
2 [{'weights': [0.2550690257394217, 0.49543508709194095]}, {'weights': [0.4494910647887381, 0.6515

Now that we know how to create and initialize a network, let’s see how we can use it to calculate an output.
START MY EMAIL COURSE

2. Forward Propagate
We can calculate an output from a neural network by propagating an input signal through each layer until
the output layer outputs its values.

We call this forward-propagation.

It is the technique we will need to generate predictions during training that will need to be corrected, and it
is the method we will need after the network is trained to make predictions on new data.

We can break forward propagation down into three parts:

1. Neuron Activation.
2. Neuron Transfer. Your Start in Machine Learning
3. Forward Propagation.

2.1. Neuron Activation

The first step is to calculate the activation of one neuron given an input.

The input could be a row from our training dataset, as in the case of the hidden layer. It may also be the
outputs from each neuron in the hidden layer, in the case of the output layer.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 5/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Neuron activation is calculated as the weighted sum of the inputs. Much like linear regression.

1 activation = sum(weight_i * input_i) + bias

Where weight is a network weight, input is an input, i is the index of a weight or an input and bias is a
special weight that has no input to multiply with (or you can think of the input as always being 1.0).

Below is an implementation of this in a function named activate(). You can see that the function assumes
that the bias is the last weight in the list of weights. This helps here and later to make the code easier to
read.

1 # Calculate neuron activation for an input

2 def activate(weights, inputs):
3 activation = weights[-1]
4
5
for i in range(len(weights)-1):
activation += weights[i] * inputs[i] Your Start in Machine ×
6 return activation
Learning
Now, let’s see how to use the neuron activation.
You can master applied Machine Learning
2.2. Neuron Transfer without math or fancy degrees.
Find out how in this free and practical course.
Once a neuron is activated, we need to transfer the activation to see what the neuron output actually is.

Different transfer functions can be used. It is traditional to Email Address

use the sigmoid activation function, but you can
also use the tanh (hyperbolic tangent) function to transfer outputs. More recently, the rectifier transfer
function has been popular with large deep learning networks.
START MY EMAIL COURSE

The sigmoid activation function looks like an S shape, it’s also called the logistic function. It can take any
input value and produce a number between 0 and 1 on an S-curve. It is also a function of which we can
easily calculate the derivative (slope) that we will need later when backpropagating error.

We can transfer an activation function using the sigmoid function as follows:

1 output = 1 / (1 + e^(-activation))

Where e is the base of the natural logarithms (Euler’s number).

Below is a function named transfer() that implements the sigmoid equation.

1 # Transfer neuron activation

2 def transfer(activation):
Your Start in Machine Learning
3 return 1.0 / (1.0 + exp(-activation))

Now that we have the pieces, let’s see how they are used.

2.3. Forward Propagation

Forward propagating an input is straightforward.

We work through each layer of our network calculating the outputs for each neuron. All of the outputs from
one layer become inputs to the neurons on the next layer.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 6/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Below is a function named forward_propagate() that implements the forward propagation for a row of data
from our dataset with our neural network.

You can see that a neuron’s output value is stored in the neuron with the name ‘output‘. You can also see
that we collect the outputs for a layer in an array named new_inputs that becomes the array inputs and is
used as inputs for the following layer.

The function returns the outputs from the last layer also called the output layer.

1 # Forward propagate input to a network output

2 def forward_propagate(network, row):
3 inputs = row
4 for layer in network:
5 new_inputs = []
6
7
for neuron in layer:
activation = activate(neuron['weights'], inputs) Your Start in Machine ×
8 neuron['output'] = transfer(activation)
9 new_inputs.append(neuron['output']) Learning
10 inputs = new_inputs
11 return inputs
You can master applied Machine Learning
without
Let’s put all of these pieces together and test out the forward math or fancy
propagation degrees.
of our network.
Find out how in this free and practical course.
We define our network inline with one hidden neuron that expects 2 input values and an output layer with
two neurons. Email Address

1 from math import exp

2   START MY EMAIL COURSE
3 # Calculate neuron activation for an input
4 def activate(weights, inputs):
5 activation = weights[-1]
6 for i in range(len(weights)-1):
7 activation += weights[i] * inputs[i]
8 return activation
9  
10 # Transfer neuron activation
11 def transfer(activation):
12 return 1.0 / (1.0 + exp(-activation))
13  
14 # Forward propagate input to a network output
15 def forward_propagate(network, row):
16 inputs = row
17 for layer in network:
18 new_inputs = []
19 for neuron in layer:
20 activation = activate(neuron['weights'], inputs)
21 neuron['output'] = transfer(activation)
Your Start in Machine Learning
22 new_inputs.append(neuron['output'])
23 inputs = new_inputs
24 return inputs
25  
26 # test forward propagation
27 network = [[{'weights': [0.13436424411240122, 0.8474337369372327, 0.763774618976614]}],
28 [{'weights': [0.2550690257394217, 0.49543508709194095]}, {'weights': [0.449491064788738
29 row = [1, 0, None]
30 output = forward_propagate(network, row)
31 print(output)

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 7/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Running the example propagates the input pattern [1, 0] and produces an output value that is printed.
Because the output layer has two neurons, we get a list of two numbers as output.

The actual output values are just nonsense for now, but next, we will start to learn how to make the weights
in the neurons more useful.

1 [0.6629970129852887, 0.7253160725279748]

3. Back Propagate Error

The backpropagation algorithm is named for the way in which weights are trained.

Error is calculated between the expected outputs and the outputs forward propagated from the network.
These errors are then propagated backward through the network from the output layer to the hidden layer,
Your
assigning blame for the error and updating weights as they go. Start in Machine
×
Learning
The math for backpropagating error is rooted in calculus, but we will remain high level in this section and
focus on what is calculated and how rather than why the You can mastertake
calculations applied
thisMachine Learning
particular form.
without math or fancy degrees.
This part is broken down into two sections. Find out how in this free and practical course.

1. Transfer Derivative. Email Address

2. Error Backpropagation.

3.1. Transfer Derivative START MY EMAIL COURSE

Given an output value from a neuron, we need to calculate it’s slope.

We are using the sigmoid transfer function, the derivative of which can be calculated as follows:

1 derivative = output * (1.0 - output)

Below is a function named transfer_derivative() that implements this equation.

1 # Calculate the derivative of an neuron output

2 def transfer_derivative(output):
3 return output * (1.0 - output)

Now, let’s see how this can be used.

3.2. Error Backpropagation Your Start in Machine Learning

The first step is to calculate the error for each output neuron, this will give us our error signal (input) to
propagate backwards through the network.

The error for a given neuron can be calculated as follows:

1 error = (expected - output) * transfer_derivative(output)

Where expected is the expected output value for the neuron, output is the output value for the neuron and
transfer_derivative() calculates the slope of the neuron’s output value, as shown above.
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 8/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

This error calculation is used for neurons in the output layer. The expected value is the class value itself. In
the hidden layer, things are a little more complicated.

The error signal for a neuron in the hidden layer is calculated as the weighted error of each neuron in the
output layer. Think of the error traveling back along the weights of the output layer to the neurons in the
hidden layer.

The back-propagated error signal is accumulated and then used to determine the error for the neuron in
the hidden layer, as follows:

1 error = (weight_k * error_j) * transfer_derivative(output)

Where error_j is the error signal from the jth neuron in the output layer, weight_k is the weight that
connects the kth neuron to the current neuron and output is the output for the current neuron.
Your Start in Machine ×
Below is a function named backward_propagate_error()Learning
that implements this procedure.

You can see that the error signal calculated for each neuron
You is stored
can masterwith the Machine
applied name ‘delta’.
LearningYou can see
that the layers of the network are iterated in reverse order, starting
without at the
math output
or fancy and working backwards.
degrees.
Find out
This ensures that the neurons in the output layer have ‘delta’ how calculated
values in this free and
firstpractical course. in the
that neurons
hidden layer can use in the subsequent iteration. I chose the name ‘delta’ to reflect the change the error
implies on the neuron (e.g. the weight delta). Email Address

You can see that the error signal for neurons in the hidden layer is accumulated from neurons in the output
layer where the hidden neuron number j is also the index ofSTART MY EMAIL COURSE
the neuron’s weight in the output layer
neuron[‘weights’][j].

1 # Backpropagate error and store in neurons

2 def backward_propagate_error(network, expected):
3 for i in reversed(range(len(network))):
4 layer = network[i]
5 errors = list()
6 if i != len(network)-1:
7 for j in range(len(layer)):
8 error = 0.0
9 for neuron in network[i + 1]:
10 error += (neuron['weights'][j] * neuron['delta'])
11 errors.append(error)
12 else:
13 for j in range(len(layer)):
14 neuron = layer[j]
15 errors.append(expected[j] - neuron['output'])
16 for j in range(len(layer)): Your Start in Machine Learning
17 neuron = layer[j]
18 neuron['delta'] = errors[j] * transfer_derivative(neuron['output'])

Let’s put all of the pieces together and see how it works.

We define a fixed neural network with output values and backpropagate an expected output pattern. The
complete example is listed below.

1 # Calculate the derivative of an neuron output

2 def transfer_derivative(output):

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 9/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

3 return output * (1.0 - output)

4  
5 # Backpropagate error and store in neurons
6 def backward_propagate_error(network, expected):
7 for i in reversed(range(len(network))):
8 layer = network[i]
9 errors = list()
10 if i != len(network)-1:
11 for j in range(len(layer)):
12 error = 0.0
13 for neuron in network[i + 1]:
14 error += (neuron['weights'][j] * neuron['delta'])
15 errors.append(error)
16 else:
17 for j in range(len(layer)):
18 neuron = layer[j]
19 errors.append(expected[j] - neuron['output'])
20 for j in range(len(layer)):
21
22
neuron = layer[j]
Your Start in Machine
neuron['delta'] = errors[j] * transfer_derivative(neuron['output'])
×
23  
24 # test backpropagation of error Learning
25 network = [[{'output': 0.7105668883115941, 'weights': [0.13436424411240122, 0.8474337369372327,
26 [{'output': 0.6213859615555266, 'weights':
You[0.2550690257394217,
can master applied Machine 0.49543508709194095]},
Learning
27 expected = [0, 1] without math or fancy degrees.
28 backward_propagate_error(network, expected)
29 for layer in network: Find out how in this free and practical course.
30 print(layer)

Email Address
Running the example prints the network after the backpropagation of error is complete. You can see that
error values are calculated and stored in the neurons for the output layer and the hidden layer.
START MY EMAIL COURSE
1 [{'output': 0.7105668883115941, 'weights': [0.13436424411240122, 0.8474337369372327, 0.763774618
2 [{'output': 0.6213859615555266, 'weights': [0.2550690257394217, 0.49543508709194095], 'delta': -

Now let’s use the backpropagation of error to train the network.

4. Train Network
The network is trained using stochastic gradient descent.

This involves multiple iterations of exposing a training dataset to the network and for each row of data
forward propagating the inputs, backpropagating the error and updating the network weights.

This part is broken down into two sections:

1. Update Weights. Your Start in Machine Learning

2. Train Network.

4.1. Update Weights

Once errors are calculated for each neuron in the network via the back propagation method above, they
can be used to update weights.

Network weights are updated as follows:

1 weight = weight + learning_rate * error * input

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 10/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Where weight is a given weight, learning_rate is a parameter that you must specify, error is the error
calculated by the backpropagation procedure for the neuron and input is the input value that caused the
error.

The same procedure can be used for updating the bias weight, except there is no input term, or input is the
fixed value of 1.0.

Learning rate controls how much to change the weight to correct for the error. For example, a value of 0.1
will update the weight 10% of the amount that it possibly could be updated. Small learning rates are
preferred that cause slower learning over a large number of training iterations. This increases the likelihood
of the network finding a good set of weights across all layers rather than the fastest set of weights that
minimize error (called premature convergence).

Below is a function named update_weights() that updatesYour Startforina Machine

the weights network given an input row of ×
data, a learning rate and assume that a forward and backward propagation have already been performed.
Learning
Remember that the input for the output layer is a collection
Youofcan
outputs
masterfrom theMachine
applied hidden Learning
layer.
without math or fancy degrees.
1 # Update network weights with error
Find out how in this free and practical course.
2 def update_weights(network, row, l_rate):
3 for i in range(len(network)):
4 inputs = row[:-1]
5 if i != 0: Email Address
6 inputs = [neuron['output'] for neuron in network[i - 1]]
7 for neuron in network[i]:
8 for j in range(len(inputs)): START MY EMAIL COURSE
9 neuron['weights'][j] += l_rate * neuron['delta'] * inputs[j]
10 neuron['weights'][-1] += l_rate * neuron['delta']

Now we know how to update network weights, let’s see how we can do it repeatedly.

4.2. Train Network

As mentioned, the network is updated using stochastic gradient descent.

This involves first looping for a fixed number of epochs and within each epoch updating the network for
each row in the training dataset.

Because updates are made for each training pattern, this type of learning is called online learning. If errors
were accumulated across an epoch before updating the weights, this is called batch learning or batch
gradient descent.
Your Start in Machine Learning

Below is a function that implements the training of an already initialized neural network with a given training
dataset, learning rate, fixed number of epochs and an expected number of output values.

The expected number of output values is used to transform class values in the training data into a one hot
encoding. That is a binary vector with one column for each class value to match the output of the network.
This is required to calculate the error for the output layer.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 11/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

You can also see that the sum squared error between the expected output and the network output is
accumulated each epoch and printed. This is helpful to create a trace of how much the network is learning
and improving each epoch.

1 # Train a network for a fixed number of epochs

2 def train_network(network, train, l_rate, n_epoch, n_outputs):
3 for epoch in range(n_epoch):
4 sum_error = 0
5 for row in train:
6 outputs = forward_propagate(network, row)
7 expected = [0 for i in range(n_outputs)]
8 expected[row[-1]] = 1
9 sum_error += sum([(expected[i]-outputs[i])**2 for i in range(len(expected))])
10 backward_propagate_error(network, expected)
11 update_weights(network, row, l_rate)
12 print('>epoch=%d, lrate=%.3f, error=%.3f' % (epoch, l_rate, sum_error))

We now have all of the pieces to train the network. We can Your Start an
put together inexample
Machine that includes
×
Learning
everything we’ve seen so far including network initialization and train a network on a small dataset.

Below is a small contrived dataset that we can use to testYou

outcan
training
masterour neural
applied network.
Machine Learning
without math or fancy degrees.
1 X1 X2 Y Find out how in this free and practical course.
2 2.7810836 2.550537003 0
3 1.465489372 2.362125076 0
4 3.396561688 4.400293529 0
Email Address
5 1.38807019 1.850220317 0
6 3.06407232 3.005305973 0
7 7.627531214 2.759262235 1
8 5.332441248 2.088626775 1 START MY EMAIL COURSE
9 6.922596716 1.77106367 1
10 8.675418651 -0.242068655 1
11 7.673756466 3.508563011 1

Below is the complete example. We will use 2 neurons in the hidden layer. It is a binary classification
problem (2 classes) so there will be two neurons in the output layer. The network will be trained for 20
epochs with a learning rate of 0.5, which is high because we are training for so few iterations.

1 from math import exp

2 from random import seed
3 from random import random
4  
5 # Initialize a network
6 def initialize_network(n_inputs, n_hidden, n_outputs):
7 network = list()
8 hidden_layer = [{'weights':[random() for i in range(n_inputs + 1)]} for i in range(n_hidde
9 network.append(hidden_layer)
10
Your Start in Machine Learning
output_layer = [{'weights':[random() for i in range(n_hidden + 1)]} for i in range(n_outpu
11 network.append(output_layer)
12 return network
13  
14 # Calculate neuron activation for an input
15 def activate(weights, inputs):
16 activation = weights[-1]
17 for i in range(len(weights)-1):
18 activation += weights[i] * inputs[i]
19 return activation
20  
21 # Transfer neuron activation
22 def transfer(activation):
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 12/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python
23 return 1.0 / (1.0 + exp(-activation))
24  
25 # Forward propagate input to a network output
26 def forward_propagate(network, row):
27 inputs = row
28 for layer in network:
29 new_inputs = []
30 for neuron in layer:
31 activation = activate(neuron['weights'], inputs)
32 neuron['output'] = transfer(activation)
33 new_inputs.append(neuron['output'])
34 inputs = new_inputs
35 return inputs
36  
37 # Calculate the derivative of an neuron output
38 def transfer_derivative(output):
39 return output * (1.0 - output)
40  
41
42
# Backpropagate error and store in neurons
def backward_propagate_error(network, expected): Your Start in Machine ×
43 for i in reversed(range(len(network))):
44 layer = network[i] Learning
45 errors = list()
46 if i != len(network)-1: You can master applied Machine Learning
47 for j in range(len(layer)): without math or fancy degrees.
48 error = 0.0
49 for neuron in network[i + 1]: Find out how in this free and practical course.
50 error += (neuron['weights'][j] * neuron['delta'])
51 errors.append(error)
52 else: Email Address
53 for j in range(len(layer)):
54 neuron = layer[j]
55 errors.append(expected[j] - neuron['output'])
START MY EMAIL COURSE
56 for j in range(len(layer)):
57 neuron = layer[j]
58 neuron['delta'] = errors[j] * transfer_derivative(neuron['output'])
59  
60 # Update network weights with error
61 def update_weights(network, row, l_rate):
62 for i in range(len(network)):
63 inputs = row[:-1]
64 if i != 0:
65 inputs = [neuron['output'] for neuron in network[i - 1]]
66 for neuron in network[i]:
67 for j in range(len(inputs)):
68 neuron['weights'][j] += l_rate * neuron['delta'] * inputs[j]
69 neuron['weights'][-1] += l_rate * neuron['delta']
70  
71 # Train a network for a fixed number of epochs
72 def train_network(network, train, l_rate, n_epoch, n_outputs):
73 for epoch in range(n_epoch):
74 sum_error = 0
Your Start in Machine Learning
75 for row in train:
76 outputs = forward_propagate(network, row)
77 expected = [0 for i in range(n_outputs)]
78 expected[row[-1]] = 1
79 sum_error += sum([(expected[i]-outputs[i])**2 for i in range(len(expected))])
80 backward_propagate_error(network, expected)
81 update_weights(network, row, l_rate)
82 print('>epoch=%d, lrate=%.3f, error=%.3f' % (epoch, l_rate, sum_error))
83  
84 # Test training backprop algorithm
85 seed(1)
86 dataset = [[2.7810836,2.550537003,0],
87 [1.465489372,2.362125076,0],
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 13/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python
88 [3.396561688,4.400293529,0],
89 [1.38807019,1.850220317,0],
90 [3.06407232,3.005305973,0],
91 [7.627531214,2.759262235,1],
92 [5.332441248,2.088626775,1],
93 [6.922596716,1.77106367,1],
94 [8.675418651,-0.242068655,1],
95 [7.673756466,3.508563011,1]]
96 n_inputs = len(dataset[0]) - 1
97 n_outputs = len(set([row[-1] for row in dataset]))
98 network = initialize_network(n_inputs, 2, n_outputs)
99 train_network(network, dataset, 0.5, 20, n_outputs)
100 for layer in network:
101 print(layer)

Running the example first prints the sum squared error each training epoch. We can see a trend of this
error decreasing with each epoch.

Your Start in Machine ×

Once trained, the network is printed, showing the learned weights. Also still in the network are output and
Learning
delta values that can be ignored. We could update our training function to delete these data if we wanted.

1 >epoch=0, lrate=0.500, error=6.350 You can master applied Machine Learning

2 >epoch=1, lrate=0.500, error=5.531 without math or fancy degrees.
3 >epoch=2, lrate=0.500, error=5.221
Find out how in this free and practical course.
4 >epoch=3, lrate=0.500, error=4.951
5 >epoch=4, lrate=0.500, error=4.519
6 >epoch=5, lrate=0.500, error=4.173
7 >epoch=6, lrate=0.500, error=3.835 Email Address
8 >epoch=7, lrate=0.500, error=3.506
9 >epoch=8, lrate=0.500, error=3.192
10 >epoch=9, lrate=0.500, error=2.898 START MY EMAIL COURSE
11 >epoch=10, lrate=0.500, error=2.626
12 >epoch=11, lrate=0.500, error=2.377
13 >epoch=12, lrate=0.500, error=2.153
14 >epoch=13, lrate=0.500, error=1.953
15 >epoch=14, lrate=0.500, error=1.774
16 >epoch=15, lrate=0.500, error=1.614
17 >epoch=16, lrate=0.500, error=1.472
18 >epoch=17, lrate=0.500, error=1.346
19 >epoch=18, lrate=0.500, error=1.233
20 >epoch=19, lrate=0.500, error=1.132
21 [{'weights': [-1.4688375095432327, 1.850887325439514, 1.0858178629550297], 'output': 0.02998030
22 [{'weights': [2.515394649397849, -0.3391927502445985, -0.9671565426390275], 'output': 0.2364879

Once a network is trained, we need to use it to make predictions.

5. Predict
Making predictions with a trained neural network is easy Your Start in Machine Learning
enough.

We have already seen how to forward-propagate an input pattern to get an output. This is all we need to do
to make a prediction. We can use the output values themselves directly as the probability of a pattern
belonging to each output class.

It may be more useful to turn this output back into a crisp class prediction. We can do this by selecting the
class value with the larger probability. This is also called the arg max function.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 14/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Below is a function named predict() that implements this procedure. It returns the index in the network
output that has the largest probability. It assumes that class values have been converted to integers
starting at 0.

1 # Make a prediction with a network

2 def predict(network, row):
3 outputs = forward_propagate(network, row)
4 return outputs.index(max(outputs))

We can put this together with our code above for forward propagating input and with our small contrived
dataset to test making predictions with an already-trained network. The example hardcodes a network
trained from the previous step.

The complete example is listed below.

1 from math import exp Your Start in Machine ×

2  
3 # Calculate neuron activation for an input Learning
4 def activate(weights, inputs):
5 activation = weights[-1] You can master applied Machine Learning
6 for i in range(len(weights)-1):
without math or fancy degrees.
7 activation += weights[i] * inputs[i]
8 return activation Find out how in this free and practical course.
9  
10 # Transfer neuron activation
11 def transfer(activation): Email Address
12 return 1.0 / (1.0 + exp(-activation))
13  
14 # Forward propagate input to a network output
15 def forward_propagate(network, row): START MY EMAIL COURSE
16 inputs = row
17 for layer in network:
18 new_inputs = []
19 for neuron in layer:
20 activation = activate(neuron['weights'], inputs)
21 neuron['output'] = transfer(activation)
22 new_inputs.append(neuron['output'])
23 inputs = new_inputs
24 return inputs
25  
26 # Make a prediction with a network
27 def predict(network, row):
28 outputs = forward_propagate(network, row)
29 return outputs.index(max(outputs))
30  
31 # Test making predictions with the network
32 dataset = [[2.7810836,2.550537003,0],
33 [1.465489372,2.362125076,0],
34 [3.396561688,4.400293529,0], Your Start in Machine Learning
35 [1.38807019,1.850220317,0],
36 [3.06407232,3.005305973,0],
37 [7.627531214,2.759262235,1],
38 [5.332441248,2.088626775,1],
39 [6.922596716,1.77106367,1],
40 [8.675418651,-0.242068655,1],
41 [7.673756466,3.508563011,1]]
42 network = [[{'weights': [-1.482313569067226, 1.8308790073202204, 1.078381922048799]}, {'weights
43 [{'weights': [2.5001872433501404, 0.7887233511355132, -1.1026649757805829]}, {'weights': [-
44 for row in dataset:
45 prediction = predict(network, row)
46 print('Expected=%d, Got=%d' % (row[-1], prediction))

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 15/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Running the example prints the expected output for each record in the training dataset, followed by the
crisp prediction made by the network.

It shows that the network achieves 100% accuracy on this small dataset.

1 Expected=0, Got=0
2 Expected=0, Got=0
3 Expected=0, Got=0
4 Expected=0, Got=0
5 Expected=0, Got=0
6 Expected=1, Got=1
7 Expected=1, Got=1
8 Expected=1, Got=1
9 Expected=1, Got=1
10 Expected=1, Got=1

Now we are ready to apply our backpropagation algorithm to a real world dataset.
Your Start in Machine ×
6. Wheat Seeds Dataset Learning
This section applies the Backpropagation algorithm to theYou can master
wheat seedsapplied Machine Learning
dataset.
without math or fancy degrees.
Find
The first step is to load the dataset and convert the loaded out to
data how in this free
numbers and
that wepractical
can use course.
in our neural
network. For this we will use the helper function load_csv() to load the file, str_column_to_float() to
convert string numbers to floats and str_column_to_int()Email Address
to convert the class column to integer values.

Input values vary in scale and need to be normalized to the range of 0 and 1. It is generally good practice
START MY EMAIL COURSE
to normalize input values to the range of the chosen transfer function, in this case, the sigmoid function that
outputs values between 0 and 1. The dataset_minmax() and normalize_dataset() helper functions were
used to normalize the input values.

We will evaluate the algorithm using k-fold cross-validation with 5 folds. This means that 201/5=40.2 or 40
records will be in each fold. We will use the helper functions evaluate_algorithm() to evaluate the
algorithm with cross-validation and accuracy_metric() to calculate the accuracy of predictions.

A new function named back_propagation() was developed to manage the application of the
Backpropagation algorithm, first initializing a network, training it on the training dataset and then using the
trained network to make predictions on a test dataset.

The complete example is listed below.

Your Start in Machine Learning
1 # Backprop on the Seeds Dataset
2 from random import seed
3 from random import randrange
4 from random import random
5 from csv import reader
6 from math import exp
7  
8 # Load a CSV file
9 def load_csv(filename):
10 dataset = list()
11 with open(filename, 'r') as file:
12 csv_reader = reader(file)
13 for row in csv_reader:
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 16/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python
14 if not row:
15 continue
16 dataset.append(row)
17 return dataset
18  
19 # Convert string column to float
20 def str_column_to_float(dataset, column):
21 for row in dataset:
22 row[column] = float(row[column].strip())
23  
24 # Convert string column to integer
25 def str_column_to_int(dataset, column):
26 class_values = [row[column] for row in dataset]
27 unique = set(class_values)
28 lookup = dict()
29 for i, value in enumerate(unique):
30 lookup[value] = i
31 for row in dataset:
32
33
row[column] = lookup[row[column]]
return lookup Your Start in Machine ×
34  
35 # Find the min and max values for each column Learning
36 def dataset_minmax(dataset):
37 minmax = list() You can master applied Machine Learning
38 stats = [[min(column), max(column)] for column in math
without zip(*dataset)]
or fancy degrees.
39 return stats
40   Find out how in this free and practical course.
41 # Rescale dataset columns to the range 0-1
42 def normalize_dataset(dataset, minmax):
43 for row in dataset: Email Address
44 for i in range(len(row)-1):
45 row[i] = (row[i] - minmax[i][0]) / (minmax[i][1] - minmax[i][0])
46  
START MY EMAIL COURSE
47 # Split a dataset into k folds
48 def cross_validation_split(dataset, n_folds):
49 dataset_split = list()
50 dataset_copy = list(dataset)
51 fold_size = int(len(dataset) / n_folds)
52 for i in range(n_folds):
53 fold = list()
54 while len(fold) < fold_size:
55 index = randrange(len(dataset_copy))
56 fold.append(dataset_copy.pop(index))
57 dataset_split.append(fold)
58 return dataset_split
59  
60 # Calculate accuracy percentage
61 def accuracy_metric(actual, predicted):
62 correct = 0
63 for i in range(len(actual)):
64 if actual[i] == predicted[i]:
65 correct += 1
Your Start in Machine Learning
66 return correct / float(len(actual)) * 100.0
67  
68 # Evaluate an algorithm using a cross validation split
69 def evaluate_algorithm(dataset, algorithm, n_folds, *args):
70 folds = cross_validation_split(dataset, n_folds)
71 scores = list()
72 for fold in folds:
73 train_set = list(folds)
74 train_set.remove(fold)
75 train_set = sum(train_set, [])
76 test_set = list()
77 for row in fold:
78 row_copy = list(row)
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 17/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python
79 test_set.append(row_copy)
80 row_copy[-1] = None
81 predicted = algorithm(train_set, test_set, *args)
82 actual = [row[-1] for row in fold]
83 accuracy = accuracy_metric(actual, predicted)
84 scores.append(accuracy)
85 return scores
86  
87 # Calculate neuron activation for an input
88 def activate(weights, inputs):
89 activation = weights[-1]
90 for i in range(len(weights)-1):
91 activation += weights[i] * inputs[i]
92 return activation
93  
94 # Transfer neuron activation
95 def transfer(activation):
96 return 1.0 / (1.0 + exp(-activation))
97
98
 
# Forward propagate input to a network output Your Start in Machine ×
99 def forward_propagate(network, row):
100 inputs = row Learning
101 for layer in network:
102 new_inputs = [] You can master applied Machine Learning
103 for neuron in layer: without math or fancy degrees.
104 activation = activate(neuron['weights'], inputs)
105 Find out how in this free and practical course.
neuron['output'] = transfer(activation)
106 new_inputs.append(neuron['output'])
107 inputs = new_inputs
108 return inputs Email Address
109  
110 # Calculate the derivative of an neuron output
111 def transfer_derivative(output):
START MY EMAIL COURSE
112 return output * (1.0 - output)
113  
114 # Backpropagate error and store in neurons
115 def backward_propagate_error(network, expected):
116 for i in reversed(range(len(network))):
117 layer = network[i]
118 errors = list()
119 if i != len(network)-1:
120 for j in range(len(layer)):
121 error = 0.0
122 for neuron in network[i + 1]:
123 error += (neuron['weights'][j] * neuron['delta'])
124 errors.append(error)
125 else:
126 for j in range(len(layer)):
127 neuron = layer[j]
128 errors.append(expected[j] - neuron['output'])
129 for j in range(len(layer)):
130 neuron = layer[j]
Your Start in Machine Learning
131 neuron['delta'] = errors[j] * transfer_derivative(neuron['output'])
132  
133 # Update network weights with error
134 def update_weights(network, row, l_rate):
135 for i in range(len(network)):
136 inputs = row[:-1]
137 if i != 0:
138 inputs = [neuron['output'] for neuron in network[i - 1]]
139 for neuron in network[i]:
140 for j in range(len(inputs)):
141 neuron['weights'][j] += l_rate * neuron['delta'] * inputs[j]
142 neuron['weights'][-1] += l_rate * neuron['delta']
143  
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 18/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python
144 # Train a network for a fixed number of epochs
145 def train_network(network, train, l_rate, n_epoch, n_outputs):
146 for epoch in range(n_epoch):
147 for row in train:
148 outputs = forward_propagate(network, row)
149 expected = [0 for i in range(n_outputs)]
150 expected[row[-1]] = 1
151 backward_propagate_error(network, expected)
152 update_weights(network, row, l_rate)
153  
154 # Initialize a network
155 def initialize_network(n_inputs, n_hidden, n_outputs):
156 network = list()
157 hidden_layer = [{'weights':[random() for i in range(n_inputs + 1)]} for i in range(n_hidde
158 network.append(hidden_layer)
159 output_layer = [{'weights':[random() for i in range(n_hidden + 1)]} for i in range(n_outpu
160 network.append(output_layer)
161 return network
162
163
 
# Make a prediction with a network Your Start in Machine ×
164 def predict(network, row):
165 outputs = forward_propagate(network, row) Learning
166 return outputs.index(max(outputs))
167   You can master applied Machine Learning
168 # Backpropagation Algorithm With Stochastic Gradient
withoutDescent
math or fancy degrees.
169 def back_propagation(train, test, l_rate, n_epoch, n_hidden):
170 n_inputs = len(train[0]) - 1 Find out how in this free and practical course.
171 n_outputs = len(set([row[-1] for row in train]))
172 network = initialize_network(n_inputs, n_hidden, n_outputs)
173 Email
train_network(network, train, l_rate, n_epoch, Address
n_outputs)
174 predictions = list()
175 for row in test:
176 prediction = predict(network, row)
START MY EMAIL COURSE
177 predictions.append(prediction)
178 return(predictions)
179  
180 # Test Backprop on Seeds dataset
181 seed(1)
182 # load and prepare data
183 filename = 'seeds_dataset.csv'
184 dataset = load_csv(filename)
185 for i in range(len(dataset[0])-1):
186 str_column_to_float(dataset, i)
187 # convert class column to integers
188 str_column_to_int(dataset, len(dataset[0])-1)
189 # normalize input variables
190 minmax = dataset_minmax(dataset)
191 normalize_dataset(dataset, minmax)
192 # evaluate algorithm
193 n_folds = 5
194 l_rate = 0.3
195 n_epoch = 500
Your Start in Machine Learning
196 n_hidden = 5
197 scores = evaluate_algorithm(dataset, back_propagation, n_folds, l_rate, n_epoch, n_hidden)
198 print('Scores: %s' % scores)
199 print('Mean Accuracy: %.3f%%' % (sum(scores)/float(len(scores))))

A network with 5 neurons in the hidden layer and 3 neurons in the output layer was constructed. The
network was trained for 500 epochs with a learning rate of 0.3. These parameters were found with a little
trial and error, but you may be able to do much better.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 19/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Running the example prints the average classification accuracy on each fold as well as the average
performance across all folds.

You can see that backpropagation and the chosen configuration achieved a mean classification accuracy of
about 93% which is dramatically better than the Zero Rule algorithm that did slightly better than 28%
accuracy.

1 Scores: [92.85714285714286, 92.85714285714286, 97.61904761904762, 92.85714285714286, 90.47619047

2 Mean Accuracy: 93.333%

Extensions
This section lists extensions to the tutorial that you may wish to explore.

Your Start in Machine

Tune Algorithm Parameters. Try larger or smaller networks trained for longer or shorter. See if you
×
can get better performance on the seeds dataset. Learning
Additional Methods. Experiment with different weight initialization techniques (such as small random
You can master applied Machine Learning
numbers) and different transfer functions (such as tanh).
without math or fancy degrees.
More Layers. Add support for more hidden layers, trained in just the same way as the one hidden
Find out how in this free and practical course.
layer used in this tutorial.
Regression. Change the network so that there is only one neuron in the output layer and that a real
Email Address
value is predicted. Pick a regression dataset to practice on. A linear transfer function could be used for
neurons in the output layer, or the output values of the chosen dataset could be scaled to values
between 0 and 1. START MY EMAIL COURSE
Batch Gradient Descent. Change the training procedure from online to batch gradient descent and
update the weights only at the end of each epoch.

Did you try any of these extensions?

Share your experiences in the comments below.

Review
In this tutorial, you discovered how to implement the Backpropagation algorithm from scratch.

Specifically, you learned:

How to forward propagate an input to calculate a network output.

Your Start in Machine Learning
How to back propagate error and update network weights.
How to apply the backpropagation algorithm to a real world dataset.

Do you have any questions?

Ask your questions in the comments below and I will do my best to answer.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 20/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Want to Code Algorithms in Python Without Math?

Code Your First Algorithm in Minutes
…with step-by-step tutorials on real-world datasets

Discover how in my new Ebook:

Machine Learning Algorithms From Scratch

It covers 18 tutorials with all the code for 12 top algorithms, like:
Linear Regression, k-Nearest Neighbors, Stochastic Gradient Descent and much
more…

Finally, Pull Back the Curtain on

Machine Learning Algorithms
Your Start in Machine ×
Skip the Academics. Just Results.
Learning
Click to learn more.
You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.

Email Address
Tweet Share Share

About Jason Brownlee START MY EMAIL COURSE

Jason Brownlee, PhD is a machine learning specialist who teaches developers how to get results with
modern machine learning methods via hands-on tutorials.
View all posts by Jason Brownlee →

 How To Implement Learning Vector Quantization (LVQ) From Scratch With Python
How To Implement The Decision Tree Algorithm From Scratch In Python 

516 Responses to How to Code a Neural Network with Backpropagation In

Your Start in Machine Learning
Python

REPLY 
Talk Data To Me November 7, 2016 at 9:28 pm #

That’s what I was looking for. Write a neural network without any libraries (scikit, keras etc.) Thnak
you very much!

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 21/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Jason Brownlee November 8, 2016 at 9:51 am #

I’m glad to hear it!

REPLY 
sari dewi August 16, 2019 at 11:55 am #

Hy Mr. jason , i try your code to make a neural network with backpropagation method, I
using jupyter notebook anaconda and pyhton 3.7 64 bit, when i try this code

seed(1)
# load and prepare data
filename =’datalatih.csv’
dataset = load_csv(filename) Your Start in Machine ×
for i in range(len(dataset[0])-1): Learning
str_column_to_float(dataset, i)
# convert class column to integers You can master applied Machine Learning
str_column_to_int(dataset, len(dataset[0])-1) without math or fancy degrees.
# normalize input variables Find out how in this free and practical course.
minmax = dataset_minmax(dataset)
normalize_dataset(dataset, minmax) Email Address
# evaluate algorithm
n_folds =5
l_rate =0.3 START MY EMAIL COURSE

n_epoch =500
n_hidden =5
scores = evaluate_algorithm(dataset, back_propagation, n_folds, l_rate, n_epoch, n_hidden)

print (‘Scores: %s’ % scores)

print (‘Mean Accuracy: %.3f%%’ % (sum(scores)/float(len(scores))))

but I get error message

IndexError Traceback (most recent call last)

in
196 n_epoch =500
197 n_hidden =5
–> 198 scores = evaluate_algorithm(dataset, back_propagation, n_folds, l_rate, n_epoch, n_hidden)
199 Your Start in Machine Learning
200 print (‘Scores: %s’ % scores)

in evaluate_algorithm(dataset, algorithm, n_folds, *args)

79 test_set.append(row_copy)
80 row_copy[-1] = None
—> 81 predicted = algorithm(train_set, test_set, *args)
82 actual = [row[-1] for row in fold]
83 accuracy = accuracy_metric(actual, predicted)

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 22/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

in back_propagation(train, test, l_rate, n_epoch, n_hidden)

171 n_outputs = len(set([row[-1] for row in train]))
172 network = initialize_network(n_inputs, n_hidden, n_outputs)
–> 173 train_network(network, train, l_rate, n_epoch, n_outputs)
174 predictions = list()
175 for row in test:

in train_network(network, train, l_rate, n_epoch, n_outputs)

148 outputs = forward_propagate(network, row)
149 expected = [0 for i in range(n_outputs)]
–> 150 expected[row[-1]] = 1
151 backward_propagate_error(network, expected)
152 update_weights(network, row, l_rate)

IndexError: list assignment index out of range Your Start in Machine ×

Learning
what my mistake? is there missing code? thankyou

You can master applied Machine Learning

without math or fancy degrees.
Find REPLY
pm out
# how in this free and practical course.

Jason Brownlee August 16, 2019 at 2:11

Sorry to hear that you are having trouble,

EmailI have some suggestions for you here:
Address
https://machinelearningmastery.com/faq/single-faq/why-does-the-code-in-the-tutorial-not-work-
for-me
START MY EMAIL COURSE

REPLY 
WB February 20, 2018 at 3:07 pm #

I experienced the following applying the Backpropagation algorithm to the wheat seeds dataset.
I am wondering how to resolve the errors? Thank you
—————————————————————————
ValueError Traceback (most recent call last)
in ()
184 dataset = load_csv(filename)
185 for i in range(len(dataset[0])-1):
–> 186 str_column_to_float(dataset, i)
187 # convert class column to integers
188 str_column_to_int(dataset, len(dataset[0])-1) Your Start in Machine Learning

in str_column_to_float(dataset, column)
20 def str_column_to_float(dataset, column):
21 for row in dataset:
—> 22 row[column] = float(row[column].strip())
23
24 # Convert string column to integer

ValueError: could not convert string to float:

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 23/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Jason Brownlee February 21, 2018 at 6:35 am #

Are you using Python 2?

REPLY 
wb February 21, 2018 at 2:51 pm #

Yes I am

harshith October 5, 2018 at 8:28 pm # Your Start in Machine ×

REPLY 

Learning
hi bro whass up
You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.
REPLY 
Mike Harney March 5, 2018 at 9:53 am #
Email Address
Hi wb, I’m on 3.6 and I found the same issue. Maybe you can answer this Jason, but it
looks like the some of the data is misaligned in the sample. When opened in Excel, there are many
open spaces followed by data jutted out to an extra START
column.
MYI assume this is unintentional, and when
EMAIL COURSE
I corrected the spacing, it appeared to work for me.

REPLY 
Jason Brownlee March 6, 2018 at 6:08 am #

The code was written and tested with Python 2.7.

JU April 23, 2018 at 7:24 am #

Mike is right – the dataset from the UCI website is slightly defective: It has two tabs
in some places where there should be only one.Start
Your Thisin
needs to beLearning
Machine corrected during the
conversion to CSV. In Excel the easiest way is to use the text importer and then click the
“Treat consecutive delimiters as one” checkbox.

Jason Brownlee April 23, 2018 at 7:37 am #

Here is the dataset ready to use:

https://raw.githubusercontent.com/jbrownlee/Datasets/master/wheat-seeds.csv

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 24/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Alexis Batyk August 29, 2018 at 6:22 am #

[SOLVED]
i have the same issue with

https://raw.githubusercontent.com/jbrownlee/Datasets/master/wheat-seeds.csv

there is still dirty that csv

use a text editor -> select search and replace tool -> search ‘,,’ replace ‘,’ and it works

Your Start in Machine ×

REPLY 
Jason Brownlee August 29, 2018 at 8:16 am #
Learning
I don’t have such problems on Py 3.6.
You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.
REPLY 
Jackson Scott October 1, 2018 at 9:08 am #
Email Address
thanks, this worked for me as well. The csv file had some tabbed over and others
correct.
START MY EMAIL COURSE

REPLY 
Deng October 14, 2018 at 5:50 pm #

The data in the seeds_dataset file contains the backspace key, and it is ok to reset the data

REPLY 
George Dong May 12, 2019 at 6:14 pm #

I echo that too!

Just one question please! In your code below, I could not understand why multiplication is used instead
of division in the last line. Though division caused divide by zero problem.
Your Start in Machine Learning
1 # Backpropagate error and store in neurons
2 def backward_propagate_error(network, expected):
3     for i in reversed(range(len(network))):
4         layer = network[i]
5         errors = list()
6         if i != len(network)-1:
7             for j in range(len(layer)):
8                 error = 0.0
9                 for neuron in network[i + 1]:
10                     error += (neuron['weights'][j] * neuron['delta'])
11                 
12                 errors.append(error)

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 25/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

13             for j in range(len(layer)):
14                 layer[j]['error'] = errors[j]
15         else:
16             for j in range(len(layer)):
17                 neuron = layer[j]
18                 errors.append(expected[j] - neuron['output'])
19                 neuron['error']=expected[j] - neuron['output']
20         for j in range(len(layer)):
21             neuron = layer[j]
22             neuron['delta'] = errors[j] * transfer_derivative(neuron['output'])
23 ##            neuron['delta'] = errors[j] / transfer_derivative(neuron['output'])

My understanding is gradient = dError / dWeights. Therefore, dWeights = dError / gradient

i.e. delta = errors[j] / derivative

Did we somehow make changes here, for calculation reasons, to use arctan instead of tan for gradient?

I’d be grateful if you could help.

Your Start in Machine ×
Learning
REPLY 
MO November 8, 2016 at 9:26 am # You can master applied Machine Learning
without math or fancy degrees.
where can i see your data set, i want to see how Find
it looked likein this free and practical course.
out how

Email Address
REPLY 
Jason Brownlee November 8, 2016 at 10:01 am #
START MY EMAIL COURSE
Hi MO.

The small contrived dataset used for testing is listed inline in the post in section 4.2

The dataset used for the full example is on the UCI ML repository, linked in the section titled “Wheat
Seeds Dataset”. Here is the direct link:
http://archive.ics.uci.edu/ml/datasets/seeds

REPLY 
prakash November 11, 2016 at 12:40 am #

in two class classification for 0 the expected value is [1,0] for 1 its is [0,1].
how will be the output vectors for more than two class??
Your Start in Machine Learning

REPLY 
Jason Brownlee November 11, 2016 at 10:02 am #

Hi prakash,

For multi-class classification, we can extend the one hot encoding.

Three class values for “red”, “green” “blue” can be represented as an output vector like:
1, 0, 0 for red

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 26/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

0, 1, 0 for green
0, 0, 1 for blue

I hope that helps.

REPLY 
Rakesh November 13, 2016 at 3:41 pm #

Hi, Jason.
You’ve mentioned that there are 3 output classes.
How do we check the values which come under the 3 classes / clusters?
Could we print the data which fall under each class?

Your Start in Machine ×

Learning REPLY 
Jason Brownlee November 14, 2016 at 7:35 am #
You can master applied Machine Learning
Hi Rakesh,
without math or fancy degrees.
The data does belong to 3 classes. We can check theFind
skill out howmodel
of our in this by
freecomparing
and practical
thecourse.
predicted
classes to the actual/expected classes and calculate an accuracy measure.
Email Address

Alex November 16, 2016 at 12:35 pm # START MY EMAIL COURSE REPLY 

I’m confused why the activation method iterates from 0 to len(inputs) – 1 instead of from 0 to
len(weights) – 1. Am I missing something?

REPLY 
Jason Brownlee November 17, 2016 at 9:47 am #

Hi Alex,

The length of weights is the length of the input + 1 (to accommodate the bias term).

We add the bias term first, then we add the weighted inputs. This is why we iterate over input values.

Does that help?

Your Start in Machine Learning

REPLY 
Alex November 17, 2016 at 12:29 pm #

When I step through the code above for the ‘forward_propagate’ test case, I see the code
correctly generate the output for the single hidden node but that output doesn’t get correctly
processed when determining the outputs for the output layer. As written above in the activate
function ‘for i in range(len(inputs)-1):’, when the calculation gets to the activate function for the

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 27/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

output node for class=0, since ‘inputs’ has a single element in it (the output from the single hidden
node), ‘len(inputs) – 1’ equals 0 so the for loop never executes. I’m assuming the code is supposed
to read ‘for i in range(len(weights) -1):’ Does that make sense?

I’m just trying to make sure I don’t fundamentally misunderstand something and improve this post for
other readers. This site has been really, really helpful for me.

REPLY 
Jason Brownlee November 18, 2016 at 8:27 am #

I’m with you now, thanks for helping me catch-up.

Nice spot. I’ll fix up the tutorial.

Update: Fixed. Thanks again mate! Your Start in Machine ×

Learning
You can master applied Machine Learning
REPLY 
Tomasz Panek November 21, 2016 at 1:23 am # without math or fancy degrees.
Find out how in this free and practical course.
# Update network weights with error
def update_weights(network, row, l_rate):
Email Address
for i in range(len(network)):
inputs = row
if i != 0: START MY EMAIL COURSE
inputs = [neuron[‘output’] for neuron in network[i – 1]]
for neuron in network[i]:
for j in range(len(inputs)-1):
neuron[‘weights’][j] += l_rate * neuron[‘delta’] * inputs[j]
neuron[‘weights’][-1] += l_rate * neuron[‘delta’]

In this fragment:
for j in range(len(inputs)-1):
neuron[‘weights’][j] += l_rate * neuron[‘delta’] * inputs[j]
neuron[‘weights’][-1] += l_rate * neuron[‘delta’]

If inputs length = 1, you are not updating weights, it’s correct? You are updating only bias, because in hidden
layer is only one neuron.

Your Start in Machine Learning

REPLY 
Tomasz November 21, 2016 at 1:34 am #

Hello. In method update_weight you are doing for j in range(len(inputs) – 1). If inputs lenght = 1,
you aren’t updating weights. It’s correct? Hidden layer have one neuron so in output layer weights aren’t
updated

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 28/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Jason Brownlee November 22, 2016 at 6:54 am #

Hi Tomasz,

The assumption here is that the input vector always contains at least one input value and an output
value, even if the output is set to None.

You may have found a bug though when updating the layers. I’ll investigate and get back to you.

REPLY 
Jason Brownlee January 3, 2017 at 10:17 am #

Thanks Tomasz, this was indeed a bug.

Your Start in Machine ×
I have updated the update_weights() function in the above code examples.
Learning
You can master applied Machine Learning
without math or fancy degrees.
REPLY 
Jerry Jones October 16, 2018 at 8:18 amFind
#
out how in this free and practical course.

I don’t understand how update_weights updates the NN. There is no global variable or
return from the function. What am I missing? Email Address

START MY EMAIL COURSE

Jason Brownlee October 16, 2018 at 2:33 pm #

The weights are passed in by reference and modified in place.

This is an advanced tutorial, I’d recommend using Keras for beginners.

REPLY 
Michael December 13, 2016 at 4:15 am #

Hi, Thanks for the tutorial, I’m doing a backpropagation project at the moment so its been really
useful.

I was a little confused on the back-propagation error calculation function. Does “if i != len(network)-1:” mean
Your Start in Machine Learning
that if the current layer isn’t the output layer then this following code is run or does it mean that the current
layer is an output layer?

REPLY 
Jason Brownlee December 13, 2016 at 8:08 am #

Glad to hear it Michael.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 29/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

The line means if the index i is not equal to the index of the last layer of the network (the output layer),
then run code inside the block.

REPLY 
Michael January 5, 2017 at 7:53 am #

I have another question.

Would it be possible to extend the code from this tutorial and create a network that trains using the MNIST
handwritten digit set? using a input unit to represent each pixel in the image. I’m also not sure whether/how I
could use feature extractors for the images.

I have a project where I have to implement the Backpropagation algorithm with possibly the MNIST
handwritten digit training set.
Your Start in Machine ×
I hope my question makes sense!
Learning
You can master applied Machine Learning

Jason Brownlee January 5, 2017 at 9:42 am # without math or fancy degrees. REPLY 

Find out how in this free and practical course.

Sure Michael, but I would recommend using a library like Keras instead as this code is not
written for performance. Email Address
Load an image as a long list of pixel integer values, convert to floats and away you go. No feature
extraction needed for a simple MLP implementation. You should
START MYget performance
EMAIL COURSE above 90%.

REPLY 
Calin January 6, 2017 at 10:40 pm #

Hi Jason,

Great post!

I have a concern though:

In train_network method there are these two lines of code:

expected = [0 for i in range(n_outputs)]

expected[row[-1]] = 1
Your Start in
Couldn’t be the case that expected[row[-1]] = 1 will throw IndexError, as Machine
n_outputsLearning
is the size of the training
set which is a subset of the dataset and row basically contains values from the whole dataset?

REPLY 
Jason Brownlee January 7, 2017 at 8:37 am #

Hi Calin,

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 30/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

If I understand you correctly, No. The n_outputs var is the length of the number of possible output
values.

Maybe put some print() statements in to help you better understand what values variables have.

REPLY 
Calin January 7, 2017 at 9:48 pm #

Hmm..I ran the entire code (with the csv file downloaded from
http://archive.ics.uci.edu/ml/datasets/seeds), added some breakpoints and this is what I got after a
few iterations:

n_outputs = 168
row[-1] = 201
Your Start in Machine ×
which is causing IndexError: list assignment index out of range.
Learning
You can master applied Machine Learning
without math or fancy degrees.
REPLY 
Adriaan January 11, 2017 at 4:27 am # Find out how in this free and practical course.

I’ve got the same error, That my list assignment index is out of range
Email Address

START MY EMAIL COURSE

Jason Brownlee January 11, 2017 at 9:29 am #

Sorry to hear that, did you try running the updated code?

Ivan January 16, 2017 at 10:28 am #

This is error of csv read. Try to reformat it with commas. For me it worked

Jason Brownlee January 16, 2017 at 10:45 am #

Your Start in Machine Learning
What was the problem and fix exactly Ivan?

Bob February 5, 2017 at 10:59 am #

The data file (http://archive.ics.uci.edu/ml/machine-learning-

databases/00236/seeds_dataset.txt) has a few lines with double tabs (\t\t) as the delimiter
— removing the double tabs and changing tabs to commas fixed it.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 31/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Thanks for the good article.

Jason Brownlee February 6, 2017 at 9:42 am #

Thanks for the note Bob.

Rowen Bruce October 20, 2018 at 8:52 pm #

updated code

Your Start in Machine ×

Learning REPLY 
Adriaan January 11, 2017 at 5:50 am #
You can master applied Machine Learning
I’ve had the same error at the ‘train_network’without
function. Is your
math dataset
or fancy fine? I’ve had some
degrees.
problems because the CSV file wasn’t loaded correctly dueout
Find tohow
my regional windows
in this free settings.
and practical I’ve had to
course.
adjust my settings and everything worked out alright.

Email Address
http://superuser.com/questions/783060/excel-save-as-csv-options-possible-to-change-comma-to-pipe-
or-tab-instead

START MY EMAIL COURSE

REPLY 
Stanley January 8, 2017 at 3:15 pm #

Thanks for such a good article.

Just one question: in the equation “weight = weight + learning_rate * error * input”, why there is an “input”?
IMO it should be: “weight = weight + learning_rate * error”?

REPLY 
Jason Brownlee January 9, 2017 at 7:47 am #

The var names and explanation are correct.

The update equation is: Your Start in Machine Learning

1 weight = weight + learning_rate * error * input

For the input layer the input are the input data, for hidden layers the input is the output of the prior layer.

REPLY 
Madwadasa January 13, 2017 at 3:31 am #

Jason,
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 32/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Thanks for the code and post.

Why is “expected” in expected = [0 for i in range(n_outputs)] initialized to [0,0] ?
Should not the o/p values be taken as expected when training the model ?
i.e for example in case of Xor should not 1 be taken as the expected ?

REPLY 
Jason Brownlee January 13, 2017 at 9:16 am #

Hi Madwadasa,

Expected is a one-hot encoding. All classes are “0” expect the actual class for the row which is marked
as a “1” on the next line.

Your Start in Machine ×

Learning REPLY 
Michael January 19, 2017 at 3:44 am #
You can master applied Machine Learning
Hello, I have a couple more questions. When training the network with a dataset, does the error at
without math or fancy degrees.
each epoch indicate the distance between the predicted outcomes and the expected outcomes together for
Find out how in this free and practical course.
the whole dataset? Also when the mean accuracy is given in my case being 13% when I used the MNIST
digit set, does this mean that the network will be correct 13% of the time and would have an error rate of
Email Address
87%?

START MY EMAIL COURSE

REPLY 
Jason Brownlee January 19, 2017 at 7:38 am #

Hi Michael,

The epoch error does capture how wrong the algorithm is on all training data. This may or may not be a
distance depending on the error measure used. RMSE is technically not a distance measure, you could
use Euclidean distance if you like, but I would not recommend it.

Yes, in generally when the model makes predictions your understanding is correct.

REPLY 
Bernardo Galvão January 24, 2017 at 3:51 am #
Your Start in Machine Learning
Hi Jason,

in the excerpt regarding error of a neuron in a hidden layer:

“Where error_j is the error signal from the jth neuron in the output layer, weight_k is the weight that connects
the kth neuron to the current neuron and output is the output for the current neuron.”

is the k-th neuron a neuron in the output layer or a neuron in the hidden layer we’re “on”? What about the
current neuron, are you referring to the neuron in the output layer? Sorry, english is not my native tongue.

Appreciate your work!

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 33/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Bernardo

REPLY 
anonymous February 1, 2017 at 1:42 am #

It would have been better if recall and precision were printed. Can somebody tell me how to print
them in the above code.

REPLY 
Jason Brownlee February 1, 2017 at 10:51 am #

You can learn more about precision and recall here:

https://en.wikipedia.org/wiki/Precision_and_recall Your Start in Machine ×
Learning
You can master applied Machine Learning
REPLY 
kehinde kolade February 6, 2017 at 8:29 pm # without math or fancy degrees.
Find out how in this free and practical course.
Hello Jason, great tutorial, I am developer and I do not really know much about this machine
learning thing but I need to extend this your code to incorporate the Momentum aspect to the training, can
Email Address
you please explain how I can achieve this extension?

START MY EMAIL COURSE

REPLY 
Jason Brownlee February 7, 2017 at 10:14 am #

Sorry, I don’t have the capacity to write or spell out this change for you.

My advice would be to read a good book on the topic, such as Neural Smithing: http://amzn.to/2ld9ds0

REPLY 
ibrahim February 18, 2017 at 2:21 am #

Hi Jason,
I have my own code written in C++, which works similar to your code. My intention is to extend my code to
convolutional deep neural nets, and i have actually written the convolution, Relu and pooling functions
Your used
however i could not begin to apply the backpropagation i have Startininmy
Machine
shallowLearning
neural net, to the
convolutional deep net, cause i really cant imagine the transition of the backpropagation calculation between
the convolutional layers and the standard shallow layers existing in the same system. I hoped to find a
source for this issue however i always come to the point that there is a standard backpropagation algorithm
given for shallow nets that i applied already. Can you please guide me on this problem?

REPLY 
Jason Brownlee February 18, 2017 at 8:42 am #

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 34/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

I”d love to guide you but I don’t have my own from scratch implementation of CNNs, sorry. I’m
not best placed to help at the moment.

I’d recommend reading code from existing open source implementations.

Good luck with your project.

REPLY 
matias February 22, 2017 at 3:34 pm #

Thank you, I was looking for exactly this kind of ann algorith. A simple thank won’t be enough tho
lol

Your Start in Machine ×

Jason Brownlee February 23, 2017 at 8:52 am # Learning REPLY 

I’m glad it helped. You can master applied Machine Learning

without math or fancy degrees.
The best way to help is to share the post with other people, or maybe purchase one of my books to
Find out how in this free and practical course.
support my ongoing work:
http://machinelearningmastery.com/products
Email Address

START MY EMAIL COURSE

REPLY 
Manohar Katam February 26, 2017 at 3:40 pm #

Great one! .. I have one doubt .. the dataset seeds contains missing features/fields for some rows..
how you are handling that …

REPLY 
Jason Brownlee February 27, 2017 at 5:49 am #

You could set the missing values to 0, you could remove the rows with missing values, you
could impute the missing values with mean column values, etc.

Try a few different methods and see what results in the best performing models.

Your Start in Machine Learning

REPLY 
Manohar Katam March 1, 2017 at 2:59 pm #

What if I have canonical forms like “male” or “female” in my dataset… Will this program
work even with string data..

REPLY 
Jason Brownlee March 2, 2017 at 8:11 am #
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 35/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Hi Manohar,

No, you will need to convert them to integers (integer encoding) or similar.

REPLY 
Wissal ARGOUBI February 27, 2017 at 11:12 pm #

Great job! this is what i was looking for ! thank you very much .
However i already have a data base and i didn’t know how to make it work with this code how can i adapt it
on my data
Thank you

Your Start in Machine ×

REPLY 
Jason Brownlee February 28, 2017 at 8:10 am #
Learning
This process will help you work through your predictive modeling problem:
You can master applied Machine Learning
http://machinelearningmastery.com/start-here/#process
without math or fancy degrees.
Find out how in this free and practical course.

Shweta Gupta March 5, 2017 at 4:37 am # Email Address REPLY 

Thanks for such a great article..

START MY EMAIL COURSE
I have one question, in update_weights why you have used weight=weight+l_rate*delta*input rather than
weight=weight+l_rate*delta?

REPLY 
Jason Brownlee March 6, 2017 at 10:55 am #

You can learn more about the math in the book on the topic.

I recommend Neural Smithing: http://amzn.to/2ld9ds0

REPLY 
Sittha March 13, 2017 at 1:23 pm #
Your Start in Machine Learning
Thanks for a good tutorial.
I have some IndexError: list assignment index out of range. And I cannot fix it with comma or full-stop
separator.

REPLY 
Jason Brownlee March 14, 2017 at 8:11 am #

What is the full error you are getting?

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 36/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Did you copy-paste the full final example and run it on the same dataset?

REPLY 
Sittha March 24, 2017 at 3:36 am #

line 151 :
expected[row[-1]] = 1
IndexError : list assignment index out of range

REPLY 
Jason Brownlee March 24, 2017 at 8:00 am #

Is this with a different dataset? Your Start in Machine ×

Learning
You can master applied Machine Learning
without
Benji Weiss May 11, 2017 at 5:31 am # math or fancy degrees.
Find out how in this free and practical course.
if it is a different dataset, what do i need to do to not get this error

Email Address

REPLY 
Karan March 16, 2017 at 6:26 pm # START MY EMAIL COURSE

The dataset that was given was for training the network. Now how do we test the network by
providing the 7 features without giving the class label(1,2 or 3) ?

REPLY 
Jason Brownlee March 17, 2017 at 8:27 am #

You will have to adapt the example to fit the model on all of the training data, then you can call
predict() to make predictions on new data.

REPLY 
Karan March 19, 2017 at 7:43 pm # Your Start in Machine Learning

Ok Jason, i’ll try that and get back to you! Thank you!

REPLY 
Karan March 19, 2017 at 7:48 pm #

Just a suggestion for the people who would be using their own dataset(not the seeds_dataset) for
training their network, make sure you add an IF loop as follows before the 45th line :

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 37/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

if minmax[i][1]!=minmax[i][0]

This is because your own dataset might contain same values in the same column and that might cause a
divide by zero error.

REPLY 
Jason Brownlee March 20, 2017 at 8:16 am #

Thanks for the tip Karan.

REPLY 
Li Qun March 25, 2017 at 5:45 pm #
Your Start in Machine ×
Thanks jason for the amazing posts of your from scratch pyhton implementations! i have learned so
much from you! Learning
Youposts,
I have followed through both your naive bayes and backprop can master
and Iapplied
have aMachine Learning
(perhaps quite naive)
question: without math or fancy degrees.
Find out how in this free and practical course.
what is the relationship between the two? did backprop actually implement bayesian inference (after all,
what i understand is that bayesian = weights being updated every cycle) already? perhaps just non-
Email Address
gaussian? so.. are non-gaussian PDF weight updates not bayesian inference?

i guess to put it simply : is backpropagation essentially a bayesian inference loop for an n number of
epochs? START MY EMAIL COURSE

I came from the naive bayes tutorial wanting to implement backpropagation together with your naive bayes
implementation but got a bit lost along the way.

sorry if i was going around in circles, i sincerely hope someone would be able to at least point me on the
right direction.

REPLY 
Jason Brownlee March 26, 2017 at 6:11 am #

Great question.

No, they are both very different. Naive bayes is a direct use of the probabilities and bayes theorem. The
neural net is approximating a mapping function from inputs and outputs
Your Start – a very
in Machine different approach that
Learning
does not directly use the joint probability.

REPLY 
Chiraag March 26, 2017 at 10:10 pm #

How did you decide that the number of folds will be 5 ? Could you please explain the significance of
this number. Thank You.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 38/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Jason Brownlee March 27, 2017 at 7:54 am #

In this case, it was pretty arbitary.

Generally, you want to split the data so that each fold is representative of the dataset. The objective
measure is how closely the mean performance reflect the actual performance of the model on unseen
data. We can only estimate this in practice (standard error?).

REPLY 
Li Qun March 27, 2017 at 10:19 pm #

Dear Jason,
Your between
thank you for the reply! I read up a bit more about the differences Start in Machine
Naive Bayes (or Bayesian Nets in
×
Learning
general) and Neural Networks and found this Quora answer that i thought was very clear. I’ll put it up here to
give other readers a good point to go from:
You can master applied Machine Learning
https://www.quora.com/What-is-the-difference-between-a-Bayesian-network-and-an-artificial-neural-network
without math or fancy degrees.
TL:DR : Find out how in this free and practical course.
– they look the same, but every node in a Bayesian Network has meaning, in that you can read a Bayesian
network structure (like a mind map) and see what’s happening
Emailwhere and why.
Address
– a Neural Network structure doesn’t have explicit meaning, its just dots that link previous dots.
– there are more reasons, but the above two highlighted the biggest difference.
START MY EMAIL COURSE
Just a quick guess after playing around with backpropagation a little: the way NB and backprop NN would
work together is by running Naive Bayes to get a good ‘first guess’ of initial weights that are then run through
and Neural Network and Backpropagated?

REPLY 
Jason Brownlee March 28, 2017 at 8:23 am #

Please note that a Bayesian network and naive bayes are very different algorithms.

REPLY 
Melissa March 27, 2017 at 10:54 pm #
Your Start in Machine Learning
Hi Jason,
Further to this update:

Update Jan/2017: Changed the calculation of fold_size in cross_validation_split() to always be an integer.

Fixes issues with Python 3.

I’m still having this same problem whilst using python 3, on both the seeds data set and my own. It returns
an error at line 75 saying ‘list object has no attribute ‘sum” and also saying than ‘an integer is required.’

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 39/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Any help would be very much appreciated.

Overall this code is very helpful. Thank you!

REPLY 
Jason Brownlee March 28, 2017 at 8:24 am #

Sorry to hear that, did you try copy-paste the complete working example from the end of the
post and run it on the same dataset from the command line?

REPLY 
Melissa March 28, 2017 at 9:29 am #

Your
Yes I’ve done that, but still the same problem! Start in Machine ×
Learning
You can master applied Machine Learning
REPLY 
david March 29, 2017 at 6:16 am # without math or fancy degrees.
Find out how in this free and practical course.
Hello jason,

please i need help on how to pass the output of the trainedEmail Address
network into a fuzzy logic system if possible a
code or link which can help understand better. Thank you

START MY EMAIL COURSE

REPLY 
Aditya April 2, 2017 at 3:57 pm #

Awesome Explanation

REPLY 
Jason Brownlee April 4, 2017 at 9:05 am #

Thanks!

Raunak Jain April 6, 2017 at 5:20 pm # Your Start in Machine Learning REPLY 

Hello Jason
I m getting list assignment index out or range error. How to handle this error?

REPLY 
Jason Brownlee April 9, 2017 at 2:37 pm #

The example was developed for Python 2, perhaps this is Python version issue?
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 40/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Marco April 6, 2017 at 9:37 pm #

Thanks but I think python is not a good choice…

REPLY 
Jason Brownlee April 9, 2017 at 2:40 pm #

I think it is a good choice for learning how backprop works.

What would be a better choice?

Your Start in Machine ×

Learning
REPLY 
Agrawal April 6, 2017 at 9:38 pm #
You can master applied Machine Learning
withoutNetwork.
Hey, Jason Thanks for this wonderful lecture on Neural math or fancy degrees.
Find out how in this free and practical course.
As I am working on Iris Recognition, I have extracted the features of each eye and store it in .csv file, Can u
suggest how further can I build my Backpropagation code.
As when I run your code I am getting many errors. Email Address
Thank you
START MY EMAIL COURSE

REPLY 
Jason Brownlee April 9, 2017 at 2:40 pm #

This process will help you work through your modeling problem:
http://machinelearningmastery.com/start-here/#process

REPLY 
Jack April 7, 2017 at 3:42 pm #

Could you please convert this iterative implementation into matrix implementation?

Your Start in Machine Learning

REPLY 
Jason Brownlee April 9, 2017 at 2:52 pm #

Perhaps in the future Jack.

REPLY 
Jk April 12, 2017 at 5:04 am #

Hi Jason,
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 41/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

In section 4.1 , may you please explain why you used ### inputs = row[:-1] ### ?

Thanks

REPLY 
Jason Brownlee April 12, 2017 at 7:58 am #

Yes. By default we are back-propagating the error of the expected output vs the network output
(inputs = row[:-1]), but if we are not the output layer, propagate the error from the previous layer in the
network (inputs = [neuron[‘output’] for neuron in network[i – 1]]).

I hope that helps.

Your Start in Machine ×

JK April 13, 2017 at 3:59 am # Learning REPLY 

You you
Thanks for your respond. I understand what can master applied
said , the part IMachine Learning
am no understanding is the
[:-1] . why eliminating the last list item ? without math or fancy degrees.
Find out how in this free and practical course.

Email Address
REPLY 
Jason Brownlee April 13, 2017 at 10:10 am #

It is a range from 0 to the second last item in the

START list, e.g.
MY EMAIL (0 to n-1)
COURSE

REPLY 
Amer April 6, 2018 at 7:22 am #

Because the last Item in the weights array is the biass

REPLY 
Prem Puri April 12, 2017 at 8:18 pm #

In function call, def backward_propagate_error(network, expected):

how much i understand is , it sequentially pass upto
if i != len(network)-1: Your Start in Machine Learning
for j in range(len(layer)):
error = 0.0
for neuron in network[i + 1]:
error += (neuron[‘weights’][j] * neuron[‘delta’])
My question is which value is used in neuron[‘delta’]

REPLY 
Jason Brownlee April 13, 2017 at 10:01 am #
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 42/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

delta is set in the previous code block. It is the error signal that is being propagated backward.

REPLY 
Nishu March 25, 2018 at 11:32 am #

I’m sorry, but I still can’t find the location where delta is set and hence, the code gives error.
Where is the delta set for the first time?

REPLY 
Prem Puri April 14, 2017 at 3:20 am #

Thanks very much!

Your Start in Machine ×
Learning
REPLY 
Jason Brownlee April 14, 2017 at 8:54 am # You can master applied Machine Learning
without math or fancy degrees.
You’re welcome. Find out how in this free and practical course.

Email Address
REPLY 
youssef oumate April 26, 2017 at 4:53 pm #
START MY EMAIL COURSE
Hi Jason

Thank you very much for this awesome implementation of neural network,
I have a question for you : I want to replace the activation function from Sigmoid
to RELU . So, what are the changes that I should perform in order to get
correct predictions?

REPLY 
Jason Brownlee April 27, 2017 at 8:34 am #

I think just a change to the transfer() and transfer_derivative() functions will do the trick.

Your Start in Machine Learning

REPLY 
youssef oumate April 27, 2017 at 10:17 am #

Awesome !

Thank you so much

REPLY 
Jason Brownlee April 28, 2017 at 7:26 am #

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 43/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

You’re welcome.

REPLY 
Yahya Alaa April 30, 2017 at 2:38 am #

Hi Jason,
Thank you very much for this wonderful implementation of Neural Network, it really helped me a lot to
understand neural networks concept,

n_inputs = len(dataset[0]) – 1
n_outputs = len(set([row[-1] for row in dataset]))
network = initialize_network(n_inputs, 2, n_outputs)
train_network(network, dataset, 0.5, 20, n_outputs)
Your Start in Machine ×
What do n_inputs and n_outputs refer to? According to the small dataset used in this section, is n_inputs
Learning
only 2 and n_outputs only 2 (0 or 1) or I am missing something?

You can master applied Machine Learning

without math or fancy degrees.
Find out how in this free and practical course. REPLY 
Jason Brownlee April 30, 2017 at 5:31 am #

Input/outputs refers to the number of input andEmail

output features (columns) in your data.
Address

START MY EMAIL COURSE

REPLY 
Yahya Alaa May 3, 2017 at 1:42 pm #

Is the program training the network for 500 epochs for each one of the k-folds and then testing
the network with the testing data set?

REPLY 
Jason Brownlee May 4, 2017 at 8:02 am #

Hi Yahya,

5-fold cross validation is used.

That means that 5 models are fit and evaluated on 5 different hold out sets. Each model is trained
for 500 epochs. Your Start in Machine Learning

I hope that makes things clearer Yahya.

REPLY 
Yahya Alaa May 4, 2017 at 8:17 am #

Yes you made things clear to me, Thank you.

I have two other questions,

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 44/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

How to know when to stop training the network to avoid overfitting?

How to choose the number of neurons in the hidden layer?

Jason Brownlee May 5, 2017 at 7:27 am #

You can use early stopping, to save network weights when the skill on a validation
set stops improving.

The number of neurons can be found through trial and error.

Yahya Alaa May 6, 2017 at 8:48 am Your

# Start in Machine ×
Learning
I am working on a program that recognizes handwritten digits, the dataset is
consisting of pictures (45*45) pixels each, which is 2025 input neurons, this causes me a
You can master applied Machine Learning
problem in the activation function, the summation of (weight[i] * input[i]) is big, then it gives
without math or fancy degrees.
me always a result of (0.99 -> 1) after putting
Find the
out value
how inof thefree
this activation function
and practical in the
course.
Sigmoid function, any suggestions?

Email Address

Jason Brownlee May 7, 2017 at 5:31START

am # MY EMAIL COURSE

I would recommend using a Convolutional Neural Network rather than a Multilayer

Perceptron.

REPLY 
morok April 30, 2017 at 3:56 am #

In section 3.2. Error Backpropagation, where did output numbers came from for testing
backpropagation

‘output’: 0.7105668883115941
‘output’: 0.6213859615555266
‘output’: 0.6573693455986976
Your Start in Machine Learning
Perhaps from outputs on test forward propagation [0.6629970129852887, 0.7253160725279748] taking dd -
> derivative = output * (1.0 – output), problem is they don’t match, so I’m a bit lost here…

thanks!

Awesome article!!!

REPLY 
Jason Brownlee April 30, 2017 at 5:34 am #

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 45/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

In that example, the output and weights were contrived to test back propagation of error. Note
the “delta” in those outputs.

REPLY 
Massa November 25, 2017 at 7:36 am #

hello Dr Jason…

I was wondering …

n_outputs = len(set([row[-1] for row in dataset]))

this line, how does it give the number of output features?

when I print it gives the number of the dataset(number of rows, not columns)
Your Start in Machine ×
Learning
REPLY 
Jason Brownlee November 25, 2017 at 10:25You
amcan
# master applied Machine Learning

without math or fancy degrees.

The length of the set of values in the finalFind
column.
out how in this free and practical course.
Perhaps this post will help with Python syntax:
https://machinelearningmastery.com/index-slice-reshape-numpy-arrays-machine-learning-python/
Email Address

START MY EMAIL COURSE

REPLY 
Massa November 26, 2017 at 4:02 am #

but I thought it gives the number of outputs…I mean the number of neurons in the
output layer.

here it’s giving the number of the dataset ….if I have 200 input/output pairs it prints 200

so I am confused…how would it be?

Jason Brownlee November 26, 2017 at 7:35 am #

If there are two class values, it should print 2. It should not print the number of
examples. Your Start in Machine Learning

REPLY 
Umamaheswaran May 8, 2017 at 9:49 pm #

Hi Jason,

I am using the MNIST data set to implement a handwritten digit classifier. How many training examples will
be needed to get a perfomance above 90%.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 46/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Jason Brownlee May 9, 2017 at 7:42 am #

I would recommend using a CNN on MNIST. See this tutorial:

http://machinelearningmastery.com/handwritten-digit-recognition-using-convolutional-neural-networks-
python-keras/

REPLY 
Huyen May 9, 2017 at 6:32 pm #

Hi Jason,

Your blog is totally awesome not only by this post but alsoYour Start
for the whole in Machine
series about neural network. Some ×
of them explained so much useful thing than others on Internet. They help me a lot to understand the core of
Learning
network instead of applying directly Keras or Tensorflow.

Just one question, if I would like to change the result fromYou can mastertoapplied
classification Machine
regression, Learning
which part in back
without math or fancy degrees.
propagation I need to change and how?
Find out how in this free and practical course.
Thank you in advance for your answer

Email Address

REPLY 
Jason Brownlee May 10, 2017 at 8:46 am # START MY EMAIL COURSE

Thanks Huyen.

You would change the activation function in the output layer to linear (e.g. no transform).

REPLY 
TGoritsky May 12, 2017 at 12:41 am #

Hi Jason,

I am playing around with your code to better understand how the ANN works. Right now I am trying to do
predictions with a NN, that is trained on my own dataset, but the program returns me one class label for all
rows in a test dataset. I understand, that normalizing dataset should help, but it doesn`t work (I am using
your minmax and normalize_dataset functions). Also, is there
YouraStart
way toinreturn prediction
Machine Learningfor one-dimensional
dataset?
Here is the code (sorry for lack of formatting):
def make_predictions():
dataset = [[29,46,107,324,56,44,121,35,1],
[29,46,109,327,51,37,123,38,1],
[28,42,107,309,55,32,124,38,1],
[40,112,287,59,35,121,36,1],
[27,43,129,306,75,41,107,38,1],
[28,38,127,289,79,40,109,37,1],
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 47/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

[29,37,126,292,77,35,100,34,1],
[30,40,87,48,77,51,272,80,2],
[26,37,88,47,84,44,250,80,2],
[29,39,91,47,84,46,247,79,2],
[28,38,85,45,80,47,249,78,2],
[28,36,81,43,76,50,337,83,2],
[28,34,75,41,83,52,344,81,2],
[30,38,80,46,71,53,347,92,2],
[28,35,72,45,64,47,360,101,2]]
network = [[{‘weights’: [0.09640510259345969, 0.37923370996257266, 0.5476265202749506,
0.9144446394025773, 0.837692750149296, 0.5343300438262426, 0.7679511829130964,
0.5325204151469501, 0.06532276962299033]}],
[{‘weights’: [0.040400453542770665, 0.13301701225112483]}, {‘weights’: [0.1665525504275246,
Your
0.5382087395561351]}, {‘weights’: [0.26800994395551214, Start in Machine
0.3322334781304659]}]] ×
# minmax = dataset_minmax(dataset)
# normalize_dataset(dataset, minmax)
Learning
for row in dataset: You can master applied Machine Learning
prediction = predict(network, row) without math or fancy degrees.
print(‘Expected=%d, Got=%d’ % (row[-1], prediction)) Find out how in this free and practical course.

Email Address
REPLY 
Jason Brownlee May 12, 2017 at 7:43 am #
START MY EMAIL COURSE
I would suggest exploring your problem with the Keras framework:
http://machinelearningmastery.com/start-here/#deeplearning

REPLY 
Tomo May 18, 2017 at 6:22 pm #

Hi Jason!
In the function “backward_propagate_error”, when you do this:

neuron[‘delta’] = errors[j] * transfer_derivative(neuron[‘output’])

The derivative should be applied on the activation of that neuron, not to the output . Am I right??

neuron[‘delta’] = errors[j] * transfer_derivative(activate(neuron[‘weights’], inputs))

Your Start in Machine Learning
And inputs is:
inputs = row[-1]
if i != 0:
inputs = [neuron[‘output’] for neuron in self.network[i-1]]

Thank you! The post was really helpful!

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 48/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Tina May 26, 2017 at 3:49 am #

Hello Jason!

This is a very interesting contribution to the community

Have you tried using the algorithm with other activation functions?
I tried with Gaussian, tanh and sinx, but the accuracy was not that high, so I think that I omitted something.
What I altered were the activation functions and the derivatives. Is there something else that needs to be
changed?

REPLY 
Jason Brownlee June 2, 2017 at 11:49 am #
Your Start in Machine ×
Sigmoid was the defacto standard for many years because it performs well on many different
problems.
Learning
Now the defacto standard is ReLU. You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.

REPLY 
Manu June 6, 2017 at 8:50 pm # Email Address

Sigmoid and ReLU are transfer functions right ?

Activation function is just the sum of all weights andSTART
inputsMY EMAIL COURSE

REPLY 
Jason Brownlee June 7, 2017 at 7:12 am #

You are correct, but in some frameworks, transfer functions are called activation
functions:
https://keras.io/activations/

REPLY 
vishwanathan May 27, 2017 at 8:08 pm #

Thanks for the great post. Here is some observation

Yourthat I am
Start in not able toLearning
Machine understand. In the back
ward propagate you are not taking all the weights and only considering the jth. Can you kindly help
understand. I was under the impression that the delta from output is applied across all the weights,
for neuron in network[i + 1]:
error += (neuron[‘weights’][j] * neuron[‘delta’])

REPLY 
vishwanathan May 27, 2017 at 8:14 pm #

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 49/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

I understand you do not want to take in the bias weight hence the exclusion of the last weight in
neuron. I kind of get stumped on bias.

REPLY 
vishwanathan May 27, 2017 at 9:12 pm #

Thanks for the great article. In the backward propagate, the delta value is applied for each weight
across the neuron and the error is summed. I am curious why is the delta not applied to individual weights of
the neuron and the error summed for that neuron. Can you please clarify?

Josue May 29, 2017 at 3:12 am #

Your Start in Machine × REPLY 

Learning
Why don’t you split the data into TrainData and TestData, like 80% of the dataset for training and
20% for testing, because if you train with 100% of rows of the dataset and then test some rows of the
dataset the accuracy will be good . But if you put new dataYou
on can
the master applied
seeds.csv the Machine Learning
model will work with less
accuracy, Right? without math or fancy degrees.
Find out how in this free and practical course.

Email Address
REPLY 
Jason Brownlee June 2, 2017 at 12:16 pm #

You can, k-fold cross validation generally givesSTART

a better
MYestimate of model performance.
EMAIL COURSE

Once we have the estimate and choose our model, we can fit the final model on all available data and
make predictions on new data:
http://machinelearningmastery.com/train-final-machine-learning-model/

REPLY 
Josue May 29, 2017 at 11:08 am #

Thanks for the post! I have a question about cross-validation. The dataset of seeds is perfect for 5
folds but for a dataset of 211? I’ll have uniformly sized subset right? (211/5) Can you give me a suggestion
how I could handle that ?
Thanks in advanced.

Your Start in Machine Learning

REPLY 
Jason Brownlee June 2, 2017 at 12:20 pm #

One way is that some records can be discarded to give even sized groups.

REPLY 
Sebastián May 30, 2017 at 9:35 am #

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 50/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Thanks so much for the tutorial. It was really helpful!

REPLY 
Jason Brownlee June 2, 2017 at 12:31 pm #

I’m glad it helped.

REPLY 
Manu June 10, 2017 at 9:00 pm #

Hello Jason,

any advice on how to handle multi-classifier problems when the classes

Your Starthave
inhigh cardinality ?
Machine ×
I’m thinking about input data of search engines linked to choosen urls.
Learning
You can master applied Machine Learning
without math or fancy degrees. REPLY 
Jason Brownlee June 11, 2017 at 8:25 am #
Find out how in this free and practical course.

Ouch, consider modeling it as regression instead (e.g. a rating or recommender system).

Email Address

START MY EMAIL COURSE REPLY 

Manuel June 13, 2017 at 1:17 am #

Ok thank you very much Jason.

But it wont work with searches unseen by the algorithm.
I red something in the books “Programming collective intelligence” about a neural net from scratch
for this king of problem but I don’t understang how it works for the moments…

REPLY 
Jason Brownlee June 13, 2017 at 8:23 am #

Consider focusing on one measure/metric that really matters in your domain, then try a
suite of framings of the problem and different algorithms to get a feeling for what might work
best.
Your Start in Machine Learning

REPLY 
Yash June 18, 2017 at 6:21 pm #

I am not able to understand the above code.So, I request you to explain me the above code

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 51/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Jason Brownlee June 19, 2017 at 8:43 am #

Which part do you not understand exactly?

REPLY 
Tathagat June 21, 2017 at 3:20 pm #

Hey Jason..am a novice in machine learning..have a small question…how can I track the timesteps
involved in the algorithm with accordance with the code?

Jason Brownlee June 22, 2017 at 6:04 am # Your Start in Machine ×

REPLY 

Learning
What do you mean by time steps?
You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.
REPLY 
bazooka June 29, 2017 at 6:52 am #
Email Address
Hi, Jason. I am so confused, in the result, why there are 4 set of [output,weight,delta]

like this:
START MY EMAIL COURSE
[{‘output’: 0.9999930495852168, ‘weights’: [0.9315463130784808, 1.0639526745114607,
0.9274685127907779], ‘delta’: -4.508489650980804e-09}, {‘output’: 0.9992087809233077, ‘weights’:
[-2.4595353900551125, 5.153506472345162, -0.5778256160239431], ‘delta’: 1.940550145482836e-06}]
[{‘output’: 0.01193860966265472, ‘weights’: [2.3512725698865053, -8.719060612965613,
1.944330467290268], ‘delta’: -0.0001408287858584854}, {‘output’: 0.988067899681387, ‘weights’:
[-2.2568526798573116, 8.720113230271012, -2.0392501730513253], ‘delta’: 0.0001406761850156443}]

after the backpropagation we find the optimal weights to get minimum error, what does these 4 group
means?
E

REPLY 
Jason Brownlee June 29, 2017 at 7:48 am #
Your Start in Machine Learning
That is the internal state of the whole trained network.

REPLY 
hassan June 29, 2017 at 7:30 am #

hi Jason
thanks for your code and good description here, i like it so much.
i run your example code and encounter with an error same others whom left note here

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 52/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

the error is:

expected[row[-1]] = 1
IndexError: list assignment index out of range

how i can fix this error?

REPLY 
Jason Brownlee June 29, 2017 at 7:49 am #

The code was written for Python 2.7, confirm that this is your Python version.

Also confirm that you have copied the code exactly.

Your Start in Machine ×

Jerome July 5, 2017 at 9:20 pm # Learning REPLY 

Dear Jason, You can master applied Machine Learning

without math or fancy degrees.
i have this question about Back Propagate Error Find out how in this free and practical course.
1- derivative sigmoid = output * (1.0 – output)
That is ok Email Address

2- error = (expected – output) * transfer_derivative(output)

Ok but it also means that error == 0 for output = 1 whateverSTART
the expected is because
MY EMAIL COURSE transfer_derivative(1)
==0

So, whatever the expected , error is nil if output is 1 …

Is there something rotten here?

Thanks

Jerome

REPLY 
wddddds July 10, 2017 at 10:01 pm #

Thank you Jason, It’s a great tutorial and really helpful for me!

But I have to say that trying to reimplement your code strongly

Your increased my ability
Start in Machine of debugging
Learning

REPLY 
Jason Brownlee July 11, 2017 at 10:32 am #

Thanks.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 53/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Victor July 17, 2017 at 7:50 pm #

Hi Jason,

Thanks for sharing your code. I’m a PhD candidate in machine learning, and I have a doubt about the
weights update in section 4.1:

weight = weight + learning_rate * error * input

Should not it be as follows?

weight = weight – learning_rate * error * input

Thanks again for sharing this.

Regards,
Your Start in Machine ×
Victor.
Learning
You can master applied Machine Learning
without math or fancy degrees. REPLY 
Víctor August 4, 2017 at 11:07 pm #
Find out how in this free and practical course.
I didn’t say anything, my mistake in understanding.
Email Address
Thanks again for sharing your work.

START MY EMAIL COURSE

REPLY 
vishnu priya July 22, 2017 at 4:26 pm #

Hi..
Thanks for ur coding. It was too helpful. can u suggest me how to use this code for classifying tamil
characters. i have tried in cnn and now i need to compare the result with bpn. can u pls suggest me.

thank you

REPLY 
Jason Brownlee July 23, 2017 at 6:20 am #

Perhaps this tutorial on classifying with a CNN would be more useful to you:
Your Start in Machine Learning
http://machinelearningmastery.com/handwritten-digit-recognition-using-convolutional-neural-networks-
python-keras/

REPLY 
vishnu priya July 23, 2017 at 4:06 pm #

Thank you sir. With this tutorial i have implemented cnn sir. but for BPN i am getting error rate
687.203 sir. i dnt know what to do sir. can u help me sir.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 54/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Thank you

REPLY 
Jason Brownlee July 24, 2017 at 6:49 am #

What is the problem exactly?

REPLY 
Vishnupriya July 24, 2017 at 4:53 pm #

Classification of Tamil characters sir. I have 144 different classes. I have taken 7 glcm features of
each character and I need to train this features in backpropagation and predict the character to which class
it belongs. Your Start in Machine ×
Learning
You can master applied Machine Learning
REPLY 
Jason Brownlee July 25, 2017 at 9:34 am # without math or fancy degrees.
Find out how in this free and practical course.
Sound like a great project!

Email Address

REPLY 
codeo July 26, 2017 at 5:37 pm # START MY EMAIL COURSE

Hi, so I wasn’t following this tutorial when implementing my neural network from scratch, and mine
is in JavaScript. I just need help with the theory. How do I calculate the error for each node in the net so that
I can incrementally change the weights? Great tutorial btw

REPLY 
codeo July 26, 2017 at 6:38 pm #

Hahaha nevermind, it was my code

Multidimensional arrays and stuff boggle the mind hah

Your Start in Machine Learning

REPLY 
Jason Brownlee July 27, 2017 at 7:56 am #

Glad to hear you worked it out.

REPLY 
PRABHAKARAN M July 31, 2017 at 4:31 pm #

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 55/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

[ 6.38491205 5.333345 4.81565798 5.43552204 9.96445304 2.57268919 4.07671018 1.5258789

6.19728301 0 1 ]
Dear sir,
the above mentioned numerical values are extracted from the dental x-ray image using gray level co
occurrence matrix [10 inputs and 1 output]. This dataset is used as a input for BPN classifier. whether the
same data set as[.csv] file can be used as the input for DEEP Convolutional Neural Network technique ?
and can i get the output as image ? for example if i give the dental x ray images as numerical values i have
to get the caries affected teeth as the output for the given dataset.

REPLY 
Jason Brownlee August 1, 2017 at 7:51 am #

That sounds like a great problem. It may be possible.

Your Start in Machine ×
I would recommend using deep CNNs.
Learning
Perhaps this tutorial will give you some ideas on how to get started:
You can master applied Machine Learning
http://machinelearningmastery.com/handwritten-digit-recognition-using-convolutional-neural-networks-
without math or fancy degrees.
python-keras/
Find out how in this free and practical course.
You may want to look at some papers on object localization in images. I don’t have material on it sorry.
Email Address

REPLY 
PRABHAKARAN M July 31, 2017 at 4:32 pm # START MY EMAIL COURSE

can i get the example code for dental caries detection using deep Convolutional Neural Network for
the given dataset as x ray images.

REPLY 
Jason Brownlee August 1, 2017 at 7:52 am #

I do not have sample code for this problem, sorry.

REPLY 
John August 1, 2017 at 3:26 am #
Your Start in Machine Learning
Very nice explanation, thank you.
I have some questions.

1) weight = weight + learning_rate * error * input

Do I really need to multiply it with input ? For example here

http://home.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html they dont it multiply it with input. At least, I think
that…

2) Is your method same as in http://home.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html?

i think yes, but again, Im not sure and Im confused by that input multiplication.
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 56/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

3) What is exactly loss function in your example (I usually found some derivations of loss (cost ?) function (in
other explanations), not transfer function derivation)? Im actually very confused by notation which I find
around …

4) momentum and weight decay. In your example, you can implement them that you substract calculated
decay and add calculated momentum (to weight update) ? Again, I found forms which substract both and
weight update as w + deltaW, so again I’m mega confused by notation for backpropagation which I found…

Sorry for dumb questions, … math is not my strong side, so many things which can be inferred by math
sense are simply hidden for me.

REPLY 
John August 1, 2017 at 3:30 am #
Your Start in Machine ×
*substract both and weight update as w + deltaW, so again
Learning
I found above sentence as nonsense, must be side effect of my confusion …
You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.
REPLY 
Jason Brownlee August 1, 2017 at 8:12 am #
Email Address
Hang in there.

Pick one tutorial and focus on it. Jumping from place to place will make things worse for sure.
START MY EMAIL COURSE

REPLY 
Jason Brownlee August 1, 2017 at 8:10 am #

Hi John, good questions.

According to my textbook, yes.

I can’t speak for random sites on the internet sorry.

Loss is prediction error. You can change this to other forms like MAE or MSE.

No decay or momentum in this example. Easy to add if you want. There are many ways to dial in the
learning process. No hard and fast rules, just some norms that people reuse.

Your Start in Machine Learning

REPLY 
Parminder Kaur August 6, 2017 at 7:50 pm #

A VERY GOOD TUTORIAL SIR…

Sir i am implementing remote sensed image classification using BPN neural network using IDL.
I am not finding good resources on constructing features for input dataset and also number of hidden layers
and number of neurons in hidden layer.
Any resources you know, can help me?

Thanks
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 57/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Jason Brownlee August 7, 2017 at 8:41 am #

The CNN will perform feature extraction automatically, you could explore using different filters
on the data to see if it helps the network.

The number of layers and neurons/filters per layer must be found using trial and error. It is common to
copy the designs from other papers as a starting point.

I hope that helps.

Your Start in Machine ×

REPLY 
pero August 9, 2017 at 1:11 am #
Learning
Nice tutorial, very clean and readable code. =) thank you!
You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.
REPLY 
Jason Brownlee August 9, 2017 at 6:37 am #
Email Address
Thanks pero.

START MY EMAIL COURSE

REPLY 
Vatandas August 15, 2017 at 3:28 am #

1. I expect that this code is deep learning (many hidden layer) but not. One sentence is easy (“you
can add more hidden layer as explained”) but to do is not as easy as you said.

2. I think your code is wrong.

neuron[‘delta’] = errors[j] * transfer_derivative(neuron[‘output’])
but
Error = Target – ActivatedOutputNode
Delta = Error * Derivative(NONActivatedOutputNode)

I mean you use the same ‘output’ variable both error and delta. But in error it must be activated one, in delta
it must be NONactivated one.
Your Start in Machine Learning

REPLY 
A Researcher May 2, 2019 at 3:10 am #

Exactly, this article is completely misleading :S

REPLY 
8CG_256 August 18, 2017 at 9:02 am #
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 58/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Nice tutorial, very clean code and beginner-friendly. Thank you very much!

REPLY 
Jason Brownlee August 18, 2017 at 4:36 pm #

Thanks, I’m glad you found it useful!

REPLY 
8CG_256 August 18, 2017 at 9:26 pm #

I only have one slight issue: I implemented this in Ruby and I tried to train it using the IRIS
dataset, keeping the network simple (1 input layer, 1 hidden layer, 1 output layer) and after decreasing
Your
for a while the error rate keeps increasing. I tried lowering the Start
learning in
rate,Machine
even making it dynamic so it
×
decreases whenever the error increases but it doesn’tLearning
seem to help. Could you give me some advice?
P.S sorry for my bad English
You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.
REPLY 
Jason Brownlee August 19, 2017 at 6:19 am #
Email Address
Here is an example of backprop I developed in Ruby:
http://cleveralgorithms.com/nature-inspired/neural/backpropagation.html
START MY EMAIL COURSE

REPLY 
Derek Martins August 22, 2017 at 9:22 pm #

Hi Jason, I enjoy so much your tutorials. Can you do a tutorial implementing BackPropagation
Through Time? Thanks man.

REPLY 
Jason Brownlee August 23, 2017 at 6:50 am #

Thanks for the suggestion.

I have a few posts on the general topic, for example:

Your Start in Machine Learning
https://machinelearningmastery.com/gentle-introduction-backpropagation-time/

REPLY 
Anubhav Singh August 24, 2017 at 1:08 pm #

Hello Jason,

Thank you for the great tutorial!

I would like to know how I can obtain the weight*input for every single neuron in the network…
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 59/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

I’ve been trying these lines –

for layer in network:

new_inputs = []
for neuron in layer:
activation = activate(neuron[‘weights’], inputs)
neuron[‘output’] = transfer(activation)
new_inputs.append(neuron[‘output’])

but the activation variable here is a single value…what I understand is that if I have set n_hidden = 5
(number of hidden layers), I should get N*5 (N = number of features in the dataset) outputs if I print the
activation…

Kindly help

Thank you! Your Start in Machine ×

Learning
You can master applied Machine Learning REPLY 
Jose Panakkel August 25, 2017 at 10:45 am #
without math or fancy degrees.
Find out how in this free and practical course.
Dear Jason,

I have a question on the delta calculation at the output layer, where

Email Address
the primary value is the difference between the neuron output and
the expected output. And we are then multiplying this difference
with the transfer_derivative. where transfer_derivative is a function
START MY EMAIL COURSE
of neuron’s output.

My question is, is it correct to find the difference between the

neuron’s output and the expected output?

In this case of the example, you have chosen digital outputs [0,1]
and hence it may not have come up .. but my point is…
one is already subjected to a transfer function, and one is not.

The neuron’s output is always subjected to a transfer function and

hence will be in a specific range, say -.5 to +.5 or something..
But the expected output is the user’s choice .. isnt it?
user can have an expected value of say 488.34, for some stock price
learning.. then is it still correct to find this primary difference
between the expected output and the neuron output, at the output
Your Start in Machine Learning
layer delta calculation?

shoulnt the expected output also be subjected to the same transfer

function before finding the difference? Or the otherway, like
shoulnt the neuron ouptut be subjected to a reverse transfer function
before comparing with the expected output directly?

Thanks and Regards,

Jose Panakkel

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 60/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
RealUser404 September 6, 2017 at 1:36 pm #

Hello Jason, great tutorial that helped me a lot!

I have a question concerning the back-propagation : what if instead of having an error function I only have a
desired gradient for the output (in the case of an actor-critic model for example)?
How can I change your backprop function to make it work? Or can I just use the gradient as the error?

REPLY 
Jason Brownlee September 7, 2017 at 12:49 pm #

Sorry, I don’t follow, perhaps you can restate your question with an example?
Your Start in Machine ×
Learning
You can master applied Machine Learning REPLY 
user28 September 8, 2017 at 9:26 pm #
without math or fancy degrees.
Hi Jason , thank you for providing this tutorial. I’mFind out howofinhow
confused this can
free Iand practical the
implement course.
same
backpropagation algorithm with output not binary. Since I noticed that your example has binary output. Like
predicting for stock price given the open, high, low and close values.
Email Regards.
Address

START MY EMAIL COURSE

REPLY 
Jason Brownlee September 9, 2017 at 11:55 am #

Use a library like Keras. Start here:

https://machinelearningmastery.com/start-here/#deeplearning

REPLY 
Lewis September 11, 2017 at 2:11 am #

Hi Jason,

great article. I have an interest in NN but I am not that good at python.

Want I wanted to try was to withhold say 5 rows from the dataset and have the trained network predict the
Your
results for those rows. these is is different from what I think the Start in Machine
example Learning
does which is rolling predictions
with the learning. Removing 5 rows from the dataset is of course easy but my pitiful attempts at predicting
with unseen data like below fail ((I guess network is not in scope at the end): any help appreciated!

# predict unseen data

unseendataset = [[12.37,13.47,0.8567,5.204,2.96,3.919,5.001],
[12.19,13.2,0.8783,5.137,2.981,3.631,4.87],
[11.23,12.88,0.8511,5.14,2.795,4.325,5.003],
[13.2,13.66,0.8883,5.236,3.232,8.315,5.056],

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 61/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

[11.84,13.21,0.8521,5.175,2.836,3.598,5.044],
[12.3,13.34,0.8684,5.243,2.974,5.637,5.063]]

for row in unseendataset:

prediction2 = predict(network, row)
print(‘Predicted=%d’ % (prediction2))

REPLY 
Jason Brownlee September 11, 2017 at 12:08 pm #

I would recommend starting with Keras rather than coding the algorithm from scratch.

Start here:
https://machinelearningmastery.com/start-here/#deeplearning
Your Start in Machine ×
Learning
You can master applied Machine Learning REPLY 
Karim September 14, 2017 at 1:27 pm #
without math or fancy degrees.
Find outtohow
Hi Jason, I am trying to generalize your implementation in this
work withfree and practical
a variable numbercourse.
of layers
and nodes. However, whenever I try to increase the number of nodes too much it stops working (the network
freezes at one error rate and all output nodes are active, i.e. giving
Email 1). Although the code would work if I
Address
decreased the layers and the errors will go down.
Is there something I am missing when using too many layers? The concepts should be the same.
START MY EMAIL COURSE
I trained a network with 4 layers: [14,10,10,4] and it worked.
I trained a network with 4 layers [14,100,40,4] and it is stuck. Same dataset.

My code is here if you are looking in more details:

https://github.com/KariMagdy/Implementing-a-neural-network

Thanks

REPLY 
Jason Brownlee September 15, 2017 at 12:10 pm #

What problem do you get exactly?

Your Start in Machine Learning

REPLY 
Laksh October 4, 2017 at 11:11 pm #

Hi, Jason Brownlee,

can we extend this code for 2 or more hidden layers ?

REPLY 
Jason Brownlee October 5, 2017 at 5:24 am #

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 62/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Sure.

REPLY 
dsliver33 October 9, 2017 at 1:52 pm #

Dear Mr. Brownlee,

I’m trying to alter the code to represent a regression problem (sigmoid on hidden layer, linear on output
layer). As far as I know, the main part of the code that would have to be modified is the FF algorithm. I’ve
rewritten the code as below:

1 # Forward propagate input to a network output

2 def forward_propagate_regression(network, row):
3     inputs = row
4
5
    new_inputs = []
    #gets the 1st layer, applies sigmoid activation
Your Start in Machine ×
6     hiddenlayer = network[0]
7     for neuron in hiddenlayer:
Learning
8         activation = activate(neuron['weights'], inputs)
9 You can master applied Machine Learning
        neuron['output'] = transfer(activation)
10         new_inputs.append(neuron['output']) without math or fancy degrees.
11     inputs = new_inputs
Find out how in this free and practical course.
12     #gets the last layer, applies linear activation
13     outputlayer = network[-1]
14     for neuron in outputlayer:
15 Email Address
            activation = activate(neuron['weights'], inputs)
16             neuron['output'] = activation
17             new_inputs.append(neuron['output'])
18     inputs = new_inputs START MY EMAIL COURSE
19     return inputs

With this code, I’m getting an “OverflowError: (34, ‘Result too large’)” error. Could you please tell what I’m
doing wrong? All the other parts of the code are as you’ve written.

REPLY 
Jason Brownlee October 9, 2017 at 4:47 pm #

What did you change exactly? Can you highlight the change for me?

Also, try using pre tags.

Your Start in Machine Learning REPLY 

dsliver33 October 10, 2017 at 4:08 am #

(I don’t know how to highlight the change, sorry!)

I got the hidden layer (network[0]), and I applied your algorithm (calculate activation, transfer the
activation to the output, append that to a new list called “new_inputs”).

After that, I get the output layer (network[-1]), I calculate the activation with the “new_inputs”, but I
do NOT apply the sigmoid transfer function (so, the outputs should be linear). The results are
appended to a new list, which is set to be the return of the function.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 63/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Would that be the best way to remove the sigmoid function from the output layer, making the code a
regression, instead of a classification?

REPLY 
Jason Brownlee October 10, 2017 at 7:52 am #

Sounds good. I don’t have any good ideas, I’d recommend stepping through some
calculations to help spot where it is going wrong.

You may want to consider moving to an open source neural net library, such as Keras:
https://machinelearningmastery.com/start-here/#deeplearning

Your Start in Machine ×

REPLY 
Liam McGoldrick October 26, 2017 at 5:23 am #
Learning
I am having the same issue with mine. i made alterations and they are just the same as yours.
You can master applied Machine Learning
Did you find a solution? without math or fancy degrees.
Find out how in this free and practical course.

Email Address REPLY 

Liam McGoldrick October 26, 2017 at 5:27 am #

I GOT IT TO WORK!!! You have to normalize your output data. Then you can apply the transfer
START MY EMAIL COURSE
function to the output layer just the same! After that it will work!

REPLY 
Steven August 20, 2019 at 8:40 pm #

But didn’t you changed the function ‘train_network’ ???

REPLY 
Chris October 12, 2017 at 11:27 am #

Hi Jason, nice posting and it really helps a lot

for j in range(len(layer)):
Your Start in Machine Learning
neuron = layer[j]
neuron[‘delta’] = errors[j] * transfer_derivative(neuron[‘output’])
Should the neuron[‘output’] be the output of the activation function instead of the transfer function here?

REPLY 
Asad October 14, 2017 at 3:24 pm #

hi jason, nice post its really helps alot.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 64/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

please tell me how we can change the neuron in hidden layer and in output layer?
and what will be the result when we change the neuron in hidden layer and in output layer?
in this tutorial u take one hidden layer,so can we use more than one hidden layer? and how?

please tell me i m waiting

REPLY 
Jason Brownlee October 15, 2017 at 5:19 am #

Perhaps you would be better served by starting with a neural network library such as Keras:
https://machinelearningmastery.com/tutorial-first-neural-network-python-keras/

Your Start in Machine ×

REPLY 
dsliver33 October 16, 2017 at 2:27 pm #
Learning
Dear Mr. Brownlee,
You can master applied Machine Learning
without
I’m trying to adapt the code to support many hidden layers. math orthe
I’ve adapted fancy
codedegrees.
as below, with a new input
called “n_layers”, to insert N hidden layers in the network.Find out how in this free and practical course.

# Initialize a network with “n_layers” hidden layers

Email Address
def initialize_network3(n_inputs, n_hidden, n_layers, n_outputs):
network = list()
for i in range(n_layers): START MY EMAIL COURSE
hidden_layer = [{‘weights’:[random() for i in range(n_inputs + 1)]} for i in range(n_hidden)]
network.append(hidden_layer)
output_layer = [{‘weights’:[random() for i in range(n_hidden)]} for i in range(n_outputs)]
network.append(output_layer)
return network

When I try to run the code, it shows the error below. Do you have any idea why?

in backward_propagate_error(network, expected)
78 error = 0.0
79 for neuron in network[i + 1]:
—> 80 error += (neuron[‘weights’][j] * neuron[‘delta’])
81 errors.append(error)
82 else:
Your Start in Machine Learning
IndexError: list index out of range

REPLY 
dna_remaps February 3, 2018 at 10:43 pm #

This took me a minute to figure out myself.

You need to add a conditional after your first layer to make sure your subsequent hidden layer weights
have the proper dimensions (n_hidden+1, n_hidden)
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 65/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

for i in range(n_layers):
hidden_layer = [{‘weights’:[random() for i in range(n_inputs + 1)]} for i in range(n_hidden)]
if i > 0:
hidden_layer = [[{‘weights’:[random() for i in range(n_hidden + 1)]} for i in range(n_hidden)]
network.append(hidden_layer)

REPLY 
Arijit Mukherjee October 17, 2017 at 1:40 am #

Hi,

In the output/last layer when we are calculating the backprop error why are we multiplying with the transfer
derivative with the (expected-output)?? transfer derivative is already canceled out for the the last layer , the
Your
update should be only (expected-output)*previous_layer_input Start in Machine
, ??? ×
Thanks
Learning
You can master applied Machine Learning
without math or fancy degrees.
REPLY 
Tanoh Henry October 18, 2017 at 8:54 pm # Find out how in this free and practical course.

Really good article. Thanks a lot.

Email Address
Need a little bit of clarification.
For backward propagation starting at the output layer,
you get the error by appending to errors expected[j] – neuron[‘output’].
START MY EMAIL COURSE
Isn’t Error = 0.5 * sum(errors)?
and then using this sum of errors for back-propagation?
Thanks.

REPLY 
Liam October 21, 2017 at 5:41 am #

Thanks for the tutorial! I am trying to modify your code to do a regression model and I am stuck. I
have an input data set (4 columns and many rows) and a single variable output data set (in range of tens of
thousands). I fed them into the train procedure and I get an error when it reaches “expected = [0 for i in
range(n_outputs)]” in the train portion. The error reads “only length-1 arrays can be converted to Python
scalar”. Now I understand this is because of the intended purpose for the code was a categorization problem
but I am wondering what I would need to modify to get thisYour
to work?
StartAny help would
in Machine go a long way as I have
Learning
been stuck on this issue for some time now.

Thanks, and again wonderful tutorial!

REPLY 
Jason Brownlee October 21, 2017 at 5:45 am #

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 66/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Perhaps start with Keras, it will be much easier for you:

https://machinelearningmastery.com/regression-tutorial-keras-deep-learning-library-python/

REPLY 
Sam October 26, 2017 at 11:15 pm #

Hi
I am implementing a 2 layer neural network with 100 hidden units in the first layer and 50
in the next using your code. Implement sigmoid activation function in each layer. Train/test your
model on the MNIST dataset subset.
But it is always giving same prediction.
[0.99999999986772, 0.99999999994584]
Expected=0, Got=1
Your Start in Machine ×
[0.99999999986772, 0.99999999994584]
Expected=1, Got=1 Learning
[0.99999999986772, 0.99999999994584]
You can master applied Machine Learning
Expected=1, Got=1
without math or fancy degrees.
[0.99999999986772, 0.99999999994584]
Find out how in this free and practical course.
Expected=1, Got=1
[0.99999999986772, 0.99999999994584]
Expected=1, Got=1 Email Address
[0.99999999986772, 0.99999999994584]
Expected=0, Got=1
START MY EMAIL COURSE
[0.99999999986772, 0.99999999994584]
Expected=0, Got=1
[0.99999999986772, 0.99999999994584]
Expected=1, Got=1
[0.99999999986772, 0.99999999994584]
Expected=0, Got=1
[0.99999999986772, 0.99999999994584]
Expected=0, Got=1
[0.99999999986772, 0.99999999994584]
Expected=0, Got=1
[0.99999999986772, 0.99999999994584]
Expected=0, Got=1

Your Start in Machine Learning

REPLY 
Jason Brownlee October 27, 2017 at 5:20 am #

I would recommend using a framework like Keras:

https://machinelearningmastery.com/handwritten-digit-recognition-using-convolutional-neural-networks-
python-keras/

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 67/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Matthias April 15, 2018 at 2:57 am #

Weights should be initialized with normally distributed random values. Try using random.gauss
for weight initialization.

REPLY 
John October 28, 2017 at 6:52 pm #

help, I dont know why i got this error.

Traceback (most recent call last):

File “a.py”, line 185, in
for i in range(len(dataset[0])-1): Your Start in Machine ×
TypeError: ‘NoneType’ object has no attribute ‘__getitem__’
Learning
You can master applied Machine Learning
without math or fancy degrees.
Jason Brownlee October 29, 2017 at 5:52 am # Find out how in this free and practical course. REPLY 

You cannot have the “-1” within the call to len()

Email Address

START MY EMAIL COURSE REPLY 

João Costa November 12, 2017 at 12:46 pm #

Hey Jason, thanks for your post!

This is helping me a lot with a college work. But in this NN, how can I set manually not the number of input
neuros, the input values?

For example, if I have 1 input neuro, I wan’t to set this value to 0.485.

Best regards!

REPLY 
Jason Brownlee November 13, 2017 at 10:11 am #

Sorry, I don’t follow. Your Start in Machine Learning

Perhaps you’d be better off using a library like Keras:

https://machinelearningmastery.com/tutorial-first-neural-network-python-keras/

REPLY 
yesta November 17, 2017 at 8:21 am #

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 68/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Hi, Jason
Thank you for this amazing tutorial!

I have a question that may be out of the topic. How do you call models or type of DL models where you feed
a model with new test data in order to make the model adaptive to the environment?

Thank you.

REPLY 
Jason Brownlee November 17, 2017 at 9:31 am #

Yes, you can update a model after it has been trained.

Your Start in Machine ×

Nil December 6, 2017 at 7:47 pm # Learning REPLY 

Hi, Dr. Jason, You can master applied Machine Learning

without math or fancy degrees.
I have been studying how to develop a neural network from scratch and this tutorial is the main one I have
Find out how in this free and practical course.
been following because it is helping me so much.
I have a doubt: When I study the theory I see the neural network scheme carrying only the weights and bias.
Email Address
And here in practice I see that the network is also carrying the output values and the delta i.e (weights, bias,
output and delta). Will the final model be saved like this? with the latter (weights, bias, output and delta)?
would this be the rule in practice? START MY EMAIL COURSE

I would appreciate it if you could help with this issue so that I could get out of where I left off.

Your posts are really very good there is where I find my way in to learning in Machine Learning.

Best Regards

REPLY 
Jason Brownlee December 7, 2017 at 7:51 am #

The final model (e.g. trained) only needs to perform the forward pass.

Nil December 8, 2017 at 5:50 am # Your Start in Machine Learning REPLY 

Understood.
Thank you Dr. Jason

REPLY 
MohamedElshazly December 8, 2017 at 9:30 pm #

Hi , there’s something i don’t understand :

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 69/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

1 for i in reversed(range(len(network))):
2 layer = network[i]
3 errors = list()
4 if i != len(network)-1:
5 for j in range(len(layer)):
6 error = 0.0
7 for neuron in network[i + 1]:

wouldn’t the last line be out of range because the current ‘ i ‘ is the last one and i can’t go beyond it by 1 ?
thanks in advance

REPLY 
Jason Brownlee December 9, 2017 at 5:41 am #

No, because of the “if” check on the 4th line down.

Your Start in Machine ×
Learning
REPLY 
Olu December 9, 2017 at 3:17 am # You can master applied Machine Learning
without math or fancy degrees.
Hi Mr Brownlee, Find out how in this free and practical course.

Thank you for your tutorial. The training for the example worked however when I try to implement the code
for the Wheat Seeds Dataset I get an error from my line 210:Email Address

for i in range(len(dataset[0]) – 1):

str_column_to_float(dataset, i) START MY EMAIL COURSE

The error is: IndexError: list index out of range

Can you please explain why it is (dataset[0])? Does (dataset[0]) means the 1st column in the dataset?

REPLY 
Jason Brownlee December 9, 2017 at 5:45 am #

Yes, I recommend learning more about Python arrays here:

https://machinelearningmastery.com/index-slice-reshape-numpy-arrays-machine-learning-python/

The example was written for Python 2.7, confirm your Python version.

Your Start in Machine Learning

REPLY 
Jonesy December 12, 2017 at 3:17 pm #

Hello Jason,

Fantastic stuff here. I had a question about the network configuration. This 2 input, 2 hidden and 2 output
seems a bit odd to me. I’m used to seeing 2, 2, 1 for XOR – can you explain why you have two output nodes
and how they work with the max function? I think it would better explain this line for me in train():

expected[row[-1]] = 1

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 70/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

And lastly, why would one choose this configuration over a 2, 2, 1.

Thanks!

REPLY 
Jason Brownlee December 12, 2017 at 4:12 pm #

The final model has the shape [7, 5, 3]. Perhaps check the tutorial again? It has 3 outputs, one
for each of the 3 classes in the dataset.

Configuration was chosen via trial and error. There is no analytical way to choose a configuration for a
neural network.

Finally, you can learn more about array indexing in Python here:
Your Start in Machine
https://machinelearningmastery.com/index-slice-reshape-numpy-arrays-machine-learning-python/ ×
Learning
You can master applied Machine Learning
REPLY 
Mohamed Elshazly December 17, 2017 at 9:34 pm # without math or fancy degrees.
Find out how in this free and practical course.
Hi Jason

In the tain_network function the line “expected[row[-1]] = 1”Email

what Address
i understand is that you take the Y value of
every row (which is either 0 or 1 ) and use it as an index in the expected array and you change the value at
that index to 1 ,First i don’t know if i understand that correctly in the first place or not but if so, Wouldn’t the
START
modification to the expected array be locked down to just only MY EMAIL
the first COURSE
and second index because
“expected[row[-1]] = 1” would only be expected[0] or expected[1] ? and how would that help in our algorithm
.

looking forward to your response and thanks for the Great Tutorial

REPLY 
Jason Brownlee December 18, 2017 at 5:22 am #

We are one hot encoding the variable.

Learn more about it here:

https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/

Your Start in Machine Learning

REPLY 
MohamedElshazly December 18, 2017 at 4:55 pm #

HI again Jason

If I’m implementing this algorithm in python 3 what should i change in expected[row[-1]]=1 in order for it to
work because I’m having this error : list assignment index out of range
thanks in advance

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 71/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Jason Brownlee December 19, 2017 at 5:15 am #

I don’t know off the cuff, I will look into porting the example to Py3 in the new year.

REPLY 
Tushar December 19, 2017 at 5:29 pm #

You are just awesome Jason. You are adding more value to people’s ML skills than most average
graduate schools do in the US.

Thanks a ton!

Your Start in Machine ×

Jason Brownlee December 20, 2017 at 5:39 am #Learning REPLY 

You can master applied Machine Learning

Thanks Tushar.
without math or fancy degrees.
Find out how in this free and practical course.

Email Address REPLY 

mark January 6, 2018 at 3:29 am #

Wow, thanks for your codes. I have a question, what if I want to add regularisation term like L2
START MY EMAIL COURSE
during back propagation, what should i do?

REPLY 
Jason Brownlee January 6, 2018 at 5:55 am #

I would recommend moving to a platform like Keras:

https://machinelearningmastery.com/start-here/#deeplearning

REPLY 
mark January 7, 2018 at 5:12 pm #

Thanks for replying. I know the keras and have been using keras for a while. But in the
Your
problem I am focusing on, I need to make changes Start
on the in Machine
back Learning
propagation. That’s why I didn’t use
keras.
So let’s go back to my original question, is the error term the cost function? Thanks.

REPLY 
Jason Brownlee January 8, 2018 at 5:42 am #

Sorry, I cannot work-through adding regularization to this tutorial for you.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 72/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Mojo January 20, 2018 at 7:04 pm #

Hello Jeson,
Thanks for the informative tutorial. I have a question.
if i want to change the error equation and as well as the equation between input with hidden and hidden with
output layer. How can i change it?
Hope you will reply in a short time.

Regards,
Mojo

Your Start in Machine ×

REPLY 
Jason Brownlee January 21, 2018 at 9:08 am #
Learning
I would recommend using a library instead, it will be much easier for you.
You can master applied Machine Learning
Here’s how to get started with Keras: without math or fancy degrees.
https://machinelearningmastery.com/start-here/#deeplearning
Find out how in this free and practical course.

Email Address

REPLY 
Aliya Anil February 16, 2018 at 9:04 pm #
START MY EMAIL COURSE
Hi Jason,

It was indeed a very informative tutorial. Could you please explain the need for seed(1) in the code?

REPLY 
Jason Brownlee February 17, 2018 at 8:44 am #

I am trying to tie down the random number generator so that you get the same results as me.

Learn more here:

https://machinelearningmastery.com/randomness-in-machine-learning/

Your Start in Machine Learning

REPLY 
Raj February 19, 2018 at 7:10 am #

Hey there,
Been following your tutorial and I’m having problems with using my dataset with it. The outputs of the hidden
neurons appear to only be exactly 1 constantly. I’m not sure what’s wrong exactly or how to fix it but its
resulting in the network not learning at all. Please let me know if you can help.
Thanks,
Raj

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 73/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Jason Brownlee February 19, 2018 at 9:10 am #

Perhaps try to get the code and data in the tutorial working first and use that as a starting point
for your own problem.

Generally, I would recommend using a library like Keras for your own projects and only code methods
from scratch as a learning exercise.

REPLY 
Aliya Anil February 20, 2018 at 4:37 pm #

Hi,
Your
I tried the first code in the tutorial with 4-parameter dataset, but it isStart in Machine
not predicting like the 2-parameter set.
×
Could you explain the reason? Learning
Thanks, You can master applied Machine Learning
Aliya without math or fancy degrees.
Find out how in this free and practical course.

Email Address REPLY 

Jason Brownlee February 21, 2018 at 6:36 am #

If you are looking to develop a neural network for your own data, I would recommend the Keras
START MY EMAIL COURSE
library:
https://machinelearningmastery.com/start-here/#deeplearning

REPLY 
Nik March 2, 2018 at 1:54 pm #

Dear Jason,

Can I use the codes for handwritten digits recognition? If yes, are there any special recommendations what
to change in the codes or I can use them with no changes?

Thanks,
Nik
Your Start in Machine Learning

REPLY 
Jason Brownlee March 2, 2018 at 3:25 pm #

I would recommend this tutorial:

https://machinelearningmastery.com/handwritten-digit-recognition-using-convolutional-neural-networks-
python-keras/

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 74/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Nik March 2, 2018 at 7:51 pm #

Yes, I have seen that tutorial. But is there any way to use the codes from this tutorial? I just
would like to understand why they work so well for seeds and do not work for handwritten digits…

REPLY 
Jason Brownlee March 3, 2018 at 8:10 am #

Yes, you can. You can develop a model with all pixels as inputs.

I cannot write the modification for you, sorry, I just don’t have the capacity.

Your Start in Machine ×

REPLY 
Filoingko March 4, 2018 at 2:58 am #
Learning
Hi, You can master applied Machine Learning
How can I use this trained network to predict another datawithout
set. math or fancy degrees.
Find out how in this free and practical course.
Thank you.

Email Address

REPLY 
Jason Brownlee March 4, 2018 at 6:04 am # START MY EMAIL COURSE

The code in this tutorial is to teach you about backprop, not for use on real problems. If you are
working through a problem, I’d recommend using Keras.

REPLY 
Jean-Michel Richer March 5, 2018 at 9:27 pm #

Dear Jason,
I have tried to use your code on a simple XOR example but get a result of [0, 0, 1, 1] instead of [0,1,1,0]
Scores: [0.0]
Mean Accuracy: 0.000%

The input xor.csv file is

0,0,0 Your Start in Machine Learning
0,1,1
1,0,1
1,1,0

For this I have modified the evaluate_algorithm function to:

def evaluate_algorithm_no_fold(dataset, algorithm, *args):
scores = list()
predicted = algorithm(dataset, dataset, *args)
print(predicted)

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 75/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

accuracy = accuracy_metric(dataset, predicted)

scores.append(accuracy)
return scores

and call the function like this:

scores = evaluate_algorithm_no_fold(dataset, back_propagation, 0.1, 500, 4)

Would you have some explanation because I can not figure out why it is not working ?
Best regards,
JM

REPLY 
Jason Brownlee March 6, 2018 at 6:12 am #
Your Start in Machine ×
Perhaps the model requires tuning to your new dataset.
Learning
You can master applied Machine Learning
without math or fancy degrees. REPLY 
Tanveer March 5, 2018 at 9:29 pm #
Find out how in this free and practical course.
Thank You So Much Jason !! Wonderful Tutorial. THANKS Much !!
Email Address

START MY EMAIL COURSE REPLY 

Jason Brownlee March 6, 2018 at 6:12 am #

You’re welcome.

REPLY 
Mojo March 9, 2018 at 10:06 pm #

If i want to calculate the training accuracy and F-measure and want to change the activation
function, how i can do it?

REPLY 
Jason Brownlee March 10, 2018 at 6:28 am #
Your Start in Machine Learning
Perhaps you would be better off using scikit-learn and Keras instead.

REPLY 
Fahad March 12, 2018 at 8:03 pm #

Is there something wrong with this code in case of using MINIST data? I tried to change the
structured of the data to be compatible with the code, but it gave me a huge error and the error did not
decrees during all training steps

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 76/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Jason Brownlee March 13, 2018 at 6:25 am #

The code was not developed for MNIST. Here is an example of working with MNIST:
http://machinelearningmastery.com/handwritten-digit-recognition-using-convolutional-neural-networks-
python-keras/

REPLY 
Fahad March 13, 2018 at 4:06 pm #

Thanks Jason for your response. I want to apply the code without keras. I tried to change the
Your
structure of the data to be each row as a vector of 784 pixel Start
followed in Machine
by a class label, but as I said it gave a ×
huge error and does not decrees at all.
Learning
I am trying to develop some algorithm for enhancing of learning, hence, I need to deal with the procedure as
step by step. So keras or any other library does not help. You can master applied Machine Learning
without math or fancy degrees.
Thanks again Jason Find out how in this free and practical course.

Email Address
REPLY 
Jason Brownlee March 14, 2018 at 6:17 am #
START MY EMAIL COURSE
Perhaps update the code to use numpy, it will be much faster.

REPLY 
kelvin March 15, 2018 at 2:39 am #

Hi Mr Brownlee,

Can you teach me how to plot the errors per epochs (validation error) and accuracy for both training and
validation in your scratch network?

REPLY 
kelvin March 15, 2018 at 2:44 am #
Your Start in Machine Learning
I only can find the training error but not validation error in the code. For the accuracy, I plot a
graph have a straight line only.

REPLY 
Jason Brownlee March 15, 2018 at 6:33 am #

Yes, you can do it easily in Keras here:

https://machinelearningmastery.com/display-deep-learning-model-training-history-in-keras/

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 77/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
kelvin March 15, 2018 at 12:19 pm #

Is there any possible way to do it on your scratch network? for example which part of the
code save the training error, validation error, training accuracy and validation accuracy? So I can
plot the graph myself since your scratch model does not have “model” for me to save the history.

REPLY 
Jason Brownlee March 15, 2018 at 2:50 pm #

Yes, perhaps change it from CV to a single/train test, then evaluate the model skill on
each dataset at the end of each epoch. Save Your Start
the results inandMachine
in a list return the lists. ×
Learning
You can master applied Machine Learning REPLY 
Zahra May 6, 2019 at 9:37 am # without math or fancy degrees.
Find out how in this free and practical course.
Hello, I’m so confuse..
I try to run this code in command prompt. But, I use my dataset (not Wheat Seeds dataset).
Email Address
And why this happened? What’s wrong? What should I do? What should I change?
Please, help me!
START MY EMAIL COURSE
Traceback (most recent call last):
File “journal.py”, line 197, in
scores = evaluate_algorithm(dataset, back_propagation, n_folds, l_rate, n_epoch, n_hidden)
File “journal.py”, line 81, in evaluate_algorithm
predicted = algorithm(train_set, test_set, *args)
File “journal.py”, line 173, in back_propagation
train_network(network, train, l_rate, n_epoch, n_outputs)
File “journal.py”, line 150, in train_network
expected[row[-1]] = 1
IndexError: list assignment index out of range

REPLY 
Jason Brownlee May 6, 2019 at 2:33 pm
Your
# Start in Machine Learning

Sorry, I cannot debug your dataset.

Perhaps start with Keras for deep learning instead:

https://machinelearningmastery.com/start-here/#deeplearning

REPLY 
Jack March 15, 2018 at 12:11 pm #

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 78/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Can I use this model for regression problem? For example us this model for boston house-prices
dataset?

REPLY 
Jason Brownlee March 15, 2018 at 2:50 pm #

Sure, some changes would be required, such as the activation in the output layer would need
to be linear.

REPLY 
Nabil March 15, 2018 at 3:49 pm #

Are you using MSE? Your Start in Machine ×

Learning
You can master applied Machine Learning
without math or fancy degrees. REPLY 
Jason Brownlee March 16, 2018 at 6:09 am #
Find out how in this free and practical course.
As mentioned in the post, we are reporting accuracy for the classification problem.
Email Address

REPLY 
Olu March 19, 2018 at 11:23 pm # START MY EMAIL COURSE

In the train section,

1 def train_network(network, train, l_rate, n_epoch, n_outputs):

2     for epoch in range(n_epoch):
3         for row in train:
4             outputs = forward_propagate(network, row)
5             expected = [0 for i in range(n_outputs)]
6             expected[row[-1]] = 1
7             backward_propagate_error(network, expected)
8             update_weights(network, row, l_rate)

Can you please explain how this expected[row[-1]] = 1 knows where to insert the 1 in the arrays of zero
created.

Your Start in Machine Learning

REPLY 
Jason Brownlee March 20, 2018 at 6:23 am #

Good question.

expected is all zeros. row[-1] is the index of the class value. therefore we set the index of the class value
in expected to 1.

Perhaps it is worth reading up on array indexing:

https://machinelearningmastery.com/index-slice-reshape-numpy-arrays-machine-learning-python/

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 79/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
kmillen November 16, 2018 at 11:27 am #

Jason,
This is an amazing piece of code that has been very beneficial.

Doesn’t that mean that only expected[0] and expected[1] will ever be set to 1 for this test data?

Thank you,

REPLY 
Jason Brownlee November 16, 2018 at 1:58 pm #

Your
Sorry, I don’t understand your question? Start in Machine ×
Learning
You can master applied Machine Learning
kmillen November 17, 2018 at 1:44 amwithout
# math or fancy degrees.
Find out how in this free and practical course.
If I understand Python (which I may not), row[-1] represents the last item in the row.
Since the last value in each of the 10 rows is only either 0 or 1, expected[row[-1]] = 1 will
Email Address
only ever set expected[0] or expected[1] to the value of 1. Or, what am I missing?

START MY EMAIL COURSE

Jason Brownlee November 17, 2018 at 5:50 am #

Are you referring to this: expected[row[-1]] = 1

If so:

“expected” is all zeros, e.g. [0, 0]

“row” is an example, e.g. […] where the value at -1 is either 0 or 1

Therefore row[-1] is an index of either 0 or 1 and we are marking the value in expected at
that index as 1.

We have created a one hot vector.

Your Start in Machine Learning

REPLY 
kmillen November 17, 2018 at 2:54 am #

Disregard my previous question; I found the answer in a previous reply. Thank you
again for this example.

REPLY 
kelvin March 21, 2018 at 2:14 am #

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 80/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Hi, I would like to use softmax as the activation function for output layer. However, I do not know
how to write the code for the derivative of softmax. Can you show me the code how to change the
sigmoid function from your code to softmax?

REPLY 
kelvin March 21, 2018 at 2:20 am #

I do try few ways to change the sigmoid to softmax, however, all of them are not working. Can
you show me how to create a softmax layer?

for transfer():
first case:
def transfer(input_value):
exp_scores = np.exp(input_value) Your Start in Machine ×
return exp_scores / np.sum(exp_scores, axis=1, keepdims=True)
Learning
second case:
You can master applied Machine Learning
def transfer(input_value):
without math or fancy degrees.
input_value -= np.max(input_value)
Find out how in this free and practical course.
return np.exp(input_value) / np.sum(np.exp(input_value))

third case: Email Address

def transfer(input_value):
input_value -= np.max(input_value)
START MY EMAIL COURSE
result = (np.exp(input_value).T / np.sum(np.exp(input_value))).T
return result

for transfer_derivative():
first case:
def transfer_derivative(output):
s = output.reshape(-1, 1)
return np.diagflat(s) – np.dot(s, s.T)

second case:
def transfer_derivative(output):
jacobian_m = np.diag(output)
for i in range(len(jacobian_m)):
for j in range(len(jacobian_m)):
if i == j:
Your Start in Machine Learning
jacobian_m[i][j] = output[i] * (1 – output[i])
else:
jacobian_m[i][j] = -output[i] * output[j]
return jacobian_m

REPLY 
Jason Brownlee March 21, 2018 at 6:39 am #

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 81/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Here you go:

https://en.wikipedia.org/wiki/Softmax_function#Artificial_neural_networks

REPLY 
Jason Brownlee March 21, 2018 at 6:38 am #

Perhaps use Keras instead?

REPLY 
Suede March 29, 2018 at 6:40 am #

hey Jason, this is very helpful. I have run the code but i keep on getting this error, can you please
help me out? the error is: Your Start in Machine ×
NameError Traceback (most recent call last) Learning
in ()
You can master applied Machine Learning
186 str_column_to_float(dataset, i)
without math or fancy degrees.
187 # convert class column to integers Find out how in this free and practical course.
–> 188 str_columnto_int(dataset, len(dataset[0])-1)
189 # normalize input variables
Email Address
190 minmax = dataset_minmax(dataset)

NameError: name ‘str_columnto_int’ is not defined

START MY EMAIL COURSE

REPLY 
Jason Brownlee March 29, 2018 at 6:42 am #

The code was written for Python 2.7, confirm that you are using this version of Python?

REPLY 
Fahri Güreşçi April 15, 2018 at 7:03 am #

The csv file is not working. Edited csv file > bit.ly/2GYX2dF
you can use python 2 or 3
results:
python2 > Mean Accuracy: 95.238% Your Start in Machine Learning
python3 > Mean Accuracy: 93.333%

Why different?

REPLY 
Jason Brownlee April 16, 2018 at 6:00 am #

Wh is the CSV file not working? The link appears to work fine.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 82/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Good question re the difference, no idea. Perhaps small differences in the API? The code was written for
Py2, so it may require changes for py3.

Also, see this post on the stochastic nature of ml algorithms:

https://machinelearningmastery.com/randomness-in-machine-learning/

REPLY 
Fahad April 18, 2018 at 8:58 pm #

I have altered the code to work with MNIST (digit numbers) , the problem I have faced that
forward_propagate function returns [1 ,1 ,1 ,1 ,1 ,1 ,1 ,1 ,1 ,1 ] for each instance !

Any help

Your Start in Machine ×

Learning
REPLY 
Jason Brownlee April 19, 2018 at 6:30 am #
You can master applied Machine Learning
Well done. without math or fancy degrees.
Find out how in this free and practical course.
The model will require tuning for the problem.

Email Address

REPLY 
Fahad April 19, 2018 at 7:09 am # START MY EMAIL COURSE

Could you explain more in some details please.

REPLY 
Jason Brownlee April 19, 2018 at 2:45 pm #

What details exactly?

REPLY 
Fahad April 19, 2018 at 8:17 pm #

As I mentioned that forward_propagation function returns [1,1,1,1,1,1,1,1,1,1], what is the possible

alter to come over this problem Your Start in Machine Learning

REPLY 
Jason Brownlee April 20, 2018 at 5:48 am #

Perhaps tune the model to your specific problem. I have some suggestions here:
http://machinelearningmastery.com/improve-deep-learning-performance/

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 83/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Fahad April 23, 2018 at 5:34 am #

I altered the code to work with XOR problem and it was working perfectly. Then, I altered the code
to work with digit numbers MNIST, but as I told you there was a problem with the the forward_propagation
function that it returned all outputs to be [1,1,1,…] instead of a probabilities for each output.
I think it is not an optimization problem, there is something wrong with the forward_propagate function.

Here is the code after alteration [it is working but with a fixed error during all training epochs

from random import seed

from random import randrange
from random import random
from csv import reader
from math import exp
Your Start in Machine ×
global network
global gl_errors Learning
# Load a CSV file You can master applied Machine Learning
def load_csv(filename): without math or fancy degrees.
dataset = list() Find out how in this free and practical course.
with open(filename, ‘r’) as file:
csv_reader = reader(file)
Email Address
for row in csv_reader:
if not row:
continue START MY EMAIL COURSE
dataset.append(row)
return dataset

# Convert string column to float

def str_column_to_float(dataset, column):
for row in dataset:
row[column] = float(row[column].strip())

def str_column_to_intX(dataset, column):

for row in dataset:
row[column] = int(row[column].strip())

# Convert string column to integer

def str_column_to_int(dataset, column):
class_values = [row[column] for row in dataset] Your Start in Machine Learning
unique = set(class_values)
lookup = dict()
for i, value in enumerate(unique):
lookup[value] = i
for row in dataset:
row[column] = lookup[row[column]]
return lookup

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 84/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

# Find the min and max values for each column

def dataset_minmax(dataset):
minmax = list()
stats = [[min(column), max(column)] for column in zip(*dataset)]
return stats

# Rescale dataset columns to the range 0-1

def normalize_dataset(dataset):
for row in dataset:
for i in range(1,len(row)):
# row[i] = (row[i] – minmax[i][0]) / (minmax[i][1] – minmax[i][0])
if row[i]>10:
row[i]=1
else:
row[i]=0 Your Start in Machine ×
# Split a dataset into k folds
Learning
def cross_validation_split(dataset, n_folds): You can master applied Machine Learning
dataset_split = list() without math or fancy degrees.
dataset_copy = list(dataset) Find out how in this free and practical course.
fold_size = int(len(dataset) / n_folds)
for i in range(n_folds):
Email Address
fold = list()
while len(fold) epoch=%d, lrate=%.3f, error=%.3f’ % (epoch, l_rate, sum_error))
START MY EMAIL COURSE
# Calculate neuron activation for an input
def activate(weights, inputs):

activation = weights[-1]
for i in range(len(weights)-1):
activation += weights[i] * inputs[i]
return activation

# Transfer neuron activation

def transfer(activation):
return 1.0 / (1.0 + exp(-activation))

# Forward propagate input to a network output

def forward_propagate(network, row):
inputs = row[1:]
i=0 Your Start in Machine Learning

for layer in network:

new_inputs = []
i+=1
for neuron in layer:
activation = activate(neuron[‘weights’], inputs)
neuron[‘output’] = transfer(activation)
new_inputs.append(neuron[‘output’])
inputs = new_inputs
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 85/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

return inputs

# Calculate the derivative of an neuron output

def transfer_derivative(output):
return output * (1.0 – output)

# Backpropagate error and store in neurons

def backward_propagate_error(network, expected):
# err =0
for i in reversed(range(len(network))):
layer = network[i]
errors = list()
if i != len(network)-1:
for j in range(len(layer)):
error = 0.0 Your Start in Machine ×
for neuron in network[i + 1]:
error += (neuron[‘weights’][j] * neuron[‘delta’])
Learning
errors.append(error) You can master applied Machine Learning
else: without math or fancy degrees.
for j in range(len(layer)): Find out how in this free and practical course.
neuron = layer[j]
errors.append(expected[j] – neuron[‘output’])
Email Address
for j in range(len(layer)):
neuron = layer[j]
START MY EMAIL COURSE
neuron[‘delta’] = errors[j] * transfer_derivative(neuron[‘output’])

# Update network weights with error

def update_weights(network, row, l_rate):

for i in range(len(network)):
inputs = row[1:]
if i != 0:
inputs = [neuron[‘output’] for neuron in network[i – 1]]
for neuron in network[i]:
for j in range(len(inputs)):
neuron[‘weights’][j] += l_rate * neuron[‘delta’] * inputs[j]
neuron[‘weights’][-1] += l_rate * neuron[‘delta’]

# Initialize a network
def initialize_network(n_inputs, n_hidden, n_outputs): Your Start in Machine Learning
global network
network = list()
hidden_layer1 = [{‘weights’:[random() for i in range(n_inputs + 1)]} for i in range(n_hidden)]
network.append(hidden_layer1)

hidden_layer2 = [{‘weights’:[random() for i in range(n_hidden + 1)]} for i in range(100)]

network.append(hidden_layer2)

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 86/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

hidden_layer3 = [{‘weights’:[random() for i in range(100 + 1)]} for i in range(50)]

network.append(hidden_layer3)

output_layer = [{‘weights’:[random() for i in range(50 + 1)]} for i in range(n_outputs)]

network.append(output_layer)
return network

# Make a prediction with a network

def predict(network, row):
outputs = forward_propagate(network, row)
return outputs.index(max(outputs))

# Test Backprop on Seeds dataset

seed(1)
# load and prepare data
filename = ‘dataset/train2.csv’ Your Start in Machine ×
dataset = load_csv(filename) Learning
for i in range(1,len(dataset[0])):
You can master applied Machine Learning
str_column_to_float(dataset, i)
without math or fancy degrees.
# convert class column to integers Find out how in this free and practical course.
str_column_to_int(dataset, 0)
Email Address
normalize_dataset(dataset)

# evaluate algorithm
n_folds = 5 START MY EMAIL COURSE

l_rate = 0.5
n_epoch = 100
n_hidden = 500

scores = evaluate_algorithm(dataset, back_propagation, n_folds, l_rate, n_epoch, n_hidden)

print(‘Scores: %s’ % scores)
print(‘Mean Accuracy: %.3f%%’ % (sum(scores)/float(len(scores))))

REPLY 
Jason Brownlee April 23, 2018 at 6:26 am #

Sorry, I don’t have the capacity to debug the modified code for you.
Your Start in Machine Learning

REPLY 
Rahmad ars April 26, 2018 at 3:23 am #

Sir, can you help me?

this is my question..

https://stackoverflow.com/questions/50027886/need-help-for-check-my-backprop-ann-using-python

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 87/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Jason Brownlee April 26, 2018 at 6:37 am #

Perhaps you can summarize your question in a sentence or two?

REPLY 
Rahmad ars April 26, 2018 at 7:19 am #

your original code sir, only have 1 hidden layer with 2 neurons. then, I modify it, so the ANN have 3
hidden layers, each consist of (128, 64, 32). and i have my own dataset, so i change it (the dataset and input
neurons). when i run this code, everything looks fine but the error value is not changing…

here’s the screen: https://i.stack.imgur.com/NQbNd.png

Your Start in Machine
modified code: https://stackoverflow.com/questions/50027886/need-help-for-check-my-backprop-ann-using-
×
python Learning
thanks sir
You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.

REPLY 
Jason Brownlee April 26, 2018 at 2:59 pm #
Email Address
If you are struggling with the code, I would recommend not coding the algorithm from scratch.

Instead, I would recommend using a library like Keras. Here

STARTis MY
a worked example:
EMAIL COURSE
https://machinelearningmastery.com/tutorial-first-neural-network-python-keras/

REPLY 
Fahad April 28, 2018 at 9:20 pm #

I have the same problem of Rahmad

The same problem occurs when you change the original code from 5 neurons in the hidden layer to 31
neuron ( the error value does not change).

I know 31 hidden neuron is not a right number of neurons for seed data set. But I would like to know what is
the wrong when you increase the number of neurons.

Logically, it should be fine and the error value decreases. But when you change the number of neurons to 30
Your Start in Machine Learning
it is still working , when change it to 31 neurons it does not decrease !

I think if this problem is fixed, then the problem of Rahmad will be fixed too.

REPLY 
Jason Brownlee April 29, 2018 at 6:26 am #

Perhaps the model requires tuning to your specific problem (e.g. layers, nodes, activation
function, etc.)
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 88/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

It might be better to use a library like Keras for your project:

https://machinelearningmastery.com/start-here/#deeplearning

REPLY 
Rocha May 2, 2018 at 9:58 pm #

Hi dude, I’m stuck in this error, could you help me?

# Forward propagate input to a network output

def forward_propagate(network, row):
inputs = row
for layer in network:
new_inputs = []
for neuron in layer: Your Start in Machine ×
—>>> activation = activate(neuron[‘weights’], inputs)
Learning
neuron[‘output’] = transfer(activation)
new_inputs.append(neuron[‘output’]) You can master applied Machine Learning
inputs = new_inputs without math or fancy degrees.
return inputs Find out how in this free and practical course.

That line is giving me this: TypeError: list indices must be integers or slices, not str
Email Address
Should be the python version? I’m using python 3…

START MY EMAIL COURSE

REPLY 
Jason Brownlee May 3, 2018 at 6:33 am #

This code was written for Python 2.7 sorry.

REPLY 
Kamrun Nahar Nisha May 8, 2018 at 4:00 pm #

hello.please help me.

I want to use breast cancer dataset instead of seed dataset.

seed(1)
# load and prepare data
Your Start in Machine Learning
filename = ‘seeds_dataset.csv’
dataset = load_csv(filename)
for i in range(len(dataset[0])-1):
str_column_to_float(dataset, i)
# convert class column to integers
str_column_to_int(dataset, len(dataset[0])-1)
# normalize input variables
minmax = dataset_minmax(dataset)
normalize_dataset(dataset, minmax)

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 89/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

# evaluate algorithm
n_folds = 5
l_rate = 0.3
n_epoch = 500
n_hidden = 5
scores = evaluate_algorithm(dataset, back_propagation, n_folds, l_rate, n_epoch, n_hidden)
print(‘Scores: %s’ % scores)
print(‘Mean Accuracy: %.3f%%’ % (sum(scores)/float(len(scores))))

In this part of your code I also want to print the error and it will be like

epoch=0, lrate=0.500, error=6.350

>epoch=1, lrate=0.500, error=5.531
>epoch=2, lrate=0.500, error=5.221
>epoch=3, lrate=0.500, error=4.951 Your Start in Machine ×
>epoch=4, lrate=0.500, error=4.519
>epoch=5, lrate=0.500, error=4.173
Learning
>epoch=6, lrate=0.500, error=3.835
You can master applied Machine Learning
>epoch=7, lrate=0.500, error=3.506 without math or fancy degrees.
>epoch=8, lrate=0.500, error=3.192 Find out how in this free and practical course.
>epoch=9, lrate=0.500, error=2.898
>epoch=10, lrate=0.500, error=2.626
Email Address
>epoch=11, lrate=0.500, error=2.377
>epoch=12, lrate=0.500, error=2.153
>epoch=13, lrate=0.500, error=1.953 START MY EMAIL COURSE
>epoch=14, lrate=0.500, error=1.774
>epoch=15, lrate=0.500, error=1.614
>epoch=16, lrate=0.500, error=1.472
>epoch=17, lrate=0.500, error=1.346
>epoch=18, lrate=0.500, error=1.233
>epoch=19, lrate=0.500, error=1.132
[{‘output’: 0.029980305604426185, ‘weights’: [-1.4688375095432327, 1.850887325439514,
1.0858178629550297], ‘delta’: -0.0059546604162323625}, {‘output’: 0.9456229000211323, ‘weights’:
[0.37711098142462157, -0.0625909894552989, 0.2765123702642716], ‘delta’: 0.0026279652850863837}]
[{‘output’: 0.23648794202357587, ‘weights’: [2.515394649397849, -0.3391927502445985,
-0.9671565426390275], ‘delta’: -0.04270059278364587}, {‘output’: 0.7790535202438367, ‘weights’:
[-2.5584149848484263, 1.0036422106209202, 0.42383086467582715], ‘delta’: 0.03803132596437354}]

please tell me the code . Using breast cancer dataset not Your
wheatStart
seedindataset.
Machine I am not so good in coding
Learning
that’s why I need your help immediately.

REPLY 
Jason Brownlee May 9, 2018 at 6:09 am #

I’m eager to help, but I do not have the capacity to outline the changes or write the code for
you.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 90/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Akefar May 11, 2018 at 4:41 am #

hi Jason,
I tried your code in my data set ,shape of my data is (576,16) .the problem is

IndexError: list assignment index out of range

is there any need to change your code for (576,16) data shape .
Thanks

—————————————————————————
IndexError Traceback (most recent call last)
in ()
195 n_epoch = 500
196 n_hidden = 1 Your Start in Machine ×
–> 197 scores = evaluate_algorithm(dataset, back_propagation, n_folds, l_rate, n_epoch, n_hidden)
Learning
198 print(‘Scores: %s’ % scores)
199 print(‘Mean Accuracy: %.3f%%’ % (sum(scores)/float(len(scores))))
You can master applied Machine Learning
without math or fancy degrees.
in evaluate_algorithm(dataset, algorithm, n_folds, *args)
Find out how in this free and practical course.
79 test_set.append(row_copy)
80 row_copy[-1] = None
—> 81 predicted = algorithm(train_set, test_set, *args) Email Address
82 actual = [row[-1] for row in fold]
83 accuracy = accuracy_metric(actual, predicted)
START MY EMAIL COURSE
in back_propagation(train, test, l_rate, n_epoch, n_hidden)
171 n_outputs = len(set([row[-1] for row in train]))
172 network = initialize_network(n_inputs, n_hidden, n_outputs)
–> 173 train_network(network, train, l_rate, n_epoch, n_outputs)
174 predictions = list()
175 for row in test:

in train_network(network, train, l_rate, n_epoch, n_outputs)

148 outputs = forward_propagate(network, row)
149 expected = [0 for i in range(n_outputs)]
–> 150 expected[row[-1]] = 1
151 backward_propagate_error(network, expected)
152 update_weights(network, row, l_rate)
Your Start in Machine Learning
IndexError: list assignment index out of range

REPLY 
Jason Brownlee May 11, 2018 at 6:39 am #

You may need to change your data to match the model or the model to match the data.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 91/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Pradeep May 14, 2018 at 3:31 am # REPLY 

Hi Jason, I tried your code on the same sample dataset, i am getting the following type error in the
function activate. I am doing it in python3.6. hope to hear from you soon

Traceback (most recent call last):

File “neural_network.py”, line 94, in
train_network(network, dataset, 0.5, 20, n_outputs)
File “neural_network.py”, line 76, in train_network
outputs = forward_propagate(network, row)
File “neural_network.py”, line 31, in forward_propagate
activation = activate(neuron[‘weights’], inputs)
File “neural_network.py”, line 18, in activate
activation += weights[i] * inputs[i]
TypeError: can’t multiply sequence by non-int of type ‘float’Your Start in Machine ×
Learning
You can master applied Machine Learning
REPLY 
Jason Brownlee May 14, 2018 at 6:39 am # without math or fancy degrees.
Find out how in this free and practical course.
The code requires Python 2.7.

Email Address

REPLY 
Deepak D May 17, 2018 at 6:49 pm # START MY EMAIL COURSE

Hi Jason Brownlee,

I tried your code and experienced the some error applying the Backpropagation algorithm to the wheat
seeds dataset. I am using python 2.7.

Error type:

File “C:\Python27\programs\backpropagation.py”, line 186,

in str_column_to_float(dataset, i)

File “C:\Python27\programs\backpropagation.py”, line 22,

in str_column_to_float row[column] = float(row[column].strip())
ValueError: could not convert string to float:

Your Start in Machine Learning

REPLY 
Jason Brownlee May 18, 2018 at 6:22 am #

I am sorry to hear that, I have some suggestions here:

https://machinelearningmastery.com/faq/single-faq/why-does-the-code-in-the-tutorial-not-work-for-me

REPLY 
Dhanya Hegde May 19, 2018 at 1:53 am #
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 92/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Hey Jason! Great work. Really helpful. I didn’t understand one part of your code. On what basis
does predict function return the predicted value as 0 or 1, after taking the maximum of the two
output neuron values?

REPLY 
Jason Brownlee May 19, 2018 at 7:44 am #

The summation of the activation is passed through a sigmoid transfer function.

REPLY 
Dhanya Hegde May 21, 2018 at 3:37 am #

I didn’t understand this part of the code Your Start in Machine ×

outputs.index(max(outputs) Learning
Is one hot encoding used or binary classification?You can master applied Machine Learning
If so, how is the actual mapping done? without math or fancy degrees.
And when is the iteration process stopped? Find out how in this free and practical course.

Email Address

REPLY 
Jason Brownlee May 21, 2018 at 6:35 am #
START MY EMAIL COURSE
As stated in the text above the code, it returns an integer for the class with the largest
probability.

REPLY 
Ionut May 27, 2018 at 12:05 am #

Hi,

I’m a beginner in neural networks and I don’t understand the dataset from the section “4.2. Train Network”.
Can anyone explain me what x1, x2 and y means?

Your Start in Machine Learning REPLY 

Jason Brownlee May 27, 2018 at 6:45 am #

Input 1, input 2 and output.

Perhaps start here:

https://machinelearningmastery.com/start-here/#algorithms

REPLY 
Rishik Mani May 28, 2018 at 7:26 am #
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 93/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Hi Jason, thank you for the highly informative post. But could you please clarify me upon this petty
little issue.

In section 4.2 Train network, you considered n_inputs = len(dataset[0]) – 1. Why did you put a -1 here, while
the number of the inputs should exactly be of the length of the dataset.

REPLY 
Jason Brownlee May 28, 2018 at 2:32 pm #

To exclude the output variable from the number of inputs.

Samih Eisa June 2, 2018 at 9:13 pm # Your Start in Machine ×

REPLY 

Thank you, jason. Learning

You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.
REPLY 
Jason Brownlee June 3, 2018 at 6:22 am #

You’re welcome. Email Address

START MY EMAIL COURSE

REPLY 
Kie Woo Nam June 5, 2018 at 3:01 am #

1 # Update network weights with error

2 def update_weights(network, row, l_rate):
3 for i in range(len(network)):
4 inputs = row[:-1]
5 if i != 0:
6 inputs = [neuron['output'] for neuron in network[i - 1]]
7 for neuron in network[i]:
8 for j in range(len(inputs)):
9 neuron['weights'][j] += l_rate * neuron['delta'] * inputs[j]
10 neuron['weights'][-1] += l_rate * neuron['delta']

Hi,

I guess I’m likely mistaken, so please but when i != 0, isn’t the last line updating the last weight for the
Your Start in Machine Learning
second time?

So, shouldn’t it be “inputs = [neuron[‘output’] for neuron in network[i – 1]][:-1]” (add “[:-1]]” at the end)?

If I’m wrong, I’ll read the code again more carefully, so please let me know.

REPLY 
Jason Brownlee June 5, 2018 at 6:46 am #

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 94/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

No. There are more weights than inputs and the -1 index of the weights is the bias.

REPLY 
Kie Woo Nam June 5, 2018 at 7:06 pm #

Ah, right. Now I see it from “output_layer = [{‘weights’:[random() for i in range(n_hidden +

1)]} for i in range(n_outputs)]”.

Thank you for your quick reply.

REPLY 
Jason Brownlee June 6, 2018 at 6:39 am #
Your Start in Machine ×
No problem.
Learning
You can master applied Machine Learning
without math or fancy degrees. REPLY 
Thomas Specht July 10, 2018 at 4:46 am # Find out how in this free and practical course.

Hi Jason,
Email Address
Great tutorial to get into ML coding. I have one question:

What library would you recommend for projects and why? I want to use NN for regression problems.
START MY EMAIL COURSE

REPLY 
Jason Brownlee July 10, 2018 at 6:53 am #

I recommend Keras because it is computationally efficient, fast for development and fun:
https://machinelearningmastery.com/start-here/#deeplearning

REPLY 
Hugo B. August 13, 2018 at 1:29 am #

Hi Jason! Thank you for this tutorial.

I try to implement the batch learning, but I have some questions about it…
Your Start in Machine Learning
– Computing the errors:
Do I have to accumulate the errors (‘delta’) in backward_propagate_error() during one epoch and performing
an average according to the number of back-propagations performed?

– Updating the weights:

In train_network(), I call update_weights() for each epoch, but I don’t know which row(s) of train (dataset) I
have to used. Currently I use only one row: train[0].

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 95/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

elizabeth August 14, 2018 at 2:40 am # REPLY 

Hello Sir
Thank you so much for this article.
Can you please tell me how i can solve this error:

File “mlp8.py”, line 186, in

str_column_to_float(dataset, i)
File “mlp8.py”, line 22, in str_column_to_float
row[column] = float(row[column].strip())
ValueError: could not convert string to float:

thankyou

Your Start in Machine ×

Jason Brownlee August 14, 2018 at 6:22 am # Learning REPLY 

You can master applied Machine Learning

I have some suggestions here:
without math or fancy degrees.
https://machinelearningmastery.com/faq/single-faq/why-does-the-code-in-the-tutorial-not-work-for-me
Find out how in this free and practical course.

Email Address
REPLY 
STEPHEN OLADEJI August 28, 2018 at 10:35 pm #

Dear Prof, START MY EMAIL COURSE

Thanks for this tutorial,

Sir, I’ve gone through a lot of your project online and they are very superb. God bless you, sir.
Sir. my question is as followings.
1. I noticed that one WEKA 3.6, Artificial Immune System was removed because this was in version 1.8, Is it
because there is no research prospect in the algorithm?
2. I want to write python version for AIRS, CSCA, Genetic Algorithm can you help proofread it sir so see if
what I write is correct

REPLY 
Jason Brownlee August 29, 2018 at 8:11 am #

I have an implementation here that you can use:

Your Start in Machine Learning
http://wekaclassalgos.sourceforge.net/

REPLY 
Tamara September 11, 2018 at 5:36 am #

Thank you so much for this tutorial.

How to see the results of work a trained network in Python?

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 96/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Jason Brownlee September 11, 2018 at 6:34 am #

What do you mean exactly?

REPLY 
ritu September 23, 2018 at 9:49 am #

How should I modify to code to always run for one output neuron in the output layer?
eg. if the output class consist of only 2 output classes ‘1’ and ‘2’ , as per the above code 2 neurons will be
created within the output layer, but what if I wanted a neural network to just have one neuron in the output
layer.

Your Start in Machine ×

Learning
REPLY 
Jason Brownlee September 24, 2018 at 6:09 am #
You can master applied Machine Learning
If you are having trouble with this tutorial, I would encourage
without math or you todegrees.
fancy use Keras to develop
neural network models. Find out how in this free and practical course.

You can get started here:

https://machinelearningmastery.com/start-here/#pythonEmail Address

START MY EMAIL COURSE

REPLY 
Parva September 27, 2018 at 9:04 am #

Why is there just one output coming from layer 1 though it contains 2 neurons. Shouldn’t there be 2
outputs one from each neuron?

[{‘output’: 0.7105668883115941, ‘weights’: [0.13436424411240122, 0.8474337369372327,

0.763774618976614], ‘delta’: -0.0005348048046610517}]
[{‘output’: 0.6213859615555266, ‘weights’: [0.2550690257394217, 0.49543508709194095], ‘delta’:
-0.14619064683582808}, {‘output’: 0.6573693455986976, ‘weights’: [0.4494910647887381,
0.651592972722763], ‘delta’: 0.0771723774346327}]

Jason Brownlee September 27, 2018 at 2:47 pm Your

# Start in Machine Learning REPLY 

Which step exactly are you having trouble with?

REPLY 
Brian September 28, 2018 at 1:17 am #

Thank you for this example. It has helped me get past the block I had with the
mathematical based descriptions and differential calculus related to back propagation.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 97/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

As I have been focusing on the back propagation portion of this example I have come up with an
alternative version of the ‘backward_propagate_error’ function that I think is a much more succinct
and logical way to write this function.

Please find below

1 # Backpropagate error and store in neurons

2 def backward_propagate_error(network, expected):
3 for i in reversed(range(len(network))):
4 layer = network[i]
5 for j in range(len(layer)):
6 fromNeuron = layer[j]
7 error = 0.0
8 if i != len(network)-1:                           #This identifies all
9 for toNeuron in network[i + 1]:
10 error += (toNeuron['weights'][j] * toNeuron['delta'])
11 else:                                             #This is the last (o
12
13
error = expected[j] - fromNeuron['output']
fromNeuron['error'] = error
Your Start in Machine ×
14 fromNeuron['delta'] = error * transfer_derivative(fromNeuron['output']
Learning
You can master applied Machine Learning
without math or fancy degrees.
REPLY 
Jason Brownlee September 28, 2018 atFind
6:16out
am how
# in this free and practical course.

Cool!
Email Address
Sorry, I don’t have the capacity to review your code.

START MY EMAIL COURSE

REPLY 
Brian October 4, 2018 at 2:59 am #

Should not the formula for the error be

sum_error += sum([0.5*(expected[i]-outputs[i])**2 for i in range(len(expected))])
as apposed to
sum_error += sum([(expected[i]-outputs[i])**2 for i in range(len(expected))])
To correctly back propagate with the derivative?

REPLY 
Jason Brownlee October 4, 2018 at 6:20 am #

Why is that Brian? Your Start in Machine Learning

REPLY 
KS October 10, 2018 at 2:15 am #

Can you please tell me how to inplant tanh and ReLu?

I tried to find examples on the internet, but I couldn’t find good examples.

I understand that these 2 codes need to be changed:

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 98/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

# Transfer neuron activation

def transfer(activation):
return 1.0 / (1.0 + exp(-activation))

# Calculate the derivative of an neuron output

def transfer_derivative(output):
return output * (1.0 – output)

1. What is the code when using tanh?

2. What is the code when using ReLu?

REPLY 
Jason Brownlee October 10, 2018 at 6:15 am #
Your Start in Machine ×
Thanks for the suggestion, sorry, I don’t have the capacity to make these changes for you.
Learning
You can master applied Machine Learning
without math or fancy degrees. REPLY 
moe October 10, 2018 at 7:08 pm #
Find out how in this free and practical course.
Thank you so much, 1 – 5 helped me to built my own generic network. Now i have a MLP network
which i can easily adjust with a few parameter changes, wouldnt
Email have been so easy with your example.
Address

START MY EMAIL COURSE

REPLY 
Jason Brownlee October 11, 2018 at 7:51 am #

Well done!

REPLY 
Anwar October 11, 2018 at 3:35 am #

I am running the same code that u have provided on python 3.6 and getting these errors please
help me:-

—————————————————————————————————————————

IndexError Traceback (most recent call last)

in () Your Start in Machine Learning
16 n_epoch = 500
17 n_hidden = 5
—> 18 scores = evaluate_algorithm(dataset, back_propagation, n_folds, l_rate, n_epoch, n_hidden)
19 print(‘Scores: %s’ % scores)
20 print(‘Mean Accuracy: %.3f%%’ % (sum(scores)/float(len(scores))))

in evaluate_algorithm(dataset, algorithm, n_folds, *args)

12 test_set.append(row_copy)
13 row_copy[-1] = None

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 99/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

—> 14 predicted = algorithm(train_set, test_set, *args)

15 actual = [row[-1] for row in fold]
16 accuracy = accuracy_metric(actual, predicted)

in back_propagation(train, test, l_rate, n_epoch, n_hidden)

4 n_outputs = len(set([row[-1] for row in train]))
5 network = initialize_network(n_inputs, n_hidden, n_outputs)
—-> 6 train_network(network, train, l_rate, n_epoch, n_outputs)
7 predictions = list()
8 for row in test:

in train_network(network, train, l_rate, n_epoch, n_outputs)

5 outputs = forward_propagate(network, row)
6 expected = [0 for i in range(n_outputs)]
—-> 7 expected[row[-1]] = 1 Your Start in Machine ×
8 backward_propagate_error(network, expected)
9 update_weights(network, row, l_rate)
Learning
IndexError: list assignment index out of range
You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.

REPLY 
Jason Brownlee October 11, 2018 at 8:00 am #
Email Address
I have some suggestions here:
https://machinelearningmastery.com/faq/single-faq/why-does-the-code-in-the-tutorial-not-work-for-me
START MY EMAIL COURSE

REPLY 
KS October 12, 2018 at 11:33 am #

How to:
“Regression. Change the network so that there is only one neuron in the output layer and that a real value is
predicted.”

How to get 1 output layer?

REPLY 
Jason Brownlee October 13, 2018 at 6:06 am #
Your Start in Machine Learning
Coding algorithms from scratch is not for beginners.

I strongly encourage you to use Keras, for example:

https://machinelearningmastery.com/regression-tutorial-keras-deep-learning-library-python/

REPLY 
Jerry October 16, 2018 at 9:08 am #

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 100/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Ok, just following up. I still don’t get how the backprop is updating:

1         for row in train:

2             outputs = forward_propagate(network, row, alpha)
3             expected = [0 for i in range(n_outputs)]
4             expected[row[-1]] = 1
5             sum_error += sum([(expected[i]-outputs[i])**2 for i in range(len(expected))])
6             backward_propagate_error(network, expected, alpha)
7             update_weights(network, row, l_rate)

1. No return statement or ‘global network’ for backward_propagate_error or update_network to actually

incorporate the new weights.

My question, are you sure this uses back-propagation? How are the weights saved and updated for each
epoch?

Your Start in Machine ×

Jason Brownlee October 16, 2018 at 2:34 pm # Learning REPLY 

The weights are passed in by reference and You can master

modified applied Machine Learning
in place.
without math or fancy degrees.
Perhaps using Keras would be a better fit Jerry: Find out how in this free and practical course.
https://machinelearningmastery.com/tutorial-first-neural-network-python-keras/

Email Address

REPLY 
Jerry November 10, 2018 at 11:53 pm # START MY EMAIL COURSE

You learn something new everyday! I was always under the impression that values were
immutable when passing to a function in Python.

https://stackoverflow.com/questions/986006/how-do-i-pass-a-variable-by-reference

Thank you Jason! I’m really amazed on how active you are on this site. Just so you know, you and
your work are referenced often in my MSDS program.

One area that I think would also be beneficial is some work on hidden node activations and their
interpretations. The majority of our work is not so much on the output/accuracy of the NN, but more
of visualizing weights, activations, and determining the features that are causing nodes to activate.

Your Start in Machine Learning REPLY 

Jason Brownlee November 11, 2018 at 6:08 am #

Thanks, happy to help.

Why do you want to view/understand the dynamics of nodes in hidden layers?

REPLY 
ben October 31, 2018 at 9:47 pm #

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 101/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

i think this is not correct:

sum_error += sum([(expected[i]-outputs[i])**2 for i in range(len(expected))])

shouldn´t this be:

sum_error += sum([abs(0.5*(expected[i]-outputs[i])**2) for i in range(len(expected))])

see https://goo.gl/iqHJf6 page 233 (5.11).

REPLY 
Jason Brownlee November 1, 2018 at 6:11 am #

This implementation is based on the book “Neural Smithing”:

https://www.amazon.com/Neural-Smithing-Supervised-Feedforward-
Artificial/dp/0262527014/ref=as_li_ss_tl?ie=UTF8&linkCode=sl1&tag=inspiredalgor-
Your Start in Machine ×
20&linkId=e3db0b57249093a94ebb073983bc8b4d&language=en_US
Learning
You can master applied Machine Learning
without math or fancy degrees. REPLY 
Ayhan October 20, 2018 at 1:43 am #
Find out how in this free and practical course.
Hey Jason, first thnx for the efforts.

Maybe one thing: Email Address

wouldnt it be better (especially for beginners to that topic) to at least use
something like numpy which would at least let all the matrix START
calculation look COURSE
MY EMAIL
a bit more compact (and therefore possible to concentrate on the real topic .. which is
backprop i would have guessed)?

Ok the title sais ‘implement from scratch’ but I would say at some point getting
the point towards backprop is maybe more important than being able to say that
it was implemented from scratch (using nothing but plain python).

Greets

REPLY 
Jason Brownlee October 20, 2018 at 5:57 am #

Thanks for the feedback.

Yourfor
This example is not intended for writing operational code, Start
thatin Machine
I would LearningKeras:
recommend
https://machinelearningmastery.com/start-here/#deeplearning

This tutorial is to teach how to develop a net (training and evaluation) without any dependencies other
than the standard lib.

REPLY 
Vadim November 3, 2018 at 11:10 pm #

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 102/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Just Brilliant stuff

REPLY 
Jason Brownlee November 4, 2018 at 6:27 am #

Thanks.

REPLY 
Himanshu November 14, 2018 at 10:38 pm #

Hi Jason
how i can use forward and backward propagation in real time data. I mean data in which we have multiple
Your
field. Fields contains Numerical values , floating values , String Start
Values. in Machine ×
Learning
For an example if i want to use this technique for Titanic data how can i use it.

How can i decide how many hidden layers should be there.

You can master applied Machine Learning
How can i decide for learning rate.
without math or fancy degrees.
How can i decide for what should be the seed value.
Find out how in this free and practical course.
How can i decide for weights for such huge data.

Email Address

REPLY 
Jason Brownlee November 15, 2018 at 5:30 am # START MY EMAIL COURSE

You must use careful experimentation to get answers to each of these questions.

REPLY 
Himanshu November 15, 2018 at 6:12 pm #

Hi Jason
I have few more questions on this.

First: why you assigned “activation = weights[-1]” , why not other weights or any random value.
Second: why you are looping only for two values
for i in range(len(weights) – 1) though we have three values.
Third: Why you have considered only two outputs here though we have four
output for layer one =.9643898158763548 Your Start in Machine Learning
output for layer two =.9185258960543243
output for layer three = .8094918973879515
output for layer two = .7734292563511262

why you considered only last two values why not first two or any other combination.

Four: here i guess some problem by mistake you have update wrong value
expected[row[-1]] = 1

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 103/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

after this line you have updated expected as [1,0] from [0,0]
and why we in this i have other question why we are updating this value.

REPLY 
Jason Brownlee November 16, 2018 at 6:12 am #

Too many questions for one comment (I’m a simple human), one at a time and can you please
reference specific examples/lines of code otherwise I can’t help.

REPLY 
Karim November 18, 2018 at 8:14 am #

Your
Hi Jason — How can i add more hidden layers? Thankx Start in Machine ×
Learning
You can master applied Machine Learning
REPLY 
Jason Brownlee November 19, 2018 at 6:41 am #without math or fancy degrees.
Find out how in this free and practical course.
I recommend using Keras to develop your network:
https://machinelearningmastery.com/start-here/#deeplearning
Email Address

START MY EMAIL COURSE

REPLY 
Himanshu November 22, 2018 at 5:24 pm #

Hi Jason,

How to apply mathematical implementation of gradient descent and logistic regression,classification in real
time data.
For example if i want use this in survivors of Titanic data how to start with.

REPLY 
Jason Brownlee November 23, 2018 at 7:44 am #

Here’s an example:
https://machinelearningmastery.com/implement-logistic-regression-stochastic-gradient-descent-scratch-
Your Start in Machine Learning
python/

REPLY 
John Sald December 6, 2018 at 2:05 pm #

Hello, could you show me an example of using one of the extensions you mentioned, which can
give us a gain in performance?

Such as using matrix operations (in the weights) and vectors (inputs, intermediate signals and outputs)

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 104/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Jason Brownlee December 7, 2018 at 5:16 am #

Thanks for the suggestion.

If you’re looking to go deeper into neural nets, I recommend using a library like Keras. You can start
here:
https://machinelearningmastery.com/start-here/#deeplearning

REPLY 
Pipo December 17, 2018 at 7:25 am #

How could i change the loss to mse in this code?Your Start

I can’t wrap in Machine
my head around it. Thanks
×
Learning
You can master applied Machine Learning
REPLY 
Jason Brownlee December 17, 2018 at 2:13 pm #without math or fancy degrees.
Find out how in this free and practical course.
Calculate error between actual and predicted using a MSE function. That’s it.

Email Address

REPLY 
Brett August 24, 2019 at 4:07 am # START MY EMAIL COURSE

I’m confused about why you chose MSE for a classification problem. I was trying to use this
tutorial to discern the differences between a classification and function approximation
implementation, and the use of MSE for classification really threw me off. I know that it technically
works, but it’s probably good to mention that it’s not ideal. It would have been nice to get exposure
to taking the derivative of a different loss function, so that someone who is new to back-propagation
will start to grasp how different functions change the derivative, etc. Otherwise, the code is
understandable and could be modified slightly to make a good tutorial for function approximation.

REPLY 
Brett August 24, 2019 at 4:19 am #

Just to clarify what I’m saying, and toYour

answer Pipo’s
Start question,
in Machine this implementation is
Learning
already using MSE. The derivative of MSE with respect to the output is: (output – expected).
The fact that you’re multiplying this by the transfer derivative just means that you’re passing the
MSE back through the activation of the output node. So if you want the code to work for function
approximation, you simply don’t multiply by the transfer derivative. However, if you want
classification to work better, you could use the derivative of a different loss function with respect
to the output and predicted, and multiply that by your transfer derivative.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 105/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Jason Brownlee August 24, 2019 at 7:59 am #

Agreed!

REPLY 
Jason Brownlee August 24, 2019 at 7:58 am #

Yes, it was what we did in the 90s. Cross entropy would be preferred today, I agree.

I really need to do a series on coding neural nets from scratch to really dig into this. Thanks for
the kick!

Your Start in Machine ×

Brett August 24, 2019 at 9:26 am #
Learning
You can master applied Machine Learning
Thanks for the reply. I just found this post on finding the derivative of cross entropy,
without math or fancy degrees.
and it turns out that you can do a really nice simplification of the math to basically get
Find out how in this free and practical course.
(output – expected) or (expected – output) for your implementation, when combining the
cross entropy derivative and sigmoid derivative. So I’m pretty sure that if you simply stop
multiplying by the transfer derivative to get Email Address
your output error, you should see a big increase
in performance of the algorithm. Worth a try at least. Here is the link, with the conclusion I
mentioned at the very end:
START MY EMAIL COURSE
https://peterroelants.github.io/posts/cross-entropy-logistic/

Jason Brownlee August 25, 2019 at 6:30 am #

Thanks for sharing.

REPLY 
muhammad December 21, 2018 at 8:48 am #

hi, thanks for this code.

I’m trying to understand why are u adding on the update weights, shouldnt
Your Start be
in Machine Learning
wi←wi−η∂E/∂wi like this?

REPLY 
Sangeeth January 20, 2019 at 4:40 am #

Hi,

This website provides a good introduction for almost all topics in machine learning. Thanks for your work.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 106/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

In backpropagation, the error at each neuron is the product of

1. Change in error w.r.t y_out
2. Change in y_out w.r.t y
3. Change in y w.r.t weight.

Could you please tell how you just multiplied 1 and 2 in backward_propagate_error (from the last layer) and
then used 3 in update_weights (from the first layer). Should we not do all steps in
backward_propagate_error and then use it to update_weights?.

REPLY 
Jason Brownlee January 20, 2019 at 5:41 am #

I show exactly how in the above tutorial.

Your Start in Machine ×
Learning
REPLY 
sangeeth January 20, 2019 at 5:44 am # You can master applied Machine Learning
without math or fancy degrees.
Sorry, I just realized what I said is same as what
Find you did.
out how in this free and practical course.
About the error, should we not use 2*error (derivative of MSE)?
Email Address

Jason Brownlee January 21, 2019 at 5:26START

am #
MY EMAIL COURSE REPLY 

No, we calculate the derivative of the error against the non linear activation function,
not the derivative of the loss function itself.

sangeeth January 21, 2019 at 12:11 pm #

Ok. I got it. Thanks,

I think this is online learning using SGD. Would you have an implementation for offline
learning using mini Batch Gradient descent?

Your Start in Machine Learning

Jason Brownlee January 22, 2019 at 6:16 am #

Correct, you can modify the above example to use batch or mini-batch gradient
descent.

sangeeth January 22, 2019 at 1:14 am #

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 107/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Is the sum_error variable same as loss in model.fit output?..I get different loss
values when testing the same datasets on model.fit and your model. Could you tell
me why this is?

Jason Brownlee January 22, 2019 at 6:25 am #

Yes, see this post:

https://machinelearningmastery.com/randomness-in-machine-learning/

Brett August 24, 2019 at 4:25 am #

Your Start in Machine × REPLY 

The loss function used in this tutorialLearning

is: (1/2)(out – expect)^2. The derivative of which
with respect to the output is: (out – expect) * 1, or simply (out – expected). This is then multiplied
by the transfer derivative, because the error isYou canpassed
being master applied
backwardMachine Learning
via the chain rule. You
without
always have to take the derivative with respect to the math or fancyitself
loss function degrees.
first. I hope this clears
up any confusion. Find out how in this free and practical course.

Email Address

REPLY 
Gary January 22, 2019 at 8:54 am #
START MY EMAIL COURSE
Hi Jason.

In the “full” seeds example you call user defined function evaluate_algorithm(). However, the “heavy lifting”
inside it is performed by the function algorithm(). That function looks like it’s a part of some standard Python
library, but I can’t find it in any reference. Also you don’t comment at all at its use.

What’s the deal here?

Thank you,

Gary

REPLY 
Jason Brownlee January 22, 2019 at 11:43 am #
Your Start in Machine Learning
The “algorithm” is a reference to a function that is passed in as an argument.

REPLY 
Gary January 22, 2019 at 1:24 pm #

Yes, thank you, I already realized that.

Regards,

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 108/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Gary

REPLY 
sangeeth January 28, 2019 at 8:39 am #

Hi,

For online machine learning, should we perform epochs?. Should not we update the model based only on
the present time input and then predict the next time step. If we do epochs that means the model is getting
updated for the whole data set up to the present time. Am I correct?. Thanks

Jason Brownlee January 28, 2019 at 11:44 am # Your Start in Machine × 

REPLY

It depends on the problem and the data. Yes,Learning

if often makes sense to update the model with
new data and with a little of the old data.
You can master applied Machine Learning
without
Note, online gradient descent does not have to be used mathlearning.
for online or fancy degrees.
Find out how in this free and practical course.

Email Address
REPLY 
kmillen February 12, 2019 at 10:49 am #

Good afternoon Jason. I have thoroughly enjoyed this solution

START both COURSE
MY EMAIL in Python and my conversion to
C#. I guess for all the learning I’ve gleaned, one thing still seems to be a mystery to me. What exactly are
the five scores telling me? Do they annotate how well the data fits a curve for each fold?

REPLY 
Jason Brownlee February 12, 2019 at 1:59 pm #

Correct.

The mean of the scores is our estimate of the model’s performance when making predictions on unseen
data.

Your Start in Machine Learning REPLY 

kmillen February 26, 2019 at 8:10 am #

Thank you.

REPLY 
MathewP February 20, 2019 at 3:25 am #

I think there is a mistake update_weights function.

inputs = row[:-1]
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 109/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

If we have, say 2 inputs and 1 neuron in hidden layer then only one weight is going to be updated, which is
clearly wrong. Correct me if I am wrong. The code works fine just taking row as inputs.

REPLY 
Jason Brownlee February 20, 2019 at 8:11 am #

I don’t follow the possible issue, can you please elaborate?

REPLY 
Venkat February 22, 2019 at 4:53 am #

Hi Jason Brownlee, back propagation implementation really excellent ,because of without using any
Your
predefined library just use functions list, set, and dictionary. I needStart in Machine
a suggestion how to write a code for
×
Learning
implement activation function like a sigmoidal at hidden layer neurons and a tangent at output neurons.
could u help me.
You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.
REPLY 
Jason Brownlee February 22, 2019 at 6:25 am #
Email Address
Yes, you can use the above as a starting point.

More on tanh here:

START MY EMAIL COURSE
https://en.wikipedia.org/wiki/Hyperbolic_function

REPLY 
Venkat February 22, 2019 at 4:49 pm #

Hi Jason Brownlee, yes , iam not asking how to write code for implementation of tanh, sigmoidial .
My request is how to modified code in forward_propagate function to implement suppose x is a activation at
hidden layer and y is another activation function at output layer.

1 def forward_propagate(network, row):

2    inputs =row  
3    for layer in network:
4       new_inputs = []
5       for neuron in layer:
6         activation = activate(neuron['weights'], inputs)
Your Start in Machine Learning
7         neuron['output'] = transfer(activation)
8         new_inputs.append(neuron['output'])
9       inputs = new_inputs    
10    return inputs

in the above code u r calling transfer function for the hidden neurons and also output neurons . I request u to
give suggestion to call different activation functions for hidden and output neurons.

REPLY 
Jason Brownlee February 23, 2019 at 6:26 am #
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 110/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Change the code in the activation function itself.

Does that help?

REPLY 
Danh Nguyen February 24, 2019 at 12:09 pm #

Example is great! The totally clean CSV wheat seed dataset is here:
https://raw.githubusercontent.com/NguyenDa18/MachineLearning_HW3/master/wheat-seeds.csv

I tried Jason’s link

https://raw.githubusercontent.com/jbrownlee/Datasets/master/wheat-seeds.csv
and the UCI Repo link and the CSVs still had double commas and so we got the str_column_to_float error

Your
Anyway, posting this here so others won’t run into the same Start
problem in Machine
I did! Thanks ×
Learning
You can master applied Machine Learning
REPLY 
Jason Brownlee February 25, 2019 at 6:37 am # without math or fancy degrees.
Find out how in this free and practical course.
Thanks for sharing.

Email Address

REPLY 
vartika sharma February 27, 2019 at 3:49 am # START MY EMAIL COURSE

Hey Jason,
While I am ruuning the following code, I am getting this error
>>> scores=evaluate_algorithm(dataset,back_propagation,n_folds,l_rate,n_epoch,n_hidden)
Traceback (most recent call last):
File “”, line 1, in
File “”, line 13, in evaluate_algorithm
File “”, line 5, in back_propagation
File “”, line 6, in train_network
TypeError: list indices must be integers, not str

REPLY 
Jason Brownlee February 27, 2019 at 7:34 am # Your Start in Machine Learning

Perhaps try saving code to a file and run from the command line, here’s how:
https://machinelearningmastery.com/faq/single-faq/how-do-i-run-a-script-from-the-command-line

REPLY 
Andy March 5, 2019 at 12:51 pm #

Hello Jason,

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 111/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

I would like to ask, can you make the data split between training data and test data, instead of using k folds
variation, I would like to get some insight in this, thanks

REPLY 
Jason Brownlee March 5, 2019 at 2:22 pm #

Yes, I show how here:

https://machinelearningmastery.com/implement-resampling-methods-scratch-python/

REPLY 
Andy March 5, 2019 at 6:33 pm #

Hello Jason, it’s me again. Your Start in Machine ×

Learning
I would like to ask another question, how do you predict using this trained network ?
Lets say I have 100 data, and I split the training and test by 70:30 ratio. I’ve trained the network using 70
You can master applied Machine Learning
data, how do I predict the rest (30 data) ?
without math or fancy degrees.
Find out how in this free and practical course.

Jason Brownlee March 6, 2019 at 7:46 am # Email Address REPLY 

You can fit one final model on all data, then use it to make predictions (see the predict section).
START MY EMAIL COURSE
Remember, this is an example for learning only. If you want a model for your data in practice, I
recommend using a robust neural network library like Keras:
https://machinelearningmastery.com/start-here/#deeplearning

REPLY 
Dini M March 11, 2019 at 12:21 am #

# Convert string column to integer

def str_column_to_int(dataset, column):
class_values = [row[column] for row in dataset]
unique = set(class_values)
lookup = dict()
for i, value in enumerate(unique): Your Start in Machine Learning
lookup[value] = i
for row in dataset:
row[column] = lookup[row[column]]
return lookup

I got the error “class_values = [row[column] for row in dataset]”

IndexError: list index out of range

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 112/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Jason Brownlee March 11, 2019 at 6:52 am #

Sorry to hear that, I have some suggestions here:

https://machinelearningmastery.com/faq/single-faq/why-does-the-code-in-the-tutorial-not-work-for-me

REPLY 
Dini M March 11, 2019 at 12:28 am #

I trying your code example and seeds_dataset.csv

giuseppe March 15, 2019 at 7:08 am # Your Start in Machine ×

REPLY 

Hi thanks for the code is amazing, Learning

I’ve included it in a class for a project, I’ve modified it so I can decide how many neurons put in each layer
You can master applied Machine Learning
because, but I have a question: during the train process using for example 4 neurons in the first layer and 3
without math or fancy degrees.
in the second one I get nice result, around 85% / 92 %. At this point I save all the weights of the neurons and
Find out how in this free and practical course.
I call another function that just load the weight that I’ve saved(skipping in this way the traning process) and
using all the dataset(the same I’ve used for train the network) as test set it gives me a really bad score, most
of the time is around 30%. I’m using the “IrisDataTrain” andEmail
what Address
I’ve noticed is that the networks fails to
recognise one of the 3 classes. Do you have any suggestion about what could be? Thanks

START MY EMAIL COURSE

REPLY 
Jason Brownlee March 15, 2019 at 2:25 pm #

Perhaps the weights are not being loaded correctly?

REPLY 
giuseppe March 28, 2019 at 8:11 pm #

Hi sorry for the late replay, actually the problem of save and loading the nn is not that
important so maybe i’ll try to solve it later. At the moment the problem is that i should reach the 99.
% on the “IrisDataTrain” set. What i’ve noticed is that the accurancy can change a lot repeating the
same training process with the same configuration. In order to get a better result I’ve tryied to repeat
Your Start in Machine Learning
the same traning process different times with the same configuration, I’ve choosen the configuration
that give me the best result in mean and variance. Now in order to improve the accurancy I’ve
modified the code so that I can connect easily the output of a nn with the input of another one so
that I can create a cascade of neural networks connected in different ways. At this point i’m stucked
at 96% in mean. To improve the accurancy I’ve implemented the relu activation function (but i’m not
sure it’s correctly implemented) and adam optimizer (but it doesn’t work at all).
I’ll link the code on pastbin (I don’t know if there is any better way to do that) in particular what I’ve
done is just insert everything in a class and modified:
1. the initialization function so that I can chose the number of neurons in each layer

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 113/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

2. the back_propagation_error function trying to add the relu and adam optimizer
3. the update weights function trying to implement adam optimizer (it doesnt’ work at all)
In the code I’m going to share I’ve just removed many parts just for a readability reason, after
cleaning it I will send it to you if you want. Sorry for the long message and thanks for you help

https://pastebin.com/RxxuVaCD

REPLY 
Jason Brownlee March 29, 2019 at 8:31 am #

Nice work!

It might be time to graduate to Keras where everything is implemented for you and you can just
use it directly and focus on tuning the model.
Your Start in Machine ×
Learning
MLnovice March 21, 2019 at 10:53 pm # You can master applied Machine Learning REPLY 

without math or fancy degrees.

Hello sir, Find out how in this free and practical course.
I am playing with your code and I am trying to figure out this error:

line 185, in Email Address

str_column_to_float(dataset, i)
line 21, in str_column_to_float START MY EMAIL COURSE
row[column] = float(row[column].strip())
ValueError: could not convert string to float:

Do you have any insides of why this is happening?

REPLY 
Jason Brownlee March 22, 2019 at 8:28 am #

Sorry to hear that, I have some suggestions here:

https://machinelearningmastery.com/faq/single-faq/why-does-the-code-in-the-tutorial-not-work-for-me

Matty March 25, 2019 at 9:55 am # Your Start in Machine Learning REPLY 

Thank you for the post Jason.

Reading this post, it seems to me that I can split the process of back propagation in large networks into
multiple steps. Am I right?

I have a large network that my current GPU runs out of memory when I try to train it. I was wondering if I can
split my network into two sub-networks, and first calculate the updates for the deeper part(that has the
ground truth outputs) and obtain the error that should be passed to the other sub-network. Then, use the
provided error to calculate the updates for the second sub-network as well. Do you think it’s possible? Do
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 114/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

you have any suggestion (or source that can be helpful) for implementing this back propagation with existing
tensorflow or pytorch builtin functions?

Thanks.

REPLY 
Jason Brownlee March 25, 2019 at 2:17 pm #

Yes, by node or by layer.

It might be possible, but also a massive pain.

It might be cheaper (in time/money) to rent an AWS EC2 instance with more GPU RAM for a few hours?

Your Start in Machine ×

Learning REPLY 
Matty March 26, 2019 at 1:03 pm #
You can master applied Machine Learning
Thanks, Jason. I think I found an easy way to split the back-propagation in tensorflow.
without math or fancy degrees.
Find out
We can define two separate optimizations with different how in this
trainable free and
variable practical
lists. course.
Something similar to:

self.optim_last_layers = tf.train.AdamOptimizer(lr, beta1=beta1) \

.minimize(loss, var_list=vars_of_last_layers) Email Address

self.optim_first_layers = tf.train.AdamOptimizer(lr, beta1=beta1) \

.minimize(loss, var_list=vars_of_first_layers) START MY EMAIL COURSE

And in each iteration, we can call the optimizations separately.

I did a small sanity check with a two-layer network, and it seems both the two-step optimization and
the one-step optimization with all the trainable parameters results in the same updates.

REPLY 
Jason Brownlee March 26, 2019 at 2:20 pm #

Glad to hear it.

Novia Puspitasari March 28, 2019 at 12:17 am # Your Start in Machine Learning REPLY 

thankyou so much jason, for your post about it

i have some problem in “‘float’ object has no attribute ‘append'”

Traceback (most recent call last):

File “backprop.py”, line 200, in
scores = evaluate_algorithm(dataset, back_propagation, n_folds, l_rate, n_epoch, n_hidden)
File “backprop.py”, line 80, in evaluate_algorithm
predicted = algorithm(train_set, test_set, *args)

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 115/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

File “backprop.py”, line 172, in back_propagation

train_network(network, train, l_rate, n_epoch, n_outputs)
File “backprop.py”, line 150, in train_network
backward_propagate_error(network, expected)
File “backprop.py”, line 123, in backward_propagate_error
error.append(error)
AttributeError: ‘float’ object has no attribute ‘append’

do you have a solve for that ? thankyou

REPLY 
Kevin March 29, 2019 at 1:21 am #

Your Start
Hi there Jason, I would like to ask if I wanted to generate in Machine
random weights and bias with range -1 to ×
1, how to do it ? Since I already import random or from uniform import random, and it said AttributeError:
Learning
‘builtin_function_or_method’. Thank you very much !
You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.
REPLY 
Jason Brownlee March 29, 2019 at 8:38 am #
Email Address
Good question, this post shows how:
https://machinelearningmastery.com/how-to-generate-random-numbers-in-python/
START MY EMAIL COURSE
You then scale them to any range you want:

result = min + value * range

REPLY 
Kevin March 29, 2019 at 4:20 pm #

Thanks for your reply ! Awesome guide, I really grateful for it. Once again, thanks a lot
Jason.

REPLY 
Jason Brownlee March 30, 2019 at 6:22 am #
Your Start in Machine Learning
You’re welcome.

REPLY 
wancong zhang March 30, 2019 at 12:45 pm #

Hi Jason, very cool tutorial.

I notice that your neural network only has 3 layers.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 116/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

If I change your “initialize network” method to initialize multiple hidden layers with arbitrary width, will your
program still work? In other words does your algorithm generalize to deeper networks?

Thanks.

REPLY 
Jason Brownlee March 31, 2019 at 9:26 am #

No idea – it is for educational purposes only, try it and see.

REPLY 
manoj April 19, 2019 at 7:38 am #
Your Start in Machine ×
Hi Jason!
Learning
Its really a helpful post, that you very much.
You can master applied Machine Learning
I wanted to see the plots of training error and testing error. (like how they finally converged by epochs by
without math or fancy degrees.
epochs). What would be the easiest way to plot those training and testing graphs
Find out how in this free and practical course.
regards
Manoj Goli Email Address

START MY EMAIL COURSE

REPLY 
Jason Brownlee April 19, 2019 at 3:03 pm #

I’d recommend using a library like Keras which provides the history directly and ready to plot:
https://machinelearningmastery.com/display-deep-learning-model-training-history-in-keras/

REPLY 
Danial April 20, 2019 at 5:46 pm #

Hi jason.
My question is how I can see my CNN code is using BP framework?

Your Start in Machine Learning

REPLY 
Jason Brownlee April 21, 2019 at 8:20 am #

You can save the model weights to a file.

Is that what you mean?

REPLY 
Danial April 21, 2019 at 1:27 pm #

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 117/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Yes. How I can see model weights? How cnn use BP framework if it is not shown in code?
for i in range(len(test)):
# Forecast the data
test_X, test_y = test[i, 0:-1], test[i, -1]
X_ = test_X.reshape(1, 28, 28, 1)
predict = model.predict(X_, batch_size=1)
predict = predict[0, 0]

# Replacing value in test scaled with the predicted value.

test_prediction = [predict] + test_prediction
if len(test_prediction) > sequence_length+1:
test_prediction = test_prediction[:-1]
if i+1 sequence_length+1:
test[i+1] = test_prediction
else: Your Start in Machine ×
Learning
test[i+1] = np.concatenate((test_prediction, test[i+1, i+1:]), axis=0)

# Inverse transform
You can master applied Machine Learning
predict = inverse_transform(scaler, test_X, predict)
without math or fancy degrees.
# Inverse the features Find out how in this free and practical course.
predict = inverse_features(data_set, predict, len(test)+1-i) – maxVal
if predict < 0:
Email Address
predict = 0
# Round the value
predict = np.round(predict, 2) START MY EMAIL COURSE
# store forecast
expected = data_set[len(train) + i + 1]
predict_data.append(predict )
real_data.append(expected )
if expected != 0:
prediction.append(predict)
real.append(expected)

REPLY 
Jason Brownlee April 22, 2019 at 6:15 am #

You can get the model weights from a Keras model by calling the get_weights()
function on a give layer. Your Start in Machine Learning

Danial April 22, 2019 at 6:38 am #

Is it right that above code is using BP framework.? It’s part of CNN code that I sent

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 118/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Danial April 22, 2019 at 11:23 am #

How keras uses BP framework.? If you have link kindly share it.

Jason Brownlee April 22, 2019 at 2:26 pm #

You can get started with Keras here:

https://machinelearningmastery.com/start-here/#deeplearning

Idris Shareef April 24, 2019 at 4:16 pm # Your Start in Machine ×

REPLY 

Learning
Hello Jason , You’re the best teacher.
You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.
REPLY 
Jason Brownlee April 25, 2019 at 8:08 am #
Email Address
Thanks.

START MY EMAIL COURSE

REPLY 
Abarni April 30, 2019 at 12:39 pm #

Nice Post !

Here is another very nice tutorial with step by step Mathematical explanation and full coding.

http://www.adeveloperdiary.com/data-science/machine-learning/understand-and-implement-the-
backpropagation-algorithm-from-scratch-in-python/

REPLY 
Jason Brownlee April 30, 2019 at 2:27 pm #

Thanks for sharing.

Your Start in Machine Learning

REPLY 
Zahra May 6, 2019 at 4:28 pm #

Hello, I’m so confuse.

I try to run this code in command prompt. But, I use my dataset (not Wheat Seeds dataset).

And why this happened? What’s wrong? What should I do? What should I change?
Please, help me!
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 119/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Traceback (most recent call last):

File “journal.py”, line 197, in
scores = evaluate_algorithm(dataset, back_propagation, n_folds, l_rate, n_epoch, n_hidden)
File “journal.py”, line 81, in evaluate_algorithm
predicted = algorithm(train_set, test_set, *args)
File “journal.py”, line 173, in back_propagation
train_network(network, train, l_rate, n_epoch, n_outputs)
File “journal.py”, line 150, in train_network
expected[row[-1]] = 1
IndexError: list assignment index out of range

Jason Brownlee May 7, 2019 at 6:12 am # Your Start in Machine ×

REPLY 

I cannot know, I can give you some advice onLearning

debugging your problem here:
https://machinelearningmastery.com/faq/single-faq/can-you-read-review-or-debug-my-code
You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.
REPLY 
Zahra May 9, 2019 at 1:11 am #
Email Address
Hello, How to import my dataset using that codes?

For example (in this codes), how to use my dataset (use excel file)MY
START in this codes..
EMAIL COURSEHow to import my dataset
in this codes? Can you teach me more detail, please..

# Test training backprop algorithm

seed(1)
dataset = [[2.7810836,2.550537003,0],
[1.465489372,2.362125076,0],
[3.396561688,4.400293529,0],
[1.38807019,1.850220317,0],
[3.06407232,3.005305973,0],
[7.627531214,2.759262235,1],
[5.332441248,2.088626775,1],
[6.922596716,1.77106367,1],
[8.675418651,-0.242068655,1],
[7.673756466,3.508563011,1]] Your Start in Machine Learning
n_inputs = len(dataset[0]) – 1
n_outputs = len(set([row[-1] for row in dataset]))
network = initialize_network(n_inputs, 2, n_outputs)
train_network(network, dataset, 0.5, 20, n_outputs)
for layer in network:
print(layer)

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 120/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Jason Brownlee May 9, 2019 at 6:46 am #

Perhaps you should start with Keras, it is much easier for beginners:
https://machinelearningmastery.com/tutorial-first-neural-network-python-keras/

REPLY 
Aziz Ahmad July 22, 2019 at 4:42 am #

Sir! I really apprecitae your work.

Your
Jason Brownlee July 22, 2019 at 8:28 am #
Start in Machine ×
REPLY 

Learning
Thanks.
You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.
REPLY 
Nirmala May 10, 2019 at 5:37 am #
Email Address
I got an error called->

IndexError: list assignment index out of range.

START MY EMAIL COURSE
but I m using python 3 itself.

REPLY 
Jason Brownlee May 10, 2019 at 8:20 am #

Sorry to hear that, I have some suggestions here:

https://machinelearningmastery.com/faq/single-faq/why-does-the-code-in-the-tutorial-not-work-for-me

REPLY 
Ido Berenbaum May 11, 2019 at 9:39 pm #

Hi Jason,
Your Start in Machine Learning
thanks for the great tutorial, I learned a lot from it.
There is one thing I didn’t really understand though,
when you update the weights you add to the weight the calculated change that is needed.
but’from what I read in other sites like wikipedia, the change to the weight needs to be multiplied by -1 and
then added to ensure it
changes the weight in the opposite direction of the gradient and so getting it closer to the local minimum.
like muhammad said in December 21, 2018:
“hi, thanks for this code.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 121/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

I’m trying to understand why are u adding on the update weights, shouldnt be
wi←wi−η∂E/∂wi like this?”

and I tried to change line 141 to: neuron[‘weights’][j] -= l_rate * neuron[‘delta’] * inputs[j]
basically doing -= and not += but it just made the sum error of the network to increase after each epoch.

so, I will be grateful if you could explain to me why are you adding and not subtracting.

thanks

REPLY 
Jason Brownlee May 12, 2019 at 6:43 am #

There are many ways to implement the algorithm description.

Your Start in Machine ×
This implementation is based on the description in “neural smithing”:
https://amzn.to/2pW6hjI Learning
You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course. REPLY 
cocoa July 16, 2019 at 8:37 am #

Jason seem to use mean square error as loss.Email

partial derivative of loss should be (output-
Address
expected). In his “backward” function, he did (expected-output). That’s why he came up with “+=” not “-
=”
START MY EMAIL COURSE

REPLY 
Jason Brownlee July 16, 2019 at 2:19 pm #

Nice explanation.

REPLY 
Nirmala May 16, 2019 at 4:23 pm #

In training code and testing code I want to link the onther dataset .txt file but it will not work.please
can u send a code for that..

Your Start in Machine Learning

REPLY 
Jason Brownlee May 17, 2019 at 5:49 am #

Sorry, I don’t have the capacity to develop custom code for you.

REPLY 
Arthur May 22, 2019 at 4:03 pm #

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 122/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Hi Jason, first of all thank you very much for this post, I’m learning ML at the moment, and writing a
neural network with backpropagation in C# to help the process.

When using the wheat seeds dataset, and the same network layout as you suggest, I get very similar results
to yours in terms of accuracy.

When I scale the network however to have 2 (or more) hidden layers of 5 neurons, I sometimes get
exploding gradients (and NaN output results because of it). Is this something you’d expect given the
dataset? ie, doesn’t this dataset allow for much more than 1 hidden layer for some reason. (Some context: I
normalize the data just like you do, and I use the same dataset as you do, the reason I test with more than 1
layer, is just out of curiosity, and to see whether the accuracy improved – it doesnt)

I’m trying to understand why it happens with this particular data, or whether my implementation fails
somehow. Note that I do get good results most of the time, but with a certain weight initialization the
exploding gradients can happen.
Your Start in Machine ×
Learning
REPLY 
Jason Brownlee May 23, 2019 at 5:54 am # You can master applied Machine Learning
without math or fancy degrees.
Perhaps try scaling the data prior to modeling, see
Find this
out howpost:
in this free and practical course.
https://machinelearningmastery.com/how-to-improve-neural-network-stability-and-modeling-
performance-with-data-scaling/
Email Address

START MY EMAIL COURSE

REPLY 
Zahra Nabila May 27, 2019 at 4:00 pm #

Hello, I have problem. Why output must be integers, not float (decimal)? Specially in Train.. How to
change output data type to float?

TypeError Traceback (most recent call last)

in train_network(network, train, l_rate, n_epoch, n_outputs)

93 outputs = forward_propagate(network, row)
94 expected = [0 for i in range(n_outputs)]
—> 95 expected[row[-1]] = 1
96 sum_error += sum([(expected[i]-outputs[i])**2 for i in range(len(expected))])
97 backward_propagate_error(network, expected)

TypeError: list indices must be integers or slices, not numpy.float64

Your Start in Machine Learning

REPLY 
Jason Brownlee May 28, 2019 at 8:08 am #

The example is classification, you can change it to regression if you wish.

Perhaps this tutorial will be more helpful:

https://machinelearningmastery.com/regression-tutorial-keras-deep-learning-library-python/

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 123/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Zahra May 28, 2019 at 1:09 pm #

I want to ask, how to display execution time (running rime) in Train code?

REPLY 
Jason Brownlee May 28, 2019 at 2:44 pm #

Sorry, I don’t have an example of calculating clock time for code examples.

Jeny June 3, 2019 at 10:06 am # Your Start in Machine ×

REPLY 

”’ Learning
Calculate the output of a recurrent neural network with tanh activation
You can master applied Machine Learning
and a linear layer on top
without math or fancy degrees.
Params: Find out how in this free and practical course.
x: input matrix [n_timesteps * n_samples * 2]
w: non-recurrent weights
Email Address
r: recurrent weights
b: biases
wo: output-layer weights START MY EMAIL COURSE
bo: output-layer biases
Returns:
h: matrix of activations (n_timesteps, n_samples, n_hiddens)
o: final predictions
”’

def forward_path(x, w, r, b, wo, bo):

h = np.empty([t_max, n, w.shape[0]], dtype=np.float32) # storage for the hidden activations
for t in range(t_max):
z = np.dot(x[t], w.T) + b
if t > 0:
z += np.dot(h[t-1], r.T)
h[t] = np.tanh(z)
o = np.dot(h[-1], wo.T) + bo Your Start in Machine Learning
return h, o

def backward_path(x, h, w, b, r, wo, bo, o, y):

n, t_max, _ = x.shape
dw = np.zeros_like(w)
db = np.zeros_like(b)
dr = np.zeros_like(r)
dwo = 0
dbo = 0

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 124/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

return dw, dr, db, dwo, dbo

def loss(w, r, b, wo, bo, x, y):

_, o = forward_path(x, w, r, b, wo, bo)
err = 0.5*np.sum(np.square(o-y))
return err

Can you help me implement the backpropagation? Please.

REPLY 
Jason Brownlee June 3, 2019 at 2:34 pm #

Sorry, I don’t have the capacity to debug your code, perhaps try posting to stackoverflow?

Your Start in Machine ×

Learning
REPLY 
Zahra Nabila Izdihar June 10, 2019 at 11:53 pm #
You can master applied Machine Learning
Hello… without math or fancy degrees.
How to display “predicted” value in your code? Find out how in this free and practical course.

Because I need to display the predicted or forecast value..

Email Address
Thank you

START MY EMAIL COURSE

REPLY 
Jason Brownlee June 11, 2019 at 7:54 am #

The forward_propagate() function makes a prediction.

REPLY 
Zahra Nabila Izdihar June 13, 2019 at 2:36 am #

I got it. So, “output” (in ForwardPropagation code)= prediction result?

But, I don’t understand How to determine the weights in forward propagation? What is the formula?

Thank you
Your Start in Machine Learning

REPLY 
Jason Brownlee June 13, 2019 at 6:21 am #

What do you mean exactly?

The weights are learned during training.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 125/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Zahra June 16, 2019 at 4:51 am #

Did you mean that “output” (in forward propagatation) is predicted result?

# test forward propagation

network = [[{‘weights’: [0.13436424411240122, 0.8474337369372327,
0.763774618976614]}],
[{‘weights’: [0.2550690257394217, 0.49543508709194095]}, {‘weights’:
[0.4494910647887381, 0.651592972722763]}]]
row = [1, 0, None]
output = forward_propagate(network, row)
print(output)

Your Start in Machine ×

Jason Brownlee June 16, 2019 atLearning
7:16 am #

Yes. You can master applied Machine Learning

without math or fancy degrees.
Perhaps this is too advanced. I recommend starting with Keras instead:
Find out how in this free and practical course.
https://machinelearningmastery.com/tutorial-first-neural-network-python-keras/

Email Address

REPLY 
Leo July 2, 2019 at 11:25 pm #
START MY EMAIL COURSE

It is crazy that nobody complain the readability of your codes. Thanks anyway

REPLY 
Jason Brownlee July 3, 2019 at 8:34 am #

Sorry that you think that the code is not readable. I thought it was very readable.

What do you believe the problem is exactly?

REPLY 
Femi July 14, 2019 at 12:17 am #
Your Start in Machine Learning
Good day Dr. I am new to machine learning but have interest in it. My question is this, can i use the
Python 2.7 in miniconda to implement your samples?

REPLY 
Jason Brownlee July 14, 2019 at 8:15 am #

Yes.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 126/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Ravi July 19, 2019 at 9:52 pm #

Hi Dr. Jason

I have developed and trained a neural network (3 layers: 1 input, 1 hidden and 1 output) for following
situation

(The code was written step by step, as i do not want to use a tool without understanding the computations)

Data set (40 input patterns):

Input: 40 samples 5 elements

Output: 40 samples 1 element
number of neurons (Input = 5; hidden = 5; output = 1)

Using the delta rule with backpropagation algorithm, i wasYour Start error
able to achieve in Machine
= 9.39E-06 for 1000 ×
iterations
Learning
My final “input to hidden layer” weight matrix size is 200 x 5 (as i have 40 samples x 5 input neurons and 5
You can master applied Machine Learning
hidden neurons)
without math or fancy degrees.
“hidden to output layer” weight matrix size is 200 x 1 (as i Find
haveout
40how
samples
in thisxfree
5 hidden neurons
and practical and 1 output
course.
neuron)

Email
Now my question is for a given test sample having 5 elements Address
(input is 1 sample 5 elements),

i need to run feed-forward computation to get a single element output.

START MY EMAIL COURSE
For running this which weights i need to select in “input to hidden layer” and “hidden to output layer” from the
trained set??

I have 200 x 5 and 200 x 1 weight matrices; but i require only 5 x 5 and 5 x 1 weight matrices for testing.

Kindly let me know if i am missing something here?

Thanks in advance

Ravi

REPLY 
Chrissie Li July 25, 2019 at 9:06 pm #

hi, how can i download the code? copy the tip’s script?
Your Start in Machine Learning

REPLY 
Jason Brownlee July 26, 2019 at 8:22 am #

I show you how right here:

https://machinelearningmastery.com/faq/single-faq/how-do-i-copy-code-from-a-tutorial

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 127/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Femi July 25, 2019 at 9:39 pm # REPLY 

Prof Sir, pls i have been finding it difficult to implement your samples when i can not prepare
environment that accept all the command. Pls help

REPLY 
Jason Brownlee July 26, 2019 at 8:23 am #

This tutorial will show you how to setup your environment:

https://machinelearningmastery.com/setup-python-environment-machine-learning-deep-learning-
anaconda/

Your Start in Machine ×

Femi July 31, 2019 at 12:39 am # Learning REPLY 

Sir, I guessed you use scipy environment. am i right?

You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.

REPLY 
Jason Brownlee July 31, 2019 at 6:53 am # Email Address

For this tutorial, a simple Python environment is enough.

START MY EMAIL COURSE

REPLY 
Majed August 4, 2019 at 8:10 am #

I wrote a neural network that consists of three layers as follows:[ 4 input neurones – 5 hidden
neurones – 3 output neurones]. first, I standerdized the data using the z-score. The accuracy of my model
exceeded 67. Note: I didn’t use the regularisation terms yet.
here is my implementation of both feedforward and back prop ..

1     while iteration < number_of_iterations:

2         z2 = np.dot(x, w1)
3         a2 = sigmoid(z2)
4         z3 = np.dot(a2, w2)
5         a3 = sigmoid(z3)  # this represents the output of the network
6         error = loss_function(a3, label_matrix)
7         delta_3 = np.multiply(-(label_matrix-a3), sigmoid_prime(z3))
Your Start in Machine Learning
8         dJW2 = np.dot(a2.transpose(), delta_3)
9         delta_2 = np.dot(delta_3, w2.transpose()) * sigmoid_prime(z2)
10         dJW1 = x.transpose() @ delta_2
11         w2 = w2 - (learning_rate * dJW2)
12         w1 = w1 - (learning_rate * dJW1)
13         iteration = iteration + 1  # update the counter :')
14     return w1, w2

Thanks …

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 128/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Majed August 4, 2019 at 8:11 am # REPLY 

The data set that I worked on is the Iris data set

REPLY 
Jason Brownlee August 5, 2019 at 6:43 am #

Well done!

REPLY 
Ekundayo August 6, 2019 at 9:40 pm #

Your
Ca I still get an help this one time? Pls sir, my project is toStart
use 14 in Machine
features extractions for plant
×
leave classification. I need to recognize one leaf out of 36Learning
leaves all with 14 features. Sir can I use your code
to achieve this?
Thanks You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.

REPLY 
Jason Brownlee August 7, 2019 at 7:52 am # Email Address

I would recommend this tutorial, e.g. transfer learning:

https://machinelearningmastery.com/how-to-use-transfer-learning-when-developing-convolutional-
START MY EMAIL COURSE
neural-network-models/

REPLY 
Mohammed August 14, 2019 at 12:20 pm #

Hi Dr. Jason

Thank you for this post, it is really very helpful.

I have one question about backpropagation in unsupervised model, e.g. extract features.
Is it possible to apply this code for it, and only replaces loss function of unsupervised model by the loss
function of supervised?

Regards
Your Start in Machine Learning

REPLY 
Jason Brownlee August 14, 2019 at 2:10 pm #

Backpropagation is for supervised learning, not unsupervised learning.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 129/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

REPLY 
Mohammed August 15, 2019 at 5:08 pm #

Oh! many thanks,So, can help me what is the way for learning parameters in unsupervised
approach.
if i need to extract the features from data as low dimension nested of data with large dimension.

REPLY 
Jason Brownlee August 16, 2019 at 7:47 am #

There are specialized techniques for unsupervised learning neural nets, perhaps start
with the SOM:
http://cleveralgorithms.com/nature-inspired/neural/som.html
Your Start in Machine ×
Learning
You can master applied Machine Learning REPLY 
Mohammed August 15, 2019 at 5:10 pm #
without math or fancy degrees.
FindFiltering!
such as Unsupervised feature learning with Sparse out how in this free and practical course.

Email Address

REPLY 
Jason Brownlee August 16, 2019 at 7:48 am #
START MY EMAIL COURSE
Sorry, I don’t have a tutorial on that topic, perhaps in the future.

REPLY 
Mohammed August 16, 2019 at 11:20 am #

Thank you so much Dr. Jason.

REPLY 
Jason Brownlee August 16, 2019 at 2:10 pm #

You’re welcome.
Your Start in Machine Learning

REPLY 
Cherinet Mores August 20, 2019 at 4:44 pm #

Jason Brownlee
Thank you for your continues help
Here I have one questions,
In case, if i want to solve the regression problem (Meaning, if I have 3 real value outputs from the input
parameters) which part of the code should be modified and How?

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 130/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Thank you very much

REPLY 
Steven Pauly August 20, 2019 at 9:59 pm #

Hi Cherinet, I’ve changed the n_outputs to 1 and the function train_network, I’ve changed the
below. I’ve increased the n_epoch to a lot higher, because else it will give you the average. Be sure to
normalize your input & output values, though.

1 def train_network(network, train, l_rate, n_epoch, n_outputs):

2 for epoch in range(n_epoch):
3 sum_error = 0
4 for row in train:
5 outputs = forward_propagate(network, row)
6
7
expected = [row[-1] for i in range(n_outputs)]
Your Start in Machine
sum_error = sum([(expected[i]-outputs[i])**2 for i in range(len(expected))
×
8 backward_propagate_error(network, expected)
9 update_weights(network, row, l_rate)
Learning
10 print('>epoch=%d, lrate=%.3f, error=%.3f' % (epoch, l_rate, sum_error))
You can master applied Machine Learning
without math or fancy degrees.
Find out how in this free and practical course.
REPLY 
Jason Brownlee August 21, 2019 at 6:42 am #
Email Address
Thanks for sharing.

START MY EMAIL COURSE

REPLY 
Cherinet Mores August 21, 2019 at 8:06 am #

Dear Steven Pauly thank you very much for your help.

REPLY 
Jason Brownlee August 21, 2019 at 6:36 am #

Change the output to be a linear activation and the loss function to mse.

REPLY 
Steven Pauly August 20, 2019 at 9:56 pm # Your Start in Machine Learning

Well Done, Jason! Great stuff!!!

REPLY 
Jason Brownlee August 21, 2019 at 6:41 am #

Thanks, I’m glad it helped.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 131/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Leave a Reply

Name (required)

Your Start in Machine ×

Email (will not be published) (required)
Learning
You can master applied Machine Learning
Website
without math or fancy degrees.
Find out how in this free and practical course.

SUBMIT COMMENT
Email Address

Welcome to Machine Learning Mastery! START MY EMAIL COURSE

Hi, I'm Jason Brownlee, PhD.

I write tutorials to help developers (like you) get results with machine learning.

Read More

More Tutorials?

Start Here
(over 800 tutorials!)

Your Start in Machine Learning

Code Algorithms From Scratch

No libraries, just simple Python code.

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 132/133
8/27/2019 How to Code a Neural Network with Backpropagation In Python

Click to Get Started Now!

Your Start in Machine ×

Learning
You can master applied Machine Learning
without math or fancy degrees.
Join Newsletter
Find out how in this free and practical course.
Subscribe Now
(join over 150,000 Email
readers!)
Address

START MY EMAIL COURSE

© 2019 Machine Learning Mastery Pty. Ltd. All Rights Reserved.

Address: PO Box 206, Vermont Victoria 3133, Australia. | ACN: 626 223 336.
RSS | Twitter | Facebook | LinkedIn

Privacy | Disclaimer | Terms | Contact | Sitemap | Search

Your Start in Machine Learning

https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ 133/133