Python TensorFlow Tutorial - Build A Neural Network - Adventures in Machine Learning
Python TensorFlow Tutorial - Build A Neural Network - Adventures in Machine Learning
ABOUT CONTACT
Python TensorFlow
POPULAR TUTORIALS
CATEGORIES
CNTK
gensim
Googles
TensorFlow Keras
hasbeen a
Neural networks
hottopic in
deep learning NLP
recently.The
Optimisation
open source
software, TensorFlow
designed to
Word2Vec
Machine Learning Course allow e cient
computation
Learn to make
of data ow
Predictive Models
graphs, is
http://adventuresinmachinelearning.com/python-tensorflow-tutorial/ 1/18
9/28/2017 Python TensorFlow Tutorial - Build a Neural Network - Adventures in Machine Learning
d = b + c FIND US ON FACEBOOK
e = c + 2
a = d e
Adventures in Machine L
Now we can represent these operations graphically as: Liked 2.1K likes
http://adventuresinmachinelearning.com/python-tensorflow-tutorial/ 2/18
9/28/2017 Python TensorFlow Tutorial - Build a Neural Network - Adventures in Machine Learning
This may seem like a silly example but notice a powerful idea in
expressing the equation this way: two of the computations (
d = b + c and e = c + 2 ) can be performed in parallel. By
splitting up these calculations across CPUs or GPUs, this can give
us signi cant gains in computational times.Thesegains are a must
for big data applications and deep learning especially for
complicated neural network architectures such as Convolutional
Neural Networks (CNNs) and Recurrent Neural Networks (RNNs).
The idea behind TensorFlow is to ability to create these
computational graphs in code and allow signi cant performance
improvements via parallel operations and other e ciency gains.
http://adventuresinmachinelearning.com/python-tensorflow-tutorial/ 3/18
9/28/2017 Python TensorFlow Tutorial - Build a Neural Network - Adventures in Machine Learning
The animated data ows between di erent nodes in the graph are
tensors which are multi-dimensional data arrays. For instance, the
input data tensor may be 5000 x 64 x 1, which represents a 64
node input layer with 5000 training samples. After the input layer
there is a hidden layer with recti ed linear units as the activation
function. There is a nal output layer (called a logit layer in the
above graph) which uses cross entropy as a cost/loss function. At
each point we see therelevant tensors owing to the Gradients
block which nally ow to the Stochastic Gradient Descent
optimiser which performs the back-propagation and gradient
descent.
http://adventuresinmachinelearning.com/python-tensorflow-tutorial/ 4/18
9/28/2017 Python TensorFlow Tutorial - Build a Neural Network - Adventures in Machine Learning
import tensorflow as tf
Its important to note that, as the Python code runs through these
commands, the variables havent actually been declared as they
would have been if you just had a standard Python declaration (i.e.
b= 2.0). Instead, all the constants, variables, operationsand the
computational graph are only created when the initialisation
commands are run.
The next step is to setup an object to initialise the variables and the
graph structure:
http://adventuresinmachinelearning.com/python-tensorflow-tutorial/ 5/18
9/28/2017 Python TensorFlow Tutorial - Build a Neural Network - Adventures in Machine Learning
Ok, so now we are all set to go. To run the operations between the
variables, we need to start a TensorFlow session tf.Session. The
TensorFlow session is an object where all operations are run.
Using the with Python syntax, we can run the graph with the
following code:
The rst command within the with block is the initialisation, which
is run with the, well, run command. Next we want to gure out
what the variable a should be. All we have to do is run the
operation which calculates a i.e. a = tf.multiply(d, e, name=a). Note
that a is an operation, not a variable and therefore it can be run.
We do just that with the sess.run(a) command and assign the
output to a_out, the value of which we then print out.
Variable a is [[ 3.]
[ 6.]
[ 9.]
[ 12.]
[ 15.]
[ 18.]
[ 21.]
[ 24.]
http://adventuresinmachinelearning.com/python-tensorflow-tutorial/ 7/18
9/28/2017 Python TensorFlow Tutorial - Build a Neural Network - Adventures in Machine Learning
[ 27.]
[ 30.]]
http://adventuresinmachinelearning.com/python-tensorflow-tutorial/ 8/18
9/28/2017 Python TensorFlow Tutorial - Build a Neural Network - Adventures in Machine Learning
Now we need to setup the weight and bias variables for the three
layer neural network. There are always L-1 number of weights/bias
tensors, where L is the number of layers. Soin this case, we need
975
to setup two tensors for each:
Shares
Ok, so lets unpack the above code a little. First, we declare some
variables for W1 and b1, the weights and bias for the connections
between the input and hidden layer. This neural network will have
300 nodes in the hidden layer, so the size of the weight tensor W1
is [784, 300]. We initialise the values of the weights using a random
normal distribution with a mean of zero and a standard deviation
of 0.03. TensorFlow has a replicated version of the numpy
random normal function, which allows you to create a matrix of a
given size populated with random samples drawn from a given
distribution. Likewise, we create W2 and b2 variables to connect
the hidden layer to the output layer of the neural network.
http://adventuresinmachinelearning.com/python-tensorflow-tutorial/ 9/18
9/28/2017 Python TensorFlow Tutorial - Build a Neural Network - Adventures in Machine Learning
(l+1) (l+1)
h = f (z )
m n
1 (i) (i)
(i) (i)
J = y log(y j _ ) + (1 y )log(1 y j _ )
j j
m
i=1 j=1
Where y j is the ith training label for output node j, y j _(i) is the
(i)
Remember that y and y_clipped in the above calculation are (mx 10)
tensors therefore we need to perform the rst sum over the
second axis. This is speci ed using the axis=1 argument, where 1
actually refers to the second axis when we have a zero-based
indices system like Python.
# add an optimiser
optimiser =
tf.train.GradientDescentOptimizer(learning_rate=learni
ng_rate).minimize(cross_entropy)
more details on gradient descent see here and here) and the
backpropagationfor you. How easy is that? TensorFlow has a
library of popular neural networktraining optimisers, see here.
http://adventuresinmachinelearning.com/python-tensorflow-tutorial/ 12/18
9/28/2017 Python TensorFlow Tutorial - Build a Neural Network - Adventures in Machine Learning
batch_x, batch_y =
mnist.train.next_batch(batch_size=batch_size)
_, c = sess.run([optimiser,
cross_entropy],
feed_dict={x: batch_x, y:
batch_y})
avg_cost += c / total_batch
print("Epoch:", (epoch + 1), "cost =", "
{:.3f}".format(avg_cost))
print(sess.run(accuracy, feed_dict={x:
mnist.test.images, y: mnist.test.labels}))
Stepping through the lines above, the rst couple relate to setting
up the with statement and running the initialisation operation. The
third line relates to our mini-batch training scheme that we are
going to run for this neural network. If you want to know about
mini-batch gradient descent, check out this post. In the third line,
we are calculating the number of batches to run through in each
training epoch. After that, we loop through each training epoch
and initialise an avg_cost variable to keep track of the average cross
entropy cost for each epoch. The next line is where we extract a
randomised batch of samples, batch_xand batch_y, from the MNIST
training dataset. The TensorFlow provided MNIST dataset has a
handy utility function, next_batch,that makes it easy to extract
batches of data for training.
Finally, we print out our progress in the average cost, and after the
training is complete, we run the accuracy operation to print out the
accuracy of our trained network on the test set. Running this
program produces the following output:
Training complete!
0.9787
Have fun!
http://adventuresinmachinelearning.com/python-tensorflow-tutorial/ 14/18
9/28/2017 Python TensorFlow Tutorial - Build a Neural Network - Adventures in Machine Learning
PREVIOUS NEXT
Stochastic Gradient Convolutional
Descent Mini- Neural Networks
batch and more Tutorial in
TensorFlow
12 COMMENTS
tomas
APRIL 14, 2017 AT 8:28 AM
hi, Iike the idea of explaining using the simple equation, great idea.
I didnt get the tensor/array output could you past all the code. Also
the code for the tensorboard visualization would be nice (I know
you are planning to go into that in more detail in another tutorial,
but would be great to take a look at now.
REPLY
Andy
APRIL 15, 2017 AT 5:51 AM
REPLY
http://adventuresinmachinelearning.com/python-tensorflow-tutorial/ 15/18
9/28/2017 Python TensorFlow Tutorial - Build a Neural Network - Adventures in Machine Learning
Bablo l
APRIL 19, 2017 AT 3:19 AM
REPLY
Lwebzem
MAY 5, 2017 AT 1:16 AM
I used the code from this post and it worked instantly. This is a
great article and great code so I added the link to the collection of
neural networks with python.
REPLY
dtdzung
JULY 17, 2017 AT 1:02 PM
REPLY
xxx
JULY 21, 2017 AT 10:21 AM
REPLY
Lucian
JULY 28, 2017 AT 11:10 AM
Hi
Thank you
REPLY
Andy
JULY 28, 2017 AT 7:44 PM
REPLY
John McDonald
AUGUST 10, 2017 AT 10:38 PM
Shouldnt
a=d*e in the 1st paragraph breakdown? Not a=d*c
REPLY
Andy
AUGUST 11, 2017 AT 7:22 PM
Hi John, yes it should thanks for picking this up. Ive xed it
REPLY
John McDonald
AUGUST 12, 2017 AT 12:58 PM
http://adventuresinmachinelearning.com/python-tensorflow-tutorial/ 17/18
9/28/2017 Python TensorFlow Tutorial - Build a Neural Network - Adventures in Machine Learning
REPLY
Pasindu Tennage
AUGUST 17, 2017 AT 5:17 PM
Thank you very much for posting this. Very informative. Keep up
the good work
REPLY
Leave a Reply
Your email address will not be published.
Comment
Name*
Email*
Website
POST COMMENT
http://adventuresinmachinelearning.com/python-tensorflow-tutorial/ 18/18