Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

Introduction to deep learning

•Download as PPTX, PDF•
•
D
doppenhe

This document provides an introduction to deep learning. It begins by discussing modeling human intelligence with machines and the history of neural networks. It then covers concepts like supervised learning, loss functions, and gradient descent. Deep learning frameworks like Theano, Caffe, Keras, and Torch are also introduced. The document provides examples of deep learning applications and discusses challenges for the future of the field like understanding videos and text. Code snippets demonstrate basic network architecture.

Related slideshows

Introduction to Deep Learning
Introduction to Deep LearningIntroduction to Deep Learning
Introduction to Deep Learning
 
Handwritten Recognition using Deep Learning with R
Handwritten Recognition using Deep Learning with RHandwritten Recognition using Deep Learning with R
Handwritten Recognition using Deep Learning with R
 
Deep learning
Deep learningDeep learning
Deep learning
 
1 of 54
INTRODUCTIONTO DEEP LEARNING Zeynep Su Kurultay
Outline ■ Modeling humans in machines ■ Introduction to neural nets ■ What makes an algorithm intelligent? ■ Learning – Supervised learning ■ Deep learning – Neural nets in detail ■ Framework discussion & sample code ■ Future
Modeling humans in machines
Modeling humans in machines But why?
Neural networks ■ The mammal brain is organized in a deep architecture (Serre, Kreiman, Kouh, Cadieu, Knoblich, & Poggio, 2007) (E.g. visual system has 5 to 10 levels) ■ Very popular at the beginning of 1990s but fell out of favor after it was found that they were not performing well ■ Why is it gaining power again now: Deep architectures might be able to represent some functions otherwise not efficiently representable. Breakthrough in 2006/2007 with Hinton, Bengio papers
Examples around us
Examples around us Date: November 2014
Examples around us
Examples around us
Examples around us
Examples around us Image: NasenSpray/Imgur
Examples around us Image: http://www.telegraph.co.uk/technology/google/11730050/deep-dream-best-images.html?frame=3370388
Examples around us Image: drkaugumon/Imgur
What makes an algorithm intelligent? Image courtesy ofToptal.com
What makes an algorithm intelligent?
Learning ■ Supervised machine learning:The program is “trained” on a pre-defined set of “training examples”, which then facilitate its ability to reach an accurate conclusion when given new data. ■ Semi-supervised machine learning:The program infers the unknown labels through “label propagation”, utilizing similarities between different examples and inferring non-existent labels from existent ones ■ Unsupervised machine learning:The program is given a bunch of data and must find patterns and relationships therein. – e.g. clustering via nearest neighbor algorithm
Supervised Learning ■ Binary classification: Does this person have that disease? ■ Regression:What is the market value of this house? ■ Multiclass classification: Digit recognition, Face recognition
Supervised Learning ■ Goal: Given a number of features, try to make sense out of it! ■ Example: Employee satisfaction rates – depends on ?  So, given these features in a dataset, try to predict the rate
Supervised Learning
Supervised Learning
Supervised Learning
Supervised Learning
Supervised Learning ■ But how do we adjust ourselves? How do we know at each step we are getting better? ■ Measurement of wrongness: Loss functions
Loss functions
Gradient descent How do we know how to “roll down the hill”? The gradient (the derivatives of the loss function over all of the individual weights of features -i.e. parameters-) tells us “which way is down”.
What exactly is deep learning? ■ “a network would need more than one hidden layer to be a deep network, networks with one or two hidden layers are traditional neural networks…….” ■ “in my experience, a network can be considered deep when there is at least one hidden layer.Although the term deep learning can be fuzzy, …” ■ “in my own thinking, deep is not related to the number of layers, but it talks about how hard the feature to be discovered is…….” ■ - a discussion from StackExchange
Deep learning ■ What is the difference? Remember the quote fromYann LeCun from before? It goes on: ■ “A pattern recognition system is like a black box with a camera at one end, a green light and a red light on top, and a whole bunch of knobs on the front…. Now, imagine a box with 500 million knobs, 1,000 light bulbs, and 10 million images to train it with. That’s what a typical Deep Learning system is.”
Introduction to deep learning
Aim: Learning features ■ Deep learning excels in tasks where the basic unit, a single pixel, a single frequency, or a single word has very little meaning in and of itself, but the combination of such units has a useful meaning. It can learn these useful combinations of values without any human intervention.
Aim: Learning features (convolutional neural networks)
Neural networks ■ An input, output, and one or more hidden layers of units/neurons/perceptrons ■ Each connection between two neurons has a weight w. Best weights can again be found with gradient descent. Image courtesy of http://ljs.academicdirect.org/A15/053_070.htm
Neural networks ■ Example: Input vector: [7, 1, 2]  Into the input units ■ Forward propagation ■ Activation function Image courtesy of http://ljs.academicdirect.org/A15/053_070.htm
Neural networks ■ Why deep? ■ Number of parameterized transformations a signal encounters as it propagates from the input layer to the output layer, where a parameterized transformation is a processing unit that has trainable parameters, such as weights. Image courtesy of http://ljs.academicdirect.org/A15/053_070.htm
Aim: Learning features ■ The goal of deep learning methods is to learn higher levels of feature from lower level features.
Other important concepts ■ Overfitting – there is such a thing as learning too much –or too specific-! ■ Regularization – a technique that prevents overfitting
Overfitting ■ Overfitting – there is such a thing as learning too much –or too specific-! ■ Regularization – a technique that prevents overfitting
Overfitting U.S. Census Population overTime
Different frameworks ■ Pylearn2, Lasagne,Caffe,Torch,Theano, Blocks, Plate,Crino,Theanet, DL4J, Keras, …
Different frameworks ■ Theano: – A mathematical expression compiler, designed with machine learning in mind. – Lets you define an objective and automatically produces the code that computes the gradient of the objective. – Good for experimenting with different loss functions – Slightly lower layer of abstraction vs more possibilities
Different frameworks ■ Caffe: – Developed by UC Berkeley – Widely used machine-vision library that ported Matlab’s implementation of fast convolutional nets to C and C++ – Not intended for other deep-learning applications such as text, sound or time series data CORRECTION: There are new implementations of RNNs and LSTMs in Caffe, so it is not only for images any more! – Very fast: over 60M images per day with a single NVIDIA K40 GPU
Different frameworks ■ Torch: – Written in Lua (a scripting language developed in Brazil in the early 1990s) – A highly customized version of it is used by large tech companies such as Google and Facebook
Different frameworks ■ Keras: – Minimalist, highly modular neural network library in the spirit ofTorch – Written in Python – UsesTheano under the hood for optimized tensor manipulation on GPU andCPU – It was developed with a focus on enabling fast experimentation – 60K images took 30 hours on Amazon g2.2xlarge 
Comparing Keras andTheano MNIST digits dataset - serves as a benchmark to compare results with as new articles come out. Multilayer Perceptron - Basic feedforward neural network
Demo Code snippets – inside the gradient descent Output =Wx+b
Demo Code snippets – inside the hidden layer
Demo Code snippets – inside the hidden layer
Demo Code snippets – inside the hidden layer
Demo Code snippets – inside the network
Demo ■ https://algorithmia.com/demo/handwriting
Future of deep learning ■ Deep learning has a lot of hype right now, and it is apparent that it is very useful for specific tasks. ■ What frontiers and challenges do you think are the most exciting for researchers in the field of neural networks in the next ten years? ■ I cannot see ten years into the future. For me, the wall of fog starts at about 5 years. ... I think that the most exciting areas over the next five years will be really understanding videos and text. I will be disappointed if in five years time we do not have something that can watch aYouTube video and tell a story about what happened. I have had a lot of disappointments. – From Geoffrey Hinton’s AMA on Reddit
Now &The future Facebook Deep Learning, March 26, 2015 Image courtesy ofVenturebeat.com
Join us! ■ Open positions: https://angel.co/algorithmia/jobs/ – Algorithm Developer [this is me!] – Backend Developer – Product Manager – Technical Evangelist
Further resources ■ Introductory: ■ Andrew Ng’s Machine Learning course on Coursera ■ Geoffrey Hinton’s Neural Networks course on Coursera ■ Advanced: ■ Stanford’s Convolutional Neural Networks forVisual Recognition http://cs231n.github.io/ ■ Who is afraid of non-convex loss functions? ByYann LeCun http://videolectures.net/eml07_lecun_wia/ ■ What is wrong with Deep Learning? ByYann Lecun http://techtalks.tv/talks/whats-wrong-with-deep-learning/61639/ ■ For those who like papers, recent advances: ■ PlayingAtari with Deep Reinforcement Learning - http://www.cs.toronto.edu/~vmnih/docs/dqn.pdf ■ Unsupervised Face Detection - http://cs.stanford.edu/~quocle/faces_full.pdf
■ Content: ■ Toptal.com, Deeplearning.net ■ http://www.computerworld.com/article/2918161/emerging-technology/the-ai-ecosystem.html ■ Introduction to Machine Learning CMU-10701 - Deep Learning slides ■ Images: ■ http://www.spyemporium.com/images/products/st-sc1720.jpg ■ http://stats.stackexchange.com/questions/128616/whats-a-real-world-example-of-overfitting ■ http://www.homedepot.com/catalog/productImages/1000/c4/c4c34d2e-56ce-4c11-94c0-67aa19b769fa_1000.jpg ■ http://www.bulborama.com/images/products/1933.jpg ■ https://xkcd.com/1122/, https://xkcd.com/1425/ ■ www.deeplearning.net

More Related Content

Introduction to deep learning

  • 2. Outline ■ Modeling humans in machines ■ Introduction to neural nets ■ What makes an algorithm intelligent? ■ Learning – Supervised learning ■ Deep learning – Neural nets in detail ■ Framework discussion & sample code ■ Future
  • 4. Modeling humans in machines But why?
  • 5. Neural networks ■ The mammal brain is organized in a deep architecture (Serre, Kreiman, Kouh, Cadieu, Knoblich, & Poggio, 2007) (E.g. visual system has 5 to 10 levels) ■ Very popular at the beginning of 1990s but fell out of favor after it was found that they were not performing well ■ Why is it gaining power again now: Deep architectures might be able to represent some functions otherwise not efficiently representable. Breakthrough in 2006/2007 with Hinton, Bengio papers
  • 7. Examples around us Date: November 2014
  • 11. Examples around us Image: NasenSpray/Imgur
  • 12. Examples around us Image: http://www.telegraph.co.uk/technology/google/11730050/deep-dream-best-images.html?frame=3370388
  • 13. Examples around us Image: drkaugumon/Imgur
  • 14. What makes an algorithm intelligent? Image courtesy ofToptal.com
  • 15. What makes an algorithm intelligent?
  • 16. Learning ■ Supervised machine learning:The program is “trained” on a pre-defined set of “training examples”, which then facilitate its ability to reach an accurate conclusion when given new data. ■ Semi-supervised machine learning:The program infers the unknown labels through “label propagation”, utilizing similarities between different examples and inferring non-existent labels from existent ones ■ Unsupervised machine learning:The program is given a bunch of data and must find patterns and relationships therein. – e.g. clustering via nearest neighbor algorithm
  • 17. Supervised Learning ■ Binary classification: Does this person have that disease? ■ Regression:What is the market value of this house? ■ Multiclass classification: Digit recognition, Face recognition
  • 18. Supervised Learning ■ Goal: Given a number of features, try to make sense out of it! ■ Example: Employee satisfaction rates – depends on ?  So, given these features in a dataset, try to predict the rate
  • 23. Supervised Learning ■ But how do we adjust ourselves? How do we know at each step we are getting better? ■ Measurement of wrongness: Loss functions
  • 25. Gradient descent How do we know how to “roll down the hill”? The gradient (the derivatives of the loss function over all of the individual weights of features -i.e. parameters-) tells us “which way is down”.
  • 26. What exactly is deep learning? ■ “a network would need more than one hidden layer to be a deep network, networks with one or two hidden layers are traditional neural networks…….” ■ “in my experience, a network can be considered deep when there is at least one hidden layer.Although the term deep learning can be fuzzy, …” ■ “in my own thinking, deep is not related to the number of layers, but it talks about how hard the feature to be discovered is…….” ■ - a discussion from StackExchange
  • 27. Deep learning ■ What is the difference? Remember the quote fromYann LeCun from before? It goes on: ■ “A pattern recognition system is like a black box with a camera at one end, a green light and a red light on top, and a whole bunch of knobs on the front…. Now, imagine a box with 500 million knobs, 1,000 light bulbs, and 10 million images to train it with. That’s what a typical Deep Learning system is.”
  • 29. Aim: Learning features ■ Deep learning excels in tasks where the basic unit, a single pixel, a single frequency, or a single word has very little meaning in and of itself, but the combination of such units has a useful meaning. It can learn these useful combinations of values without any human intervention.
  • 31. Neural networks ■ An input, output, and one or more hidden layers of units/neurons/perceptrons ■ Each connection between two neurons has a weight w. Best weights can again be found with gradient descent. Image courtesy of http://ljs.academicdirect.org/A15/053_070.htm
  • 32. Neural networks ■ Example: Input vector: [7, 1, 2]  Into the input units ■ Forward propagation ■ Activation function Image courtesy of http://ljs.academicdirect.org/A15/053_070.htm
  • 33. Neural networks ■ Why deep? ■ Number of parameterized transformations a signal encounters as it propagates from the input layer to the output layer, where a parameterized transformation is a processing unit that has trainable parameters, such as weights. Image courtesy of http://ljs.academicdirect.org/A15/053_070.htm
  • 34. Aim: Learning features ■ The goal of deep learning methods is to learn higher levels of feature from lower level features.
  • 35. Other important concepts ■ Overfitting – there is such a thing as learning too much –or too specific-! ■ Regularization – a technique that prevents overfitting
  • 36. Overfitting ■ Overfitting – there is such a thing as learning too much –or too specific-! ■ Regularization – a technique that prevents overfitting
  • 38. Different frameworks ■ Pylearn2, Lasagne,Caffe,Torch,Theano, Blocks, Plate,Crino,Theanet, DL4J, Keras, …
  • 39. Different frameworks ■ Theano: – A mathematical expression compiler, designed with machine learning in mind. – Lets you define an objective and automatically produces the code that computes the gradient of the objective. – Good for experimenting with different loss functions – Slightly lower layer of abstraction vs more possibilities
  • 40. Different frameworks ■ Caffe: – Developed by UC Berkeley – Widely used machine-vision library that ported Matlab’s implementation of fast convolutional nets to C and C++ – Not intended for other deep-learning applications such as text, sound or time series data CORRECTION: There are new implementations of RNNs and LSTMs in Caffe, so it is not only for images any more! – Very fast: over 60M images per day with a single NVIDIA K40 GPU
  • 41. Different frameworks ■ Torch: – Written in Lua (a scripting language developed in Brazil in the early 1990s) – A highly customized version of it is used by large tech companies such as Google and Facebook
  • 42. Different frameworks ■ Keras: – Minimalist, highly modular neural network library in the spirit ofTorch – Written in Python – UsesTheano under the hood for optimized tensor manipulation on GPU andCPU – It was developed with a focus on enabling fast experimentation – 60K images took 30 hours on Amazon g2.2xlarge 
  • 43. Comparing Keras andTheano MNIST digits dataset - serves as a benchmark to compare results with as new articles come out. Multilayer Perceptron - Basic feedforward neural network
  • 44. Demo Code snippets – inside the gradient descent Output =Wx+b
  • 45. Demo Code snippets – inside the hidden layer
  • 46. Demo Code snippets – inside the hidden layer
  • 47. Demo Code snippets – inside the hidden layer
  • 48. Demo Code snippets – inside the network
  • 50. Future of deep learning ■ Deep learning has a lot of hype right now, and it is apparent that it is very useful for specific tasks. ■ What frontiers and challenges do you think are the most exciting for researchers in the field of neural networks in the next ten years? ■ I cannot see ten years into the future. For me, the wall of fog starts at about 5 years. ... I think that the most exciting areas over the next five years will be really understanding videos and text. I will be disappointed if in five years time we do not have something that can watch aYouTube video and tell a story about what happened. I have had a lot of disappointments. – From Geoffrey Hinton’s AMA on Reddit
  • 51. Now &The future Facebook Deep Learning, March 26, 2015 Image courtesy ofVenturebeat.com
  • 52. Join us! ■ Open positions: https://angel.co/algorithmia/jobs/ – Algorithm Developer [this is me!] – Backend Developer – Product Manager – Technical Evangelist
  • 53. Further resources ■ Introductory: ■ Andrew Ng’s Machine Learning course on Coursera ■ Geoffrey Hinton’s Neural Networks course on Coursera ■ Advanced: ■ Stanford’s Convolutional Neural Networks forVisual Recognition http://cs231n.github.io/ ■ Who is afraid of non-convex loss functions? ByYann LeCun http://videolectures.net/eml07_lecun_wia/ ■ What is wrong with Deep Learning? ByYann Lecun http://techtalks.tv/talks/whats-wrong-with-deep-learning/61639/ ■ For those who like papers, recent advances: ■ PlayingAtari with Deep Reinforcement Learning - http://www.cs.toronto.edu/~vmnih/docs/dqn.pdf ■ Unsupervised Face Detection - http://cs.stanford.edu/~quocle/faces_full.pdf
  • 54. ■ Content: ■ Toptal.com, Deeplearning.net ■ http://www.computerworld.com/article/2918161/emerging-technology/the-ai-ecosystem.html ■ Introduction to Machine Learning CMU-10701 - Deep Learning slides ■ Images: ■ http://www.spyemporium.com/images/products/st-sc1720.jpg ■ http://stats.stackexchange.com/questions/128616/whats-a-real-world-example-of-overfitting ■ http://www.homedepot.com/catalog/productImages/1000/c4/c4c34d2e-56ce-4c11-94c0-67aa19b769fa_1000.jpg ■ http://www.bulborama.com/images/products/1933.jpg ■ https://xkcd.com/1122/, https://xkcd.com/1425/ ■ www.deeplearning.net

Editor's Notes

  1. Sensing, reasoning and communicating speech and image recognition different flavors of reasoning logic versus evidence-based How do you tell a car from a dog as a human? Netflix, Amazon recommendations, how do they do it, how do we do it? – similar people (you like matrix, I liked matrix, I also liked terminator, maybe you’ll like it too?)
  2. WHY are we doing this? Siri, Cortana, Google now -> we do it so you don’t have to
  3. Specialized brain cells, take signal, carry it, Shallow architectures took much less time to train and had comparable or even higher accuracies 2006 paper - A fast learning algorithm for deep belief nets.
  4. How would you (as a human) describe this?
  5. Computer generated/dreamed images
  6. I will talk about ml first
  7. Challenge: Teaching a machine to tell the difference between a dog and a car Yann Lecun – knobs, parameters
  8. salary, happiness at a job
  9. Important thing vs non-important thing
  10. What do we do in this case, as humans? – Draw the best fit line
  11. Typically, the dataset is represented in a matrix where rows are examples and columns are features The features generally have “coefficients” in the equation that we call weights.
  12. This coefficient for our feature becomes the weight – something that implies how important this is
  13. Grandmother example!
  14. Ask 0-1 error, gradient
  15. 3D, more complicated cases
  16. Units -> Very specialized little workers Specialized cells brain
  17. The figure above shows a network with a 3-unit input layer, 5-unit hidden layer and an output layer with 2 units Weight is what makes neurons specialized, each gets different weight (toilet size less weight, salary more weight) think of it as thicker/thinner connection lines in between
  18. - These values are then propagated forward to the hidden units using the weighted sum transfer function for each hidden unit - Sigmoid, hyperbolic tangent -> helps with differentiation, rectifiers becoming the norm in deep learning - Lots of small impulses, if below a threshold nothing happens but if you cross the threshold, you get an action potential
  19. texture neurons Hair face body part neurons Posture neurons
  20. White dog example
  21. Paul Allen AI Research Institute project Computer learning from text, graphics to pass the 4th grade exam on its own