Introduction to Recurrent Neural Network
•
8 likes•5,030 views
Basic concepts of RNN and introduction to Long short term memory network; Presented at Houston Machine Learning meetup.
Introduction to Recurrent Neural Network
- 1. Yan Xu Houston Machine Learning Meetup May 20, 2017 Introduction to Recurrent Neural Network
- 2. Roadmap • Tour of machine learning algorithms (1 session) • Feature engineering (1 session) • Feature selection - Yan • Supervised learning (4 sessions) • Regression models -Yan • SVM and kernel SVM - Yan • Tree-based models - Dario • Bayesian method - Xiaoyang • Ensemble models - Yan • Unsupervised learning (3 sessions) • K-means clustering • DBSCAN - Cheng • Mean shift • Agglomerative clustering – Kunal • Spectral clustering – Yan • Dimension reduction for data visualization - Yan • Deep learning (4 sessions) • Neural network - Yan • Convolutional neural network – Hengyang Lu • Recurrent neural networks – Yan • Hands-on session with deep nets Slides posted on: http://www.slideshare.net/xuyangela
- 3. More deep learning coming up! • Optimization in Deep learning • Behind AlphaGo • Mastering the game of Go with deep neural networks and tree search • Deep learning showcase: Share your experience!
- 4. Outline • Recap on neural network • Recurrent neural network overview • Application of RNN • Long short term memory network • An example
- 5. Recap: Feed-forward neural network Activation function
- 6. Activation function rectified linear unit (ReLU)
- 7. Training with gradient descent
- 8. Convolutional Neural Network Full-connected neural nets Convolutional neural nets (kernel size = 2)
- 9. Recurrent NN: Considering Sequence
- 10. Recurrent Neural Network http://colah.github.io/posts/2015-08-Understanding-LSTMs/ Weights are kept the same in cell A!
- 13. Conversation Bot
- 15. Image Search
- 17. Wide application of RNN Image classification Image Captioning Sentiment analysis Machine translation Labeling each frame of video
- 18. Special RNN: LSTM NN • Short term memory • Long term memory the clouds are in the sky I grew up in China … I speak fluent Chinese.
- 19. Special RNN: LSTM NN SLTM in products! • Google Translate • Apple Siri • Amazon Alexa Cell https://www.youtube.com/watch?v=93rzMHtYT_0
- 21. Keep gate N × 1 M × 1 （N+M) × 1N × (N+M) N × 1 N × 1N × 1 http://colah.github.io/posts/2015-08-Understanding-LSTMs/
- 25. Training LSTM • Back propagates like feed-forward nets • Sum up all updates and applied to all
- 26. Example: Predicting next word https://medium.com/towards-data-science/lstm-by-example-using-tensorflow-feb0c1968537
- 27. Each word represented by an integer. Output is a one-hot vector. 512 hidden units Improvement? Example: Predicting next word
- 28. Generating a story! Input: a general council had a general council to consider what measures they could take to outwit their common enemy , the cat . some said this , and some said that but at last a young mouse got Input: mouse mouse mouse mouse mouse mouse , neighbourhood and could receive a outwit always the neck of the cat . some said this , and some said that but at last a young mouse got up and said
- 29. Great reference • http://colah.github.io/posts/2015-08-Understanding-LSTMs/ • https://medium.com/@ageitgey/machine-learning-is-fun-part-5-language- translation-with-deep-learning-and-the-magic-of-sequences-2ace0acca0aa • Visualizing and Understanding RNN: • https://skillsmatter.com/skillscasts/6611-visualizing-and-understanding-recurrent-networks
- 30. Summary • Learn about RNN, how it relates to feed forward NN • Long short term memory RNN • Keep gate • Write gate • Read gate • Application and Example
- 31. Roadmap • Tour of machine learning algorithms (1 session) • Feature engineering (1 session) • Feature selection - Yan • Supervised learning (4 sessions) • Regression models -Yan • SVM and kernel SVM - Yan • Tree-based models - Dario • Bayesian method - Xiaoyang • Ensemble models - Yan • Unsupervised learning (3 sessions) • K-means clustering • DBSCAN - Cheng • Mean shift • Agglomerative clustering – Kunal • Spectral clustering – Yan • Dimension reduction for data visualization - Yan • Deep learning (4 sessions) • Neural network - Yan • Convolutional neural network – Hengyang Lu • Recurrent neural networks – Yan • Hands-on session with deep nets Slides posted on: http://www.slideshare.net/xuyangela More deep learning coming up!
- 32. Thank you Data Disruptors Conference, ddc (energy) @ Houston, June 14 PROMO: HEDS99 to get 99$ off Slides will be posted at: http://www.slideshare.net/xuyangela Leave a group review please