Optimization in Deep Learning
•Download as PPTX, PDF•
7 likes•3,270 views
This document summarizes various optimization techniques for deep learning models, including gradient descent, stochastic gradient descent, and variants like momentum, Nesterov's accelerated gradient, AdaGrad, RMSProp, and Adam. It provides an overview of how each technique works and comparisons of their performance on image classification tasks using MNIST and CIFAR-10 datasets. The document concludes by encouraging attendees to try out the different optimization methods in Keras and provides resources for further deep learning topics.
Report
Share
More Related Content
What's hot
What's hot (20)
Similar to Optimization in Deep Learning
Similar to Optimization in Deep Learning (20)
More from Yan Xu
More from Yan Xu (20)
Recently uploaded
Recently uploaded (20)
Optimization in Deep Learning
- 1. Optimization in Deep Learning Houston Machine Learning Deep Learning Series
- 2. How to get to the lake
- 3. Roadmap • Tour of machine learning algorithms (1 session) • Feature engineering (1 session) • Feature selection - Yan • Supervised learning (4 sessions) • Regression models -Yan • SVM and kernel SVM - Yan • Tree-based models - Dario • Bayesian method - Xiaoyang • Ensemble models - Yan • Unsupervised learning (3 sessions) • K-means clustering • DBSCAN - Cheng • Mean shift • Agglomerative clustering – Kunal • Spectral clustering – Yan • Dimension reduction for data visualization - Yan • Deep learning • Neural network - Yan • Convolutional neural network – Hengyang Lu • Recurrent neural networks – Yan • Hands-on session with deep nets - Yan Slides posted on: http://www.slideshare.net/xuyangela
- 4. More deep learning coming up! • Optimization in Deep learning (today’s session) • Behind AlphaGo • Mastering the game of Go with deep neural networks and tree search • Attention network • Application of Deep Learning and showcase
- 5. Outline • Gradient Descent • Stochastic Gradient Descent (SGD) • Variants of SGD • Use “momentum” • Nestrov’s Accelerated Gradient (NAG) • Adaptive Gradient (AdaGrad) • Root Mean Square Propagation (RMSProp) • Adaptive Moment Estimation (Adam)
- 8. Scaling to large N
- 9. Stochastic Gradient Descent (SGD)
- 10. Mini-batch SGD
- 12. SGD recommendation • Randomly shuffle training samples • Monitor training and validation error • Experiment learning rates using small sample of training set • Leverage sparsity of training samples • Varying learning rate:
- 13. Variants of SGD • Use “momentum” • Nestrov’s Accelerated Gradient (NAG) • Adaptive Gradient (AdaGrad) • Root Mean Square Propagation (RMSProp) • Adaptive Moment Estimation (Adam) Ref: https://moodle2.cs.huji.ac.il/nu15/pluginfile.php/316969/mod_resource/conte nt/1/adam_pres.pdf
- 17. The momentum method by Dr. Geoffrey Hinton https://www.youtube.com/watch?v=LdkkZglLZ0Q&list=PLoRl3Ht4JOcdU872GhiYWf6jwrk_SNhz9&index=27
- 18. SGD with momentum Start with 0.5
- 19. NAG (Nesterov’s Accelerated Gradient)
- 20. AdaGrad Adaptive learning rate: • weights that receive high gradients will have their effective learning rate reduced • weights that receive small or infrequent updates will have their effective learning rate increased
- 21. RMSProp
- 22. Adam
- 23. Adam
- 24. Comparisons of Different Optimization Methods
- 25. MINIST Comparisons of Different Optimization Methods
- 26. CIFAR-10 Comparisons of Different Optimization Methods
- 27. Summary of learning methods for DL https://www.youtube.com/watch?v=defQQqkXEfE&list=PLoRl3Ht4JOcdU872GhiYWf6jwrk_SNhz9&index=29 from:7:33
- 28. Try it out! From hands-on session: https://www.dropbox.com/s/92sckhnf1hjgjlo/CNN.zip?dl=0 model = Sequential() model.add(Conv2D(32, (3, 3), input_shape=input_shape)) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(2, 2))) ……. model.add(Flatten()) model.add(Dense(64)) model.add(Activation('relu')) model.add(Dropout(0.5)) model.add(Dense(1)) model.add(Activation('sigmoid')) model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy']) Optimizer: SGD, RMSprop, Adagrad, Adam…. (https://keras.io/optimizers/)
- 29. Summary Full-batch GD SGD Momentum SGD NAG AdaGrad RMSProp Adam Speed up by momentum Adaptive learning rate
- 30. More deep learning coming up! • Optimization in Deep learning (today’s session) • Behind AlphaGo • Mastering the game of Go with deep neural networks and tree search • Attention network • Application of Deep Learning and showcase • Any proposal?
- 31. Thank you Slides will be posted at: http://www.slideshare.net/xuyangela Leave a group review please
Editor's Notes
- Perpendicular contour