Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

GAN - Theory and Applications

•
•
E
Emanuele Ghelfi

GANs are the new hottest topic in the ML arena; however, they present a challenge for the researchers and the engineers alike. Their design, and most importantly, the code implementation has been causing headaches to the ML practitioners, especially when moving to production. Starting from the very basic of what a GAN is, passing trough Tensorflow implementation, using the most cutting-edge APIs available in the framework, and finally, production-ready serving at scale using Google Cloud ML Engine. Slides for the talk: https://www.pycon.it/conference/talks/deep-diving-into-gans-form-theory-to-production Github repo: https://github.com/zurutech/gans-from-theory-to-production

1 of 41
Download to read offline
GAN - Theory and Applications Emanuele Ghelfi @manughelfi Paolo Galeone @paolo_galeone Federico Di Mattia @_iLeW_ Michele De Simoni @mr_ubik https://bit.ly/2Y1nqay May 4, 2019 1
GAN - Theory and Applications
Overview 1. Introduction 2. Models definition 3. GANs Training 4. Types of GANs 5. GANs Applications 3
Introduction
“ Generative Adversarial Networks is the most interesting idea in the last ten years in machine learning. Yann LeCun, Director, Facebook AI ” 4
Generative Adversarial Networks Two components, the generator and the discriminator: • The generator G needs to capture the data distribution. • The discriminator D estimates the probability that a sample comes from the training data rather than from G. Figure 1: Credits: Silva 5
Generative Adversarial Networks GANs game: min G max D VGAN(D, G) = E x∼pdata(x) [log D(x)] + E z∼pz(z) [log(1 − D(G(z)))] 6
Generative Adversarial Networks GANs game: min G max D VGAN(D, G) = E x∼pdata(x) [log D(x)] real samples + E z∼pz(z) [log(1 − D(G(z)))] 6
Generative Adversarial Networks GANs game: min G max D VGAN(D, G) = E x∼pdata(x) [log D(x)] real samples + E z∼pz(z) [log(1 − D(G(z)))] generated samples 6
GANs - Discriminator • Discriminator needs to: • Correctly classify real data: max D E x∼pdata(x) [log D(x)] D(x) → 1 • Correctly classify wrong data: max D E z∼pz(z) [log(1 − D(G(z)))] D(G(z)) → 0 • The discriminator is an adaptive loss function. 7
GAN - Theory and Applications
GANs - Generator • Generator needs to fool the discriminator: • Generate samples similar to the real ones: min G E z∼pz(z) [log(1 − D(G(z)))] D(G(z)) → 1 9
GANs - Generator • Generator needs to fool the discriminator: • Generate samples similar to the real ones: min G E z∼pz(z) [log(1 − D(G(z)))] D(G(z)) → 1 • Non saturating objective (Goodfellow et al., 2014): min G E z∼pz(z) [− log(D(G(z)))] 9
GANs - Generator Objectives • Minimax: log(1 − D(G(z))) 0 0.5 1 −6 −4 −2 0 2 4 D(G(z)) JG Minimax 10
GANs - Generator Objectives • Minimax: log(1 − D(G(z))) • Non-saturating: − log(D(G(z))) 0 0.5 1 −6 −4 −2 0 2 4 D(G(z)) JG Minimax Non-saturating 10
Models definition
GANs - Models definition • Different architectures for different data types. • Tuple of numbers? Fully Connected Neural Networks 11
GANs - Models definition • Different architectures for different data types. • Text or sequences? Recurrent Neural Networks 11
GANs - Models definition • Different architectures for different data types. • Images? Convolutional Neural Networks 1 latent 1 fc 32768 256 *conv1 32 128 *conv2 64 64 *conv3 128 K *conv4 128 3 128 conv5 Latent Vector Conv/Deconv Fully Connected Batch Norm Relu 11
GANs Training
GANs - Training • D and G are competing against each other. • Alternating execution of training steps. • Use minibatch stochastic gradient descent/ascent. 12
GANs - Training - Discriminator How to train the discriminator? Repeat from 1 to k: 1. Sample minibatch of m noise samples z(1), . . . , z(m) from pz(z) 13
GANs - Training - Discriminator How to train the discriminator? Repeat from 1 to k: 1. Sample minibatch of m noise samples z(1), . . . , z(m) from pz(z) 2. Sample minibatch of m examples x(1), . . . , x(m) from pdata(x) 13
GANs - Training - Discriminator How to train the discriminator? Repeat from 1 to k: 1. Sample minibatch of m noise samples z(1), . . . , z(m) from pz(z) 2. Sample minibatch of m examples x(1), . . . , x(m) from pdata(x) 3. Update D: J = 1 m m∑ i=1 log D(x(i) ) + log(1 − D(G(z(i) ))) D performance θd = θd + λ∇θd J 13
GANs - Training - Generator How to train the generator? Update executed only once after D updates: 1. Sample minibatch of m noise samples z(1), . . . , z(m) from pz(z) 14
GANs - Training - Generator How to train the generator? Update executed only once after D updates: 1. Sample minibatch of m noise samples z(1), . . . , z(m) from pz(z) 2. Update G: J = 1 m m∑ i=1 log(D(G(z(i) ))) G performance θg = θg + λ∇θgJ 14
GANs - Training - Considerations • Optimizers: Adam, Momentum, RMSProp. • Arbitrary number of steps or epochs. • Training is completed when D is completely fooled by G. • Goal: reach a Nash Equilibrium where the best D can do is random guessing. 15
Types of GANs
Types of GANs Two big families: • Unconditional GANs (just described). • Conditional GANs (Mirza and Osindero, 2014). 16
Conditional GANs • Both G and D are conditioned on some extra information y. • In practice: perform conditioning by feeding y into D and G. Figure 2: From Mirza and Osindero (2014) 17
Conditional GANs The GANs game becomes: min G max D E x∼pdata(x|y) [log D(x, y)] + E z∼pz(z) [log(1 − D(G(z|y), y))] Notice: the same representation of the condition has to be presented to both network. 18
GANs Applications
Unconditional - Face Generation - Karras et al. (2017) 19
Conditional - Domain Translation - Isola et al. (2016) 20
Conditional - Semantic Image Synthesis - Park et al. (2018) 21
Conditional - Image Super Resolution - Ledig et al. (2016) 22
Real-world GANs • Semi-Supervised Learning (Salimans et al., 2016) • Image Generation (almost all GAN papers) • Image Captioning • Anomalies Detection (Zenati et al., 2018) • Program Synthesis (Ganin et al., 2018) • Genomics and Proteomics (Killoran et al., 2017) (De Cao and Kipf, 2018) • Personalized GANufactoring (Hwang et al., 2018) • Planning
References [De Cao and Kipf 2018] De Cao, Nicola ; Kipf, Thomas: MolGAN: An Implicit Generative Model for Small Molecular Graphs. (2018). – (2018) [Ganin et al. 2018] Ganin, Yaroslav ; Kulkarni, Tejas ; Babuschkin, Igor ; Eslami, S. M. A. ; Vinyals, Oriol: Synthesizing Programs for Images Using Reinforced Adversarial Learning. (2018). – (2018) [Goodfellow et al. 2014] Goodfellow, Ian J. ; Pouget-Abadie, Jean ; Mirza, Mehdi ; Xu, Bing ; Warde-Farley, David ; Ozair, Sherjil ; Courville, Aaron ; Bengio, Yoshua: Generative Adversarial Networks. (2014). – (2014)
[Hwang et al. 2018] Hwang, Jyh-Jing ; Azernikov, Sergei ; Efros, Alexei A. ; Yu, Stella X.: Learning Beyond Human Expertise with Generative Models for Dental Restorations. (2018). – (2018) [Isola et al. 2016] Isola, Phillip ; Zhu, Jun-Yan ; Zhou, Tinghui ; Efros, Alexei A.: Image-to-Image Translation with Conditional Adversarial Networks. (2016). – (2016) [Karras et al. 2017] Karras, Tero ; Aila, Timo ; Laine, Samuli ; Lehtinen, Jaakko: Progressive Growing of GANs for Improved Quality, Stability, and Variation. (2017). – (2017) [Killoran et al. 2017] Killoran, Nathan ; Lee, Leo J. ; Delong, Andrew ; Duvenaud, David ; Frey, Brendan J.: Generating and Designing DNA with Deep Generative Models. (2017). – (2017)
[Ledig et al. 2016] Ledig, Christian ; Theis, Lucas ; Huszar, Ferenc ; Caballero, Jose ; Cunningham, Andrew ; Acosta, Alejandro ; Aitken, Andrew ; Tejani, Alykhan ; Totz, Johannes ; Wang, Zehan ; Shi, Wenzhe: Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. (2016). – (2016) [Mirza and Osindero 2014] Mirza, Mehdi ; Osindero, Simon: Conditional Generative Adversarial Nets. (2014). – (2014) [Park et al. 2018] Park, Taesung ; Liu, Ming-Yu ; Wang, Ting-Chun ; Zhu, Jun-Yan: Semantic Image Synthesis with Spatially-Adaptive Normalization. (2018). – (2018) [Salimans et al. 2016] Salimans, Tim ; Goodfellow, Ian ; Zaremba, Wojciech ; Cheung, Vicki ; Radford, Alec ; Chen, Xi: Improved Techniques for Training GANs. (2016). – (2016)
[Silva ] Silva, Thalles: An Intuitive Introduction to Generative Adversarial Networks (GANs) [Zenati et al. 2018] Zenati, Houssam ; Foo, Chuan S. ; Lecouat, Bruno ; Manek, Gaurav ; Chandrasekhar, Vijay R.: Efficient GAN-Based Anomaly Detection. (2018). – (2018)

More Related Content

GAN - Theory and Applications

  • 1. GAN - Theory and Applications Emanuele Ghelfi @manughelfi Paolo Galeone @paolo_galeone Federico Di Mattia @_iLeW_ Michele De Simoni @mr_ubik https://bit.ly/2Y1nqay May 4, 2019 1
  • 3. Overview 1. Introduction 2. Models definition 3. GANs Training 4. Types of GANs 5. GANs Applications 3
  • 5. “ Generative Adversarial Networks is the most interesting idea in the last ten years in machine learning. Yann LeCun, Director, Facebook AI ” 4
  • 6. Generative Adversarial Networks Two components, the generator and the discriminator: • The generator G needs to capture the data distribution. • The discriminator D estimates the probability that a sample comes from the training data rather than from G. Figure 1: Credits: Silva 5
  • 7. Generative Adversarial Networks GANs game: min G max D VGAN(D, G) = E x∼pdata(x) [log D(x)] + E z∼pz(z) [log(1 − D(G(z)))] 6
  • 8. Generative Adversarial Networks GANs game: min G max D VGAN(D, G) = E x∼pdata(x) [log D(x)] real samples + E z∼pz(z) [log(1 − D(G(z)))] 6
  • 9. Generative Adversarial Networks GANs game: min G max D VGAN(D, G) = E x∼pdata(x) [log D(x)] real samples + E z∼pz(z) [log(1 − D(G(z)))] generated samples 6
  • 10. GANs - Discriminator • Discriminator needs to: • Correctly classify real data: max D E x∼pdata(x) [log D(x)] D(x) → 1 • Correctly classify wrong data: max D E z∼pz(z) [log(1 − D(G(z)))] D(G(z)) → 0 • The discriminator is an adaptive loss function. 7
  • 12. GANs - Generator • Generator needs to fool the discriminator: • Generate samples similar to the real ones: min G E z∼pz(z) [log(1 − D(G(z)))] D(G(z)) → 1 9
  • 13. GANs - Generator • Generator needs to fool the discriminator: • Generate samples similar to the real ones: min G E z∼pz(z) [log(1 − D(G(z)))] D(G(z)) → 1 • Non saturating objective (Goodfellow et al., 2014): min G E z∼pz(z) [− log(D(G(z)))] 9
  • 14. GANs - Generator Objectives • Minimax: log(1 − D(G(z))) 0 0.5 1 −6 −4 −2 0 2 4 D(G(z)) JG Minimax 10
  • 15. GANs - Generator Objectives • Minimax: log(1 − D(G(z))) • Non-saturating: − log(D(G(z))) 0 0.5 1 −6 −4 −2 0 2 4 D(G(z)) JG Minimax Non-saturating 10
  • 17. GANs - Models definition • Different architectures for different data types. • Tuple of numbers? Fully Connected Neural Networks 11
  • 18. GANs - Models definition • Different architectures for different data types. • Text or sequences? Recurrent Neural Networks 11
  • 19. GANs - Models definition • Different architectures for different data types. • Images? Convolutional Neural Networks 1 latent 1 fc 32768 256 *conv1 32 128 *conv2 64 64 *conv3 128 K *conv4 128 3 128 conv5 Latent Vector Conv/Deconv Fully Connected Batch Norm Relu 11
  • 21. GANs - Training • D and G are competing against each other. • Alternating execution of training steps. • Use minibatch stochastic gradient descent/ascent. 12
  • 22. GANs - Training - Discriminator How to train the discriminator? Repeat from 1 to k: 1. Sample minibatch of m noise samples z(1), . . . , z(m) from pz(z) 13
  • 23. GANs - Training - Discriminator How to train the discriminator? Repeat from 1 to k: 1. Sample minibatch of m noise samples z(1), . . . , z(m) from pz(z) 2. Sample minibatch of m examples x(1), . . . , x(m) from pdata(x) 13
  • 24. GANs - Training - Discriminator How to train the discriminator? Repeat from 1 to k: 1. Sample minibatch of m noise samples z(1), . . . , z(m) from pz(z) 2. Sample minibatch of m examples x(1), . . . , x(m) from pdata(x) 3. Update D: J = 1 m m∑ i=1 log D(x(i) ) + log(1 − D(G(z(i) ))) D performance θd = θd + λ∇θd J 13
  • 25. GANs - Training - Generator How to train the generator? Update executed only once after D updates: 1. Sample minibatch of m noise samples z(1), . . . , z(m) from pz(z) 14
  • 26. GANs - Training - Generator How to train the generator? Update executed only once after D updates: 1. Sample minibatch of m noise samples z(1), . . . , z(m) from pz(z) 2. Update G: J = 1 m m∑ i=1 log(D(G(z(i) ))) G performance θg = θg + λ∇θgJ 14
  • 27. GANs - Training - Considerations • Optimizers: Adam, Momentum, RMSProp. • Arbitrary number of steps or epochs. • Training is completed when D is completely fooled by G. • Goal: reach a Nash Equilibrium where the best D can do is random guessing. 15
  • 29. Types of GANs Two big families: • Unconditional GANs (just described). • Conditional GANs (Mirza and Osindero, 2014). 16
  • 30. Conditional GANs • Both G and D are conditioned on some extra information y. • In practice: perform conditioning by feeding y into D and G. Figure 2: From Mirza and Osindero (2014) 17
  • 31. Conditional GANs The GANs game becomes: min G max D E x∼pdata(x|y) [log D(x, y)] + E z∼pz(z) [log(1 − D(G(z|y), y))] Notice: the same representation of the condition has to be presented to both network. 18
  • 33. Unconditional - Face Generation - Karras et al. (2017) 19
  • 34. Conditional - Domain Translation - Isola et al. (2016) 20
  • 35. Conditional - Semantic Image Synthesis - Park et al. (2018) 21
  • 36. Conditional - Image Super Resolution - Ledig et al. (2016) 22
  • 37. Real-world GANs • Semi-Supervised Learning (Salimans et al., 2016) • Image Generation (almost all GAN papers) • Image Captioning • Anomalies Detection (Zenati et al., 2018) • Program Synthesis (Ganin et al., 2018) • Genomics and Proteomics (Killoran et al., 2017) (De Cao and Kipf, 2018) • Personalized GANufactoring (Hwang et al., 2018) • Planning
  • 38. References [De Cao and Kipf 2018] De Cao, Nicola ; Kipf, Thomas: MolGAN: An Implicit Generative Model for Small Molecular Graphs. (2018). – (2018) [Ganin et al. 2018] Ganin, Yaroslav ; Kulkarni, Tejas ; Babuschkin, Igor ; Eslami, S. M. A. ; Vinyals, Oriol: Synthesizing Programs for Images Using Reinforced Adversarial Learning. (2018). – (2018) [Goodfellow et al. 2014] Goodfellow, Ian J. ; Pouget-Abadie, Jean ; Mirza, Mehdi ; Xu, Bing ; Warde-Farley, David ; Ozair, Sherjil ; Courville, Aaron ; Bengio, Yoshua: Generative Adversarial Networks. (2014). – (2014)
  • 39. [Hwang et al. 2018] Hwang, Jyh-Jing ; Azernikov, Sergei ; Efros, Alexei A. ; Yu, Stella X.: Learning Beyond Human Expertise with Generative Models for Dental Restorations. (2018). – (2018) [Isola et al. 2016] Isola, Phillip ; Zhu, Jun-Yan ; Zhou, Tinghui ; Efros, Alexei A.: Image-to-Image Translation with Conditional Adversarial Networks. (2016). – (2016) [Karras et al. 2017] Karras, Tero ; Aila, Timo ; Laine, Samuli ; Lehtinen, Jaakko: Progressive Growing of GANs for Improved Quality, Stability, and Variation. (2017). – (2017) [Killoran et al. 2017] Killoran, Nathan ; Lee, Leo J. ; Delong, Andrew ; Duvenaud, David ; Frey, Brendan J.: Generating and Designing DNA with Deep Generative Models. (2017). – (2017)
  • 40. [Ledig et al. 2016] Ledig, Christian ; Theis, Lucas ; Huszar, Ferenc ; Caballero, Jose ; Cunningham, Andrew ; Acosta, Alejandro ; Aitken, Andrew ; Tejani, Alykhan ; Totz, Johannes ; Wang, Zehan ; Shi, Wenzhe: Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. (2016). – (2016) [Mirza and Osindero 2014] Mirza, Mehdi ; Osindero, Simon: Conditional Generative Adversarial Nets. (2014). – (2014) [Park et al. 2018] Park, Taesung ; Liu, Ming-Yu ; Wang, Ting-Chun ; Zhu, Jun-Yan: Semantic Image Synthesis with Spatially-Adaptive Normalization. (2018). – (2018) [Salimans et al. 2016] Salimans, Tim ; Goodfellow, Ian ; Zaremba, Wojciech ; Cheung, Vicki ; Radford, Alec ; Chen, Xi: Improved Techniques for Training GANs. (2016). – (2016)
  • 41. [Silva ] Silva, Thalles: An Intuitive Introduction to Generative Adversarial Networks (GANs) [Zenati et al. 2018] Zenati, Houssam ; Foo, Chuan S. ; Lecouat, Bruno ; Manek, Gaurav ; Chandrasekhar, Vijay R.: Efficient GAN-Based Anomaly Detection. (2018). – (2018)