Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

Generative Adversarial Networks and Their Applications in Medical Imaging

•Download as PPTX, PDF•
•
Sanghoon Hong
Sanghoon Hong

의료 인공지능 교육 워크샵 (17.09.16) 발표자료 - Generative Adversarial Network에 대한 소개 - Medical imaging 분야에서의 연구 사례 - Our research

1 of 58
Copyright © KakaoBrain Corp. All rights reserved. Generative Adversarial Networks and Their Applications in Medical Imaging September 16, 2017 Sanghoon Hong
1. Generative Adversarial Network (GAN) 2. GAN in Medical Imaging + Our Research Project Contents
• Discriminative vs. Generative Generative Model CNN classifier GAN “cat” an inverse function?
• Discriminative vs. Generative Generative Model https://duphan.wordpress.com/tag/generative-model/ data distribution p(x,z) or p(x|z) Difficult, but important
• Data distribution Generative Model A slide from “Crash Course on Machine Learning Part II” 주요 특징만 필요 세밀한 특성, 분포를 완전히 알아야함
• Applications? • Structured prediction (e.g., output text) • Much more robust prediction • Anomaly detection • … Generative Model
Generative Model Slide credit Goodfellow 2016
“the coolest idea in ML in the last twenty years” - Yann LeCun Generative Adversarial Network (GAN)
Generative Adversarial Network (GAN) https://github.com/hindupuravinash/the-gan-zoo/
• Variational Auto-Encoder • Data의 generation은 잘 됨 • 그러나 blurry face Why GANs are different? “Variational Autoencoder and Extensions” by Aaron CouCourville Sample interpolation
• Why are generated samples blurry? • Regress to the mean => blurry images Why GANs are different? Ledig, C. et al. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. MSE or Euclidean distance
Why GANs are different? from “Tutorial on Theory and Application of Generative Adversarial Networks” (CVPR17)
• Difficult to hand-craft a good perceptual loss function • => 그럴듯한지 여부를 판단하는 neural network가 있다면? Generative Adversarial Network
Generative Adversarial Network Deep Learning for Computer Vision: Generative models and adversarial training (UPC2016)
Generative Adversarial Network Deep Learning for Computer Vision: Generative models and adversarial training (UPC2016) real / fake 판단하는 네트워크
Generative Adversarial Network https://ibmathsresources.com/2014/08/27/zenos-paradox-achilles-and-the-tortoise/ DG Berthelot, D., Schumm, T., & Metz, L. (2017). BEGAN: Boundary Equilibrium Generative Adversarial Networks. DG
Training GANs • Discriminator training Deep Learning for Computer Vision: Generative models and adversarial training (UPC2016)
Training GANs • Generator training Deep Learning for Computer Vision: Generative models and adversarial training (UPC2016)
• Adversarial loss (vanilla GAN) Training GANs D Real sample은 D(x) -> 1 Generated sample은 D(x) -> 0 G Generated sample도 D(x) -> 1
• Iterate these two steps until convergence • Eventually (we hope) that the generator gets so good that it is impossible for the discriminator to tell the difference between real and generated images. Discriminator accuracy = 0.5 Training GANs Discriminator Data Model Distribution Random guess
Training GANs Tutorial on Theory and Application of Generative Adversarial Networks (CVPR17)
• In practice? • Training stability (D vs. G balance, …) • Mode collapse Training GANs Generative Models II (CIFAR-CRM DLSS 2017) by Aaron Courville An image from “Generative adversarial networks” by Namju Kim
Training GANs https://github.com/hindupuravinash/the-gan-zoo
• Realistic image! GAN Results & Applications Berthelot, D., Schumm, T., & Metz, L. (2017). BEGAN: Boundary Equilibrium Generative Adversarial Networks. realistic & diverse samples Nguyen, A. et al. (2016). Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space
• Representation interpolation GAN Results & Applications Berthelot, D., Schumm, T., & Metz, L. (2017). BEGAN: Boundary Equilibrium Generative Adversarial Networks. Memorizing X
• Super-resolution GAN Results & Applications Ledig, C. et al. (2016) Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network.
GAN Results & Applications Low-res image Generated high-res image Generated or not?
• Image-to-image translation GAN Results & Applications Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2016). Image-to- Image Translation with Conditional Adversarial Networks.
GAN Results & Applications
GAN Results & Applications
Copyright © KakaoBrain Corp. All rights reserved. GAN in Medical Imaging
• Low-dose CT (noisy) => Routine-dose CT Literature Review (1) De-noising
Literature Review (1) De-noising loss = (voxel-similarity loss) + adversarial loss
Literature Review (1) De-noising Low-dose Routine-doseGAN-denoised “Training with an adversarial network allows the generator to better learn the noise distribution in routine-dose CT and produce more realistic images for more accurate coronary calcium quantification.” MSE-denoised
• CT => PET image translation Literature Review (2) - Image-to-image translation
Literature Review (2) - Image-to-image translation
Literature Review (2) - Image-to-image translation SL + GAN A mask with high predicted SUV values (>2.5) (???)
Literature Review (2) - Image-to-image translation input CT ground-truth PET generated PET
• MRI => CT image translation Literature Review (2) - Image-to-image translation
Literature Review (2) - Image-to-image translation Supervised training w/ real data Blurry More realistic
Literature Review (2) - Image-to-image translation Pelvic datasetBrain dataset Best mean absolute error & peak signal-to-noise ratio
• GAN research가 있지만, 아직 결과물의 수준이나 양 이 부족 • 그럼 우리는 무엇을 해볼 수 있을까? Our Research
• Medical image generation with GAN Our Research Noise
• Applications? • Unsupervised or semi-supervised training • Progression forecast & Visualization Our Research
• Medical image generation from scratch? • Structure + Dynamics + Variations + … How to Tackle
• Medical image generation from scratch? • Structure + Dynamics + Variations + … • Might be too difficult How to Tackle (GAN은 global structure / counting 같은 것에서 특히 약함)
• (First step) image generation w/ structural hints • Structure + Dynamics + Variations + … • Image-to-image translation과 유사 How to Tackle
• Gaze estimation task • Model-based synthetic data => GAN => Realistic data How to Tackle
How to Tackle Self-regularization Visual Turing test => 51.7% acc. State of the art w/o label real data
How to Tackle Self-regularization (minimizing feature dist.) Local adversarial loss (=PatchGAN) History of refined images History 안쓰면 artifacts 발생
• Possible hints? • Model-based synthetic data • Normal image + conditioning • Normal image + synthetic or expert-guided label How to Tackle Synthetic or guided label Generator
First Trial Generator Discriminator Real normal Real patient Image + label (for controllability) Generated patient Our target - synthetic or supervised label - Matching image-segmentation pair? - Realistic image?
First Trial Generator Discriminator Image + label (for controllability) PatchGAN DCGAN or MAD-GAN Real normal Real patient Image matching Generated patient
• Typical GAN issues (training, …) • Sample quality? (feat. M.D. researchers) • How to evaluate generated samples • Practical effectiveness? • Unsupervised (or semi-supervised) segmentation/classification in medical domain Have far to go
• One more thing…
Visualization of disease progression Identity Age
• GAN: an interesting & effective way to generate data • GAN in Medical Imaging? => 아직 초기 단계 • “Medical Image Generation with GAN” Conclusion
THANK YOU

More Related Content

Generative Adversarial Networks and Their Applications in Medical Imaging

  • 1. Copyright © KakaoBrain Corp. All rights reserved. Generative Adversarial Networks and Their Applications in Medical Imaging September 16, 2017 Sanghoon Hong
  • 2. 1. Generative Adversarial Network (GAN) 2. GAN in Medical Imaging + Our Research Project Contents
  • 3. • Discriminative vs. Generative Generative Model CNN classifier GAN “cat” an inverse function?
  • 4. • Discriminative vs. Generative Generative Model https://duphan.wordpress.com/tag/generative-model/ data distribution p(x,z) or p(x|z) Difficult, but important
  • 5. • Data distribution Generative Model A slide from “Crash Course on Machine Learning Part II” 주요 특징만 필요 세밀한 특성, 분포를 완전히 알아야함
  • 6. • Applications? • Structured prediction (e.g., output text) • Much more robust prediction • Anomaly detection • … Generative Model
  • 8. “the coolest idea in ML in the last twenty years” - Yann LeCun Generative Adversarial Network (GAN)
  • 9. Generative Adversarial Network (GAN) https://github.com/hindupuravinash/the-gan-zoo/
  • 10. • Variational Auto-Encoder • Data의 generation은 잘 됨 • 그러나 blurry face Why GANs are different? “Variational Autoencoder and Extensions” by Aaron CouCourville Sample interpolation
  • 11. • Why are generated samples blurry? • Regress to the mean => blurry images Why GANs are different? Ledig, C. et al. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. MSE or Euclidean distance
  • 12. Why GANs are different? from “Tutorial on Theory and Application of Generative Adversarial Networks” (CVPR17)
  • 13. • Difficult to hand-craft a good perceptual loss function • => 그럴듯한지 여부를 판단하는 neural network가 있다면? Generative Adversarial Network
  • 14. Generative Adversarial Network Deep Learning for Computer Vision: Generative models and adversarial training (UPC2016)
  • 15. Generative Adversarial Network Deep Learning for Computer Vision: Generative models and adversarial training (UPC2016) real / fake 판단하는 네트워크
  • 16. Generative Adversarial Network https://ibmathsresources.com/2014/08/27/zenos-paradox-achilles-and-the-tortoise/ DG Berthelot, D., Schumm, T., & Metz, L. (2017). BEGAN: Boundary Equilibrium Generative Adversarial Networks. DG
  • 17. Training GANs • Discriminator training Deep Learning for Computer Vision: Generative models and adversarial training (UPC2016)
  • 18. Training GANs • Generator training Deep Learning for Computer Vision: Generative models and adversarial training (UPC2016)
  • 19. • Adversarial loss (vanilla GAN) Training GANs D Real sample은 D(x) -> 1 Generated sample은 D(x) -> 0 G Generated sample도 D(x) -> 1
  • 20. • Iterate these two steps until convergence • Eventually (we hope) that the generator gets so good that it is impossible for the discriminator to tell the difference between real and generated images. Discriminator accuracy = 0.5 Training GANs Discriminator Data Model Distribution Random guess
  • 21. Training GANs Tutorial on Theory and Application of Generative Adversarial Networks (CVPR17)
  • 22. • In practice? • Training stability (D vs. G balance, …) • Mode collapse Training GANs Generative Models II (CIFAR-CRM DLSS 2017) by Aaron Courville An image from “Generative adversarial networks” by Namju Kim
  • 24. • Realistic image! GAN Results & Applications Berthelot, D., Schumm, T., & Metz, L. (2017). BEGAN: Boundary Equilibrium Generative Adversarial Networks. realistic & diverse samples Nguyen, A. et al. (2016). Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space
  • 25. • Representation interpolation GAN Results & Applications Berthelot, D., Schumm, T., & Metz, L. (2017). BEGAN: Boundary Equilibrium Generative Adversarial Networks. Memorizing X
  • 26. • Super-resolution GAN Results & Applications Ledig, C. et al. (2016) Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network.
  • 27. GAN Results & Applications Low-res image Generated high-res image Generated or not?
  • 28. • Image-to-image translation GAN Results & Applications Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2016). Image-to- Image Translation with Conditional Adversarial Networks.
  • 29. GAN Results & Applications
  • 30. GAN Results & Applications
  • 31. Copyright © KakaoBrain Corp. All rights reserved. GAN in Medical Imaging
  • 32. • Low-dose CT (noisy) => Routine-dose CT Literature Review (1) De-noising
  • 33. Literature Review (1) De-noising loss = (voxel-similarity loss) + adversarial loss
  • 34. Literature Review (1) De-noising Low-dose Routine-doseGAN-denoised “Training with an adversarial network allows the generator to better learn the noise distribution in routine-dose CT and produce more realistic images for more accurate coronary calcium quantification.” MSE-denoised
  • 35. • CT => PET image translation Literature Review (2) - Image-to-image translation
  • 36. Literature Review (2) - Image-to-image translation
  • 37. Literature Review (2) - Image-to-image translation SL + GAN A mask with high predicted SUV values (>2.5) (???)
  • 38. Literature Review (2) - Image-to-image translation input CT ground-truth PET generated PET
  • 39. • MRI => CT image translation Literature Review (2) - Image-to-image translation
  • 40. Literature Review (2) - Image-to-image translation Supervised training w/ real data Blurry More realistic
  • 41. Literature Review (2) - Image-to-image translation Pelvic datasetBrain dataset Best mean absolute error & peak signal-to-noise ratio
  • 42. • GAN research가 있지만, 아직 결과물의 수준이나 양 이 부족 • 그럼 우리는 무엇을 해볼 수 있을까? Our Research
  • 43. • Medical image generation with GAN Our Research Noise
  • 44. • Applications? • Unsupervised or semi-supervised training • Progression forecast & Visualization Our Research
  • 45. • Medical image generation from scratch? • Structure + Dynamics + Variations + … How to Tackle
  • 46. • Medical image generation from scratch? • Structure + Dynamics + Variations + … • Might be too difficult How to Tackle (GAN은 global structure / counting 같은 것에서 특히 약함)
  • 47. • (First step) image generation w/ structural hints • Structure + Dynamics + Variations + … • Image-to-image translation과 유사 How to Tackle
  • 48. • Gaze estimation task • Model-based synthetic data => GAN => Realistic data How to Tackle
  • 49. How to Tackle Self-regularization Visual Turing test => 51.7% acc. State of the art w/o label real data
  • 50. How to Tackle Self-regularization (minimizing feature dist.) Local adversarial loss (=PatchGAN) History of refined images History 안쓰면 artifacts 발생
  • 51. • Possible hints? • Model-based synthetic data • Normal image + conditioning • Normal image + synthetic or expert-guided label How to Tackle Synthetic or guided label Generator
  • 52. First Trial Generator Discriminator Real normal Real patient Image + label (for controllability) Generated patient Our target - synthetic or supervised label - Matching image-segmentation pair? - Realistic image?
  • 53. First Trial Generator Discriminator Image + label (for controllability) PatchGAN DCGAN or MAD-GAN Real normal Real patient Image matching Generated patient
  • 54. • Typical GAN issues (training, …) • Sample quality? (feat. M.D. researchers) • How to evaluate generated samples • Practical effectiveness? • Unsupervised (or semi-supervised) segmentation/classification in medical domain Have far to go
  • 55. • One more thing…
  • 56. Visualization of disease progression Identity Age
  • 57. • GAN: an interesting & effective way to generate data • GAN in Medical Imaging? => 아직 초기 단계 • “Medical Image Generation with GAN” Conclusion