Abstract
We propose a novel generative adversarial architecture to generate realistic smoke sequences. Physically based smoke simulation methods are difficult to match with real-captured data since smoke is vulnerable to disturbance. In our work, we design a generator that takes into account the temporal movement of smoke as well as detailed structures. With the help of convolutional neural networks and long short-term memory-based autoencoder, our generator can predict the future frames using temporal information while preserving details. We use generative adversarial networks to train the model on both simulated and real-captured data and propose a combined loss function that reflects both the physical laws and the data distributions. We also demonstrate a multi-phase training strategy that significantly speeds up convergence and increases stability of training on real-captured data. To test our approach, we set up experiments to capture real smoke sequences and show that our method can achieve realistic visual effects.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al.: Tensorflow: a system for large-scale machine learning. OSDI 16, 265–283 (2016)
Angelidis, A., Neyret, F.: Simulation of smoke based on vortex filament primitives. In: Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on computer animation, ACM, pp. 87–96 (2005)
Angelidis, A., Neyret, F., Singh, K., Nowrouzezahrai, D.: A controllable, fast and stable basis for vortex based smoke simulation. In: Proceedings of the 2006 ACM SIGGRAPH/Eurographics symposium on computer animation, Eurographics Association, pp. 25–32 (2006)
Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein gan. (2017). arXiv:1701.07875
Barron, J.L., Fleet, D.J., Beauchemin, S.S.: Performance of optical flow techniques. Int. J. Comput. Vis. 12(1), 43–77 (1994)
Brackbill, J., Ruppel, H.: Flip: a method for adaptively zoned, particle-in-cell calculations of fluid flows in two dimensions. J. Comput. Phys. 65(2), 314–343 (1986)
Brochu, T., Keeler, T., Bridson, R.: Linear-time smoke animation with vortex sheet meshes. In: Proceedings of the ACM SIGGRAPH/Eurographics symposium on computer animation, Eurographics Association, pp. 87–95 (2012)
Browning, M., Barnes, C., Ritter, S., Finkelstein, A.: Stylized keyframe animation of fluid simulations. In: Proceedings of the workshop on non-photorealistic animation and rendering, ACM, pp. 63–70 (2014)
Chu, M., Thuerey, N.: Data-driven synthesis of smoke flows with cnn-based feature descriptors. ACM Trans. Graph. 36(4), 69 (2017)
Digital meals Smoke atmosphere. (2012). http://digitalmeals.blogspot.com/2012/03/smoke-atmosphere.html
Farimani, A.B., Gomes, J., Pande, V.S.: Deep learning the physics of transport phenomena. (2017). arXiv:1709.02432
Fedkiw, R., Stam, J., Jensen, H.W.: Visual simulation of smoke. In: Proceedings of the 28th annual conference on computer graphics and interactive techniques, ACM, pp. 15–22 (2001)
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, vol. 27, pp. 2672–2680 (2014). http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf
Gregson, J., Krimerman, M., Hullin, M.B., Heidrich, W.: Stochastic tomography and its applications in 3d imaging of mixing fluids. ACM Trans. Graph 31(4), 52–61 (2012)
Gregson, J., Ihrke, I., Thuerey, N., Heidrich, W.: From capture to simulation: connecting forward and inverse problems in fluids. ACM Trans. Graph. 33(4), 139 (2014)
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of wasserstein gans. In: Advances in Neural Information Processing Systems, vol. 30, pp. 5769–5779 (2017)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. (2017). arXiv Preprint
Jeong, S., Solenthaler, B., Pollefeys, M., Gross, M., et al.: Data-driven fluid simulations using regression forests. ACM Trans. Graph. 34(6), 199 (2015)
Kalchbrenner, N., Oord, A., Simonyan, K., Danihelka, I., Vinyals, O., Graves, A., Kavukcuoglu, K.: Video pixel networks. (2016). arXiv:1610.00527
Kim, B., Liu, Y., Llamas, I., Rossignac, J.R.: Flowfixer: Using bfecc for fluid simulation. Technical report, Georgia Institute of Technology (2005)
Kim, B., Azevedo, V.C., Thuerey, N., Kim, T., Gross, M., Solenthaler, B.: Deep fluids: a generative network for parameterized fluid simulations. (2018). arXiv:1806.02071
Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1646–1654 (2016)
Kim, T., Thürey, N., James, D., Gross, M.: Wavelet turbulence for fluid simulation. In: ACM Trans. Graph. (2008). https://doi.org/10.1145/1360612.1360649
Kingma, D.P., Welling, M.: Auto-encoding variational bayes. (2013). arXiv:1312.6114
Koshizuka, S., Oka, Y.: Moving-particle semi-implicit method for fragmentation of incompressible fluid. Nucl. Sci. Eng. 123(3), 421–434 (1996)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in neural information processing systems, vol. 25, pp. 1097–1105 (2012)
Lee, A.X., Zhang, R., Ebert, F., Abbeel, P., Finn, C., Levine, S.: Stochastic adversarial video prediction. (2018). arXiv:1804.01523
Li, X., Liu, L., Wu, W., Liu, X., Wu, E.: Dynamic bfecc characteristic mapping method for fluid simulations. Vis. Comput. 30(6–8), 787–796 (2014)
Long, Z., Lu, Y., Ma, X., Dong, B.: Pde-net: learning pdes from data. (2017). arXiv:1710.09668
Lotter, W., Kreiman, G., Cox, D.: Unsupervised learning of visual structure using predictive generative networks. (2015). arXiv:1511.06380
Maggioni, M., Boracchi, G., Foi, A., Egiazarian, K.: Video denoising using separable 4d nonlocal spatiotemporal transforms. In: Image processing: algorithms and systems IX, International Society for Optics and Photonics, vol. 7870, p. 787003 (2015)
Maggioni, M., Boracchi, G., Foi, A., Egiazarian, K.: Video denoising, deblocking, and enhancement through separable 4-d nonlocal spatiotemporal transforms. IEEE Trans. Image Process. 21(9), 3952–3966 (2012). https://doi.org/10.1109/TIP.2012.2199324
Mathieu, M., Couprie, C., LeCun, Y.: Deep multi-scale video prediction beyond mean square error. (2015). arXiv:1511.05440. https://doi.org/10.1109/TIP.2012.2199324
McAdams, A., Sifakis, E., Teran, J.: A parallel multigrid Poisson solver for fluids simulation on large grids. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics symposium on computer animation, Eurographics Association, pp. 65–74 (2010)
Meng, Z., Weixin, S., Yinling, Q., Hanqiu, S., Jing, Q., Heng, P.A.: Vortex filaments in grids for scalable, fine smoke simulation. IEEE Comput. Graph. Appl. 35(6), 60–68 (2015)
Mercier, O., Beauchemin, C., Thuerey, N., Kim, T., Nowrouzezahrai, D.: Surface turbulence for particle-based liquid simulations. ACM Trans. Graph. 34(6), 202 (2015)
Mirza, M., Osindero, S.: Conditional generative adversarial nets. (2014). arXiv:1411.1784
Okabe, M., Anjyor, K., Onai, R.: Creating fluid animation from a single image using video database. Comput. Graph. Forum. 30(7), 1973–1982 (2011). https://doi.org/10.1111/j.1467-8659.2011.02062.x
Okabe, M., Dobashi, Y., Anjyo, K., Onai, R.: Fluid volume modeling from sparse multi-view images by appearance transfer. ACM Trans. Graph. 34(4), 93 (2015)
Pfaff, T., Thuerey, N., Gross, M.: Lagrangian vortex sheets for animating fluids. ACM Trans. Graph. 31(4), 112 (2012)
Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. (2015). arXiv:1511.06434
Ranzato, M., Szlam, A., Bruna, J., Mathieu, M., Collobert, R., Chopra, S.: Video (language) modeling: a baseline for generative models of natural videos. (2014). arXiv:1412.6604
Rasmussen, N., Nguyen, D.Q., Geiger, W., Fedkiw, R: Smoke simulation for large scale phenomena. ACM Trans. Graph. 22(3), 703–707 (2003). https://doi.org/10.1145/882262.882335
Saito, M., Matsumoto, E., Saito, S.: Temporal generative adversarial nets with singular value clipping. In: IEEE international conference on computer vision (ICCV), pp. 2830–2839 (2017)
Selle, A., Rasmussen, N., Fedkiw, R.: A vortex particle method for smoke, water and explosions. ACM Trans. Graph. 24(3), 910–914 (2005). https://doi.org/10.1145/1073204.1073282
Selle, A., Fedkiw, R., Kim, B., Liu, Y., Rossignac, J.: An unconditionally stable maccormack method. J. Sci. Comput. 35(2–3), 350–371 (2008)
Shi, X., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., Woo, W.-C.: Convolutional lstm network: a machine learning approach for precipitation nowcasting. Adv. Neural Inf. Process. Syst. 28, 802–810 (2015)
Shrivastava, A., Pfister, T., Tuzel, O., Susskind, J., Wang, W., Webb, R.: Learning from simulated and unsupervised images through adversarial training. In: The IEEE conference on computer vision and pattern recognition (CVPR), vol. 3, p. 6 (2017)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. (2014). arXiv:1409.1556
Srivastava, N., Mansimov, E., Salakhudinov, R.: Unsupervised learning of video representations using lstms. In: International conference on machine learning, pp. 843–852 (2015)
Stam, J.: Stable fluids. In: Proceedings of the 26th annual conference on computer graphics and interactive techniques, ACM Press/Addison-Wesley Publishing Co., pp. 121–128 (2015)
Thuerey, N.: Interpolations of smoke and liquid simulations. ACM Trans. Graph. 36(1), 3 (2017)
Thuong, H.: Video background HD—smoke HD—style proshow. (2014). https://www.youtube.com/watch?v=B6H34IccWks
Tompson, J., Schlachter, K., Sprechmann, P., Perlin, K.: Accelerating Eulerian fluid simulation with convolutional networks. (2016). arXiv:1607.03597
Um, K., Hu, X., Thuerey, N.: Liquid splash modeling with neural networks. Comput. Graph. Forum. 37(8), 171–182 (2018). https://doi.org/10.1111/cgf.13522
Wang, C., Wang, C., Qin, H., Zhang, T.: Video-based fluid reconstruction and its coupling with sph simulation. Vis. Comput. 33(9), 1211–1224 (2017)
Wen, J., Ma, H.: Real-time smoke simulation based on vorticity preserving lattice Boltzmann method. Vis.Comput. 35(9), 1279–1292 (2019)
Wiewel, S., Becher, M., Thuerey, N.: Latent-space physics: Towards learning the temporal evolution of fluid flow. (2018). arXiv:1802.10123
Xie, Y., Franz, E., Chu, M., Thuerey, N.: tempogan: a temporally coherent, volumetric gan for super-resolution fluid flow. (2018). arXiv:1801.09710
Yoon, J.C., Kam, H.R., Hong, J.M., Kang, S.J., Kim, C.H.: Procedural synthesis using vortex particle method for fluid simulation. Comput. Graph. Forum. 28(7), 1853–1859 (2009). https://doi.org/10.1111/j.1467-8659.2009.01563.x
Zhang, X., Bridson, R., Greif, C.: Restoring the missing vorticity in advection-projection fluid solvers. ACM Trans. Graph. 34(4), 52 (2015)
Zhu, Y., Bridson, R.: Animating sand as a fluid. ACM Trans. Graph. 24(3), 965–972 (2005)
Funding
This work was funded by National Key Basic Research Program of China (No. 2016YFB0100900) and National Natural Science Foundation of China (No. 61773231).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Supplementary material 1 (mp4 63522 KB)
Rights and permissions
About this article
Cite this article
Wen, J., Ma, H. & Luo, X. Deep generative smoke simulator: connecting simulated and real data. Vis Comput 36, 1385–1399 (2020). https://doi.org/10.1007/s00371-019-01738-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00371-019-01738-y