research-article

Path replay backpropagation: differentiating light paths using constant memory and linear time

Authors:

Sébastien Speierer,

Wenzel JakobAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 40, Issue 4

Article No.: 108, Pages 1 - 14

https://doi.org/10.1145/3450626.3459804

Published: 19 July 2021 Publication History

Abstract

Differentiable physically-based rendering has become an indispensable tool for solving inverse problems involving light. Most applications in this area jointly optimize a large set of scene parameters to minimize an objective function, in which case reverse-mode differentiation is the method of choice for obtaining parameter gradients.

However, existing techniques that perform the necessary differentiation step suffer from either statistical bias or a prohibitive cost in terms of memory and computation time. For example, standard techniques for automatic differentiation based on program transformation or Wengert tapes lead to impracticably large memory usage when applied to physically-based rendering algorithms. A recently proposed adjoint method by Nimier-David et al. [2020] reduces this to a constant memory footprint, but the computation time for unbiased gradient estimates then becomes quadratic in the number of scattering events along a light path. This is problematic when the scene contains highly scattering materials like participating media.

In this paper, we propose a new unbiased backpropagation algorithm for rendering that only requires constant memory, and whose computation time is linear in the number of scattering events (i.e., just like path tracing). Our approach builds on the invertibility of the local Jacobian at scattering interactions to recover the various quantities needed for reverse-mode differentiation. Our method also extends to specular materials such as smooth dielectrics and conductors that cannot be handled by prior work.

Supplementary Material

VTT File (3450626.3459804.vtt)

Download
28.62 KB

ZIP File (a108-vicini.zip)

a108-vicini.zip

Download
425.38 KB

MP4 File (3450626.3459804.mp4)

Presentation.

Download
249.96 MB

References

[1]

Dejan Azinović, Tzu-Mao Li, Anton Kaplanyan, and Matthias Nießner. 2019. Inverse Path Tracing for Joint Material and Lighting Estimation. In Proceedings of Computer Vision and Pattern Recognition (CVPR), IEEE.

[2]

Sai Bangaru, Tzu-Mao Li, and Frédo Durand. 2020. Unbiased Warped-Area Sampling for Differentiable Rendering. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 39, 6 (2020), 245:1--245:18.

[3]

Chengqian Che, Fujun Luan, Shuang Zhao, Kavita Bala, and Ioannis Gkioulekas. 2020. Towards Learning-based Inverse Subsurface Scattering. In 2020 IEEE International Conference on Computational Photography (ICCP). IEEE, 1--12.

[4]

Min Chen and James Arvo. 2000. Theory and Application of Specular Path Perturbation. ACM Trans. Graph. 19, 4 (Oct. 2000), 246--278.

Digital Library

[5]

Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. 2018. Neural ordinary differential equations. In Advances in neural information processing systems. 6571--6583.

[6]

Luc Devroye. 1986. Non-Uniform Random Variate Generation. Springer-Verlag.

[7]

Laurent Dinh, David Krueger, and Yoshua Bengio. 2015. NICE: Non-linear Independent Components Estimation. In 3rd International Conference on Learning Representations (ICLR), San Diego, CA, USA, Workshop Track Proceedings. arXiv:1410.8516

[8]

Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. 2016. Density estimation using Real NVP. arXiv:1605.08803

[9]

Mathieu Galtier, Stéphane Blanco, Cyril Caliot, Christophe Coustet, Jérémi Dauchet, Mouna El Hafi, Vincent Eymet, Richard Fournier, Jacques Gautrais, Anaïs Khuong, et al. 2013. Integral formulation of null-collision Monte Carlo algorithms. Journal of Quantitative Spectroscopy and Radiative Transfer 125 (2013).

[10]

Ioannis Gkioulekas, Anat Levin, and Todd Zickler. 2016. An evaluation of computational imaging techniques for heterogeneous inverse scattering. In European Conference on Computer Vision. Springer, 685--701.

[11]

Ioannis Gkioulekas, Shuang Zhao, Kavita Bala, Todd Zickler, and Anat Levin. 2013. Inverse Volume Rendering with Material Dictionaries. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 32, 6, Article 162 (Nov. 2013).

[12]

Aidan N. Gomez, Mengye Ren, Raquel Urtasun, and Roger B. Grosse. 2017. The Reversible Residual Network: Backpropagation without Storing Activations. In NIPS.

[13]

Andreas Griewank and Andrea Walther. 2000. Algorithm 799: revolve: an implementation of checkpointing for the reverse or adjoint mode of computational differentiation. ACM Transactions on Mathematical Software (TOMS) 26, 1 (2000), 19--45.

Digital Library

[14]

Andreas Griewank and Andrea Walther. 2008. Evaluating derivatives: principles and techniques of algorithmic differentiation. Vol. 105. SIAM.

[15]

Laurent Hascoet and Valérie Pascual. 2013. The Tapenade automatic differentiation tool: Principles, model, and specification. ACM Transactions on Mathematical Software (TOMS) 39, 3 (2013), 1--43.

Digital Library

[16]

Carole K. Hayakawa, Jerome Spanier, Frédéric Bevilacqua, Andrew K. Dunn, Joon S. You, Bruce J. Tromberg, and Vasan Venugopalan. 2001. Perturbation Monte Carlo methods to solve inverse photon migration problems in heterogeneous tissues. Opt. Lett. 26, 17 (Sep 2001), 1335--1337.

[17]

Shayan Hoshyari, Hongyi Xu, Espen Knoop, Stelian Coros, and Moritz Bächer. 2019. Vibration-minimizing motion retargeting for robotic characters. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--14.

Digital Library

[18]

Binh-Son Hua, Adrien Gruson, Victor Petitjean, Matthias Zwicker, Derek Nowrouzezahrai, Elmar Eisemann, and Toshiya Hachisuka. 2019. A Survey on Gradient-Domain Rendering. Computer Graphics Forum 38, 2 (2019), 455--472.

[19]

Homan Igehy. 1999. Tracing Ray Differentials. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 99). ACM Press/Addison-Wesley Publishing Co., USA, 179--186.

Digital Library

[20]

Wenzel Jakob and Steve Marschner. 2012. Manifold Exploration: A Markov Chain Monte Carlo Technique for Rendering Scenes with Difficult Specular Transport. ACM Trans. Graph. 31, 4 (July 2012).

Digital Library

[21]

James T. Kajiya. 1986. The Rendering Equation. In Proc. of Computer Graphics and Interactive Techniques (SIGGRAPH '86). ACM, New York, NY, USA, 143--150.

Digital Library

[22]

Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. 2018. Neural 3D Mesh Renderer. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]

Pramook Khungurn, Daniel Schroeder, Shuang Zhao, Kavita Bala, and Steve Marschner. 2015. Matching Real Fabrics with Micro-Appearance Models. ACM Trans. Graph. 35, 1, Article 1 (Dec. 2015), 26 pages.

Digital Library

[24]

Peter Kutz, Ralf Habel, Yining Karl Li, and Jan Novák. 2017. Spectral and Decomposition Tracking for Rendering Heterogeneous Volumes. ACM Trans. Graph. (Proc. SIGGRAPH) 36, 4, Article 111 (2017), 111:1--111:16 pages.

Digital Library

[25]

Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. 2020. Modular Primitives for High-Performance Differentiable Rendering. ACM Transactions on Graphics 39, 6 (2020).

Digital Library

[26]

Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable Monte Carlo Ray Tracing through Edge Sampling. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 37, 6 (2018), 222:1--222:11.

[27]

Seppo Linnainmaa. 1976. Taylor expansion of the accumulated rounding error. BIT Numerical Mathematics 16, 2 (1976), 146--160.

Digital Library

[28]

Shichen Liu, Weikai Chen, Tianye Li, and Hao Li. 2019. Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction. CoRR (2019). arXiv:1901.05567

[29]

Matthew M Loper and Michael J Black. 2014. OpenDR: An approximate differentiable renderer. In European Conference on Computer Vision. Springer.

[30]

Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob. 2019. Reparameterizing Discontinuous Integrands for Differentiable Rendering. ACM Transactions on Graphics 38, 6 (Dec. 2019).

Digital Library

[31]

Iván Lux and Lázló Koblinger. 1990. Monte Carlo Particle Transport Methods: Neutron and Photon Calculations. CRC Press, Boston.

[32]

Jiahui Lyu, Bojian Wu, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. 2020. Differentiable Refraction-Tracing for Mesh Reconstruction of Transparent Objects. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 39, 6 (2020), 195:1--195:13.

[33]

Antoine McNamara, Adrien Treuille, Zoran Popović, and Jos Stam. 2004. Fluid control using the adjoint method. In ACM Transactions On Graphics (TOG), Vol. 23. ACM, 449--456.

Digital Library

[34]

Don Mitchell and Pat Hanrahan. 1992. Illumination from curved reflectors. In Proceedings of the 19th annual conference on Computer graphics and interactive techniques. 283--291.

Digital Library

[35]

Merlin Nimier-David, Sébastien Speierer, Benoît Ruiz, and Wenzel Jakob. 2020. Radiative Backpropagation: An Adjoint Method for Lightning-Fast Differentiable Rendering. ACM Trans. Graph. (Proc. SIGGRAPH) 39, 4, Article 146 (July 2020), 15 pages.

Digital Library

[36]

Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: A Retargetable Forward and Inverse Renderer. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38, 6 (Nov. 2019), 17 pages.

Digital Library

[37]

Jan Novák, Iliyan Georgiev, Johannes Hanika, Jaroslav Křivánek, and Wojciech Jarosz. 2018. Monte Carlo Methods for Physically Based Volume Rendering. In ACM SIGGRAPH 2018 Courses (Vancouver, British Columbia, Canada) (SIGGRAPH '18). ACM, New York, NY, USA, 14:1--14:1.

[38]

Jan Novák, Andrew Selle, and Wojciech Jarosz. 2014. Residual Ratio Tracking for Estimating Attenuation in Participating Media. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia) 33, 6 (Nov. 2014).

Digital Library

[39]

Steven G. Parker, James Bigler, Andreas Dietrich, Heiko Friedrich, Jared Hoberock, David Luebke, David McAllister, Morgan McGuire, Keith Morley, Austin Robison, and Martin Stich. 2010. OptiX: A General Purpose Ray Tracing Engine. ACM Trans. Graph. (Proc. SIGGRAPH), Article 66 (2010), 13 pages.

[40]

Felix Petersen, Amit H. Bermano, Oliver Deussen, and Daniel Cohen-Or. 2019. Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer. CoRR (2019). arXiv:1903.11149

[41]

Lev Semenovich Pontryagin. 1962. Mathematical theory of optimal processes. CRC Press.

[42]

Ravi Ramamoorthi, Dhruv Mahajan, and Peter Belhumeur. 2007. A first-order analysis of lighting, shading, and shadows. ACM Transactions on Graphics (TOG) 26, 1 (2007).

Digital Library

[43]

Helge Rhodin, Nadia Robertini, Christian Richardt, Hans-Peter Seidel, and Christian Theobalt. 2015. A Versatile Scene Model with Differentiable Visibility Applied to Generative Pose Estimation. In Proceedings of ICCV 2015.

Digital Library

[44]

David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. 1986. Learning Representations by Back-propagating Errors. Nature 323, 6088 (1986), 533--536. http://www.nature.com/articles/323533a0

[45]

NH Shimada and Toshiya Hachisuka. 2020. Quantum Coin Method for Numerical Integration. In Computer Graphics Forum. Wiley Online Library.

[46]

Jeffrey Mark Siskind and Barak A. Pearlmutter. 2008. Nesting forward-mode AD in a functional framework. Higher-Order and Symbolic Computation 21, 4 (2008), 361--376.

Digital Library

[47]

JM Tregan, S Blanco, J Dauchet, M Hafi, R Fournier, L Ibarrart, P Lapeyre, and N Villefranque. 2019. Convergence issues in derivatives of Monte Carlo null-collision integral formulations: a solution. (2019). arXiv:1903.06508

[48]

Eric Veach and Leonidas Guibas. 1995. Bidirectional estimators for light transport. In Photorealistic Rendering Techniques. Springer, 145--167.

[49]

Greg Ward and Paul Heckbert. 1992. Irradiance gradients. Technical Report. Lawrence Berkeley Lab., CA (United States); Ecole Polytechnique Federale, Lausanne (Switzerland); Technische Hogeschool Delft (Netherlands). Dept. of Technical Mathematics and Informatics.

[50]

E. R. Woodcock, T. Murphy, P. J. Hemmings, and T. C. Longworth. 1965. Techniques used in the GEM code for Monte Carlo neutronics calculations in reactors and other systems of complex geometry. Applications of Computing Methods to Reactor Problems (1965).

[51]

Tizian Zeltner, Iliyan Georgiev, and Wenzel Jakob. 2020. Specular Manifold Sampling for Rendering High-Frequency Caustics and Glints. Transactions on Graphics (Proceedings of SIGGRAPH) 39, 4 (July 2020).

Digital Library

[52]

Tizian Zeltner, Sébastien Speierer, Iliyan Georgiev, and Wenzel Jakob. 2021. Monte Carlo Estimators for Differential Light Transport. Transactions on Graphics (Proceedings of SIGGRAPH) 40, 4 (Aug. 2021).

Digital Library

[53]

Cheng Zhang, Bailey Miller, Kai Yan, Ioannis Gkioulekas, and Shuang Zhao. 2020. Path-Space Differentiable Rendering. ACM Trans. Graph. (Proc. SIGGRAPH) 39, 4, Article 143 (July 2020), 19 pages.

Digital Library

[54]

Cheng Zhang, Lifan Wu, Changxi Zheng, Ioannis Gkioulekas, Ravi Ramamoorthi, and Shuang Zhao. 2019. A Differential Theory of Radiative Transfer. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38, 6, Article 227 (Nov. 2019), 16 pages.

[55]

Shaung Zhao, Lifan Wu, Frédo Durand, and Ravi Ramamoorthi. 2016. Downsampling Scattering Parameters for Rendering Anisotropic Media. ACM Transactions on Graphics 35, 6 (2016).

Digital Library

Cited By

Yilmazer EVicini DJakob W(2024)Solving Inverse PDE Problems using Grid-Free Monte Carlo EstimatorsACM Transactions on Graphics10.1145/368799043:6(1-18)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687990
Xing JLi ZLuan FXu K(2024)Differentiable Photon Mapping using Generalized Path GradientsACM Transactions on Graphics10.1145/368795843:6(1-15)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687958
Nicolet BWechsler FMadrid-Wolff JMoser CJakob W(2024)Inverse Rendering for Tomographic Volumetric Additive ManufacturingACM Transactions on Graphics10.1145/368792443:6(1-17)Online publication date: 19-Nov-2024
https://doi.org/10.1145/3687924
Show More Cited By

Index Terms

Path replay backpropagation: differentiating light paths using constant memory and linear time
1. Computing methodologies
  1. Computer graphics
    1. Rendering

Recommendations

Reparameterizing discontinuous integrands for differentiable rendering

Differentiable rendering has recently opened the door to a number of challenging inverse problems involving photorealistic images, such as computational material design and scattering-aware reconstruction of geometry and materials from photographs. ...
Unbiased inverse volume rendering with differential trackers

Volumetric representations are popular in inverse rendering because they have a simple parameterization, are smoothly varying, and transparently handle topology changes. However, incorporating the full volumetric transport of light is costly and ...
Differentiable Heightfield Path Tracing with Accelerated Discontinuities
SIGGRAPH '23: ACM SIGGRAPH 2023 Conference Proceedings

We investigate the problem of accelerating a physically-based differentiable renderer for heightfields based on path tracing with global illumination. On a heightfield with 1 million vertices (1024 × 1024 resolution), our differentiable renderer ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 40, Issue 4

August 2021

2170 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3450626

Editor:
Sylvain Paris
Adobe Inc.

Issue’s Table of Contents

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 July 2021

Published in TOG Volume 40, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Swiss National Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

62
Total Citations
View Citations
678
Total Downloads

Downloads (Last 12 months)173
Downloads (Last 6 weeks)31

Reflects downloads up to 23 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yilmazer EVicini DJakob W(2024)Solving Inverse PDE Problems using Grid-Free Monte Carlo EstimatorsACM Transactions on Graphics10.1145/368799043:6(1-18)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687990
Xing JLi ZLuan FXu K(2024)Differentiable Photon Mapping using Generalized Path GradientsACM Transactions on Graphics10.1145/368795843:6(1-15)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687958
Nicolet BWechsler FMadrid-Wolff JMoser CJakob W(2024)Inverse Rendering for Tomographic Volumetric Additive ManufacturingACM Transactions on Graphics10.1145/368792443:6(1-17)Online publication date: 19-Nov-2024
https://doi.org/10.1145/3687924
Chen HMiller BGkioulekas I(2024)3D Reconstruction with Fast Dipole SumsACM Transactions on Graphics10.1145/368791443:6(1-19)Online publication date: 19-Nov-2024
https://doi.org/10.1145/3687914
Miller BSawhney RCrane KGkioulekas I(2024)Differential Walk on SpheresACM Transactions on Graphics10.1145/368791343:6(1-18)Online publication date: 19-Nov-2024
https://doi.org/10.1145/3687913
Deng XWu LWalter BRamamoorthi Rd'Eon EMarschner SWeidlich A(2024)Reconstructing translucent thin objects from photosSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687572(1-11)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3680528.3687572
Lipp LHahn DEcormier-Nocca PRist FWimmer M(2024)View-Independent Adjoint Light Tracing for Lighting Design OptimizationACM Transactions on Graphics10.1145/366218043:3(1-16)Online publication date: 22-May-2024
https://dl.acm.org/doi/10.1145/3662180
Gu JBack JYoon SMoon B(2024)Target-Aware Image Denoising for Inverse Monte Carlo RenderingACM Transactions on Graphics10.1145/365818243:4(1-11)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658182
Fan ZShi PGuo MFu RGuo YGuo J(2024)Conditional Mixture Path Guiding for Differentiable RenderingACM Transactions on Graphics10.1145/365813343:4(1-11)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658133
Seyb Dd'Eon EBitterli BJarosz W(2024)From microfacets to participating media: A unified theory of light transport with stochastic geometryACM Transactions on Graphics10.1145/365812143:4(1-17)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658121
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents