research-article

Path-space differentiable rendering

Authors:

Ioannis Gkioulekas,

Shuang ZhaoAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 39, Issue 4

Article No.: 143, Pages 143:1 - 143:19

https://doi.org/10.1145/3386569.3392383

Published: 12 August 2020 Publication History

Abstract

Physics-based differentiable rendering, the estimation of derivatives of radiometric measures with respect to arbitrary scene parameters, has a diverse array of applications from solving analysis-by-synthesis problems to training machine learning pipelines incorporating forward rendering processes. Unfortunately, general-purpose differentiable rendering remains challenging due to the lack of efficient estimators as well as the need to identify and handle complex discontinuities such as visibility boundaries.

In this paper, we show how path integrals can be differentiated with respect to arbitrary differentiable changes of a scene. We provide a detailed theoretical analysis of this process and establish new differentiable rendering formulations based on the resulting differential path integrals. Our path-space differentiable rendering formulation allows the design of new Monte Carlo estimators that offer significantly better efficiency than state-of-the-art methods in handling complex geometric discontinuities and light transport phenomena such as caustics.

We validate our method by comparing our derivative estimates to those generated using the finite-difference method. To demonstrate the effectiveness of our technique, we compare inverse-rendering performance with a few state-of-the-art differentiable rendering methods.

Supplementary Material

VTT File (3386569.3392383.vtt)

Download
28.33 KB

ZIP File (a143-zhang.zip)

Supplemental files.

Download
220.61 MB

MP4 File (3386569.3392383.mp4)

Presentation video

Download
580.25 MB

References

[1]

Luke Anderson, Tzu-Mao Li, Jaakko Lehtinen, and Frédo Durand. 2017. Aether: an embedded domain specific sampling language for Monte Carlo rendering. ACM

[2]

Trans. Graph. 36, 4 (2017), 99:1--99:16.

[3]

James Arvo. 1994. The Irradiance Jacobian for partially occluded polyhedral sources. In SIGGRAPH '94. 343--350.

Digital Library

[4]

Dejan Azinovic, Tzu-Mao Li, Anton Kaplanyan, and Matthias Niessner. 2019. Inverse Path Tracing for Joint Material and Lighting Estimation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]

Paolo Cermelli, Eliot Fried, and Morton E Gurtin. 2005. Transport relations for surface integrals arising in the formulation of balance laws for evolving fluid interfaces. Journal of Fluid Mechanics 544 (2005), 339--351.

[6]

Subrahmanyan Chandrasekhar. 1960. Radiative Transfer. Courier Corporation.

[7]

Chengqian Che, Fujun Luan, Shuang Zhao, Kavita Bala, and Ioannis Gkioulekas. 2018. Inverse transport networks. arXiv preprint arXiv:1809.10820 (2018).

[8]

Min Chen and James Arvo. 2000. Theory and application of specular path perturbation. ACM Trans. Graph. 19, 4 (2000), 246--278.

Digital Library

[9]

Ioannis Gkioulekas, Anat Levin, and Todd Zickler. 2016. An Evaluation of Computational Imaging Techniques for Heterogeneous Inverse Scattering. In Computer Vision - ECCV 2016. Springer International Publishing, 685--701.

[10]

Ioannis Gkioulekas, Shuang Zhao, Kavita Bala, Todd Zickler, and Anat Levin. 2013. Inverse volume rendering with material dictionaries. ACM Trans. Graph. 32, 6 (2013), 162:1--162:13.

Digital Library

[11]

Andreas Griewank and Andrea Walther. 2008. Evaluating derivatives: principles and techniques of algorithmic differentiation. Vol. 105. Siam.

[12]

Pavel Grinfeld. 2013. Introduction to tensor analysis and the calculus of moving surfaces. Springer.

[13]

Morton E Gurtin. 1981. An introduction to continuum mechanics. Academic press.

[14]

Milovš Hašan and Ravi Ramamoorthi. 2013. Interactive albedo editing in path-traced volumetric materials. ACM Trans. Graph. 32, 2 (2013), 11:1--11:11.

Digital Library

[15]

Homan Igehy. 1999. Tracing ray differentials. In SIGGRAPH '99. 179--186.

Digital Library

[16]

Wenzel Jakob and Steve Marschner. 2012. Manifold Exploration: A Markov Chain Monte Carlo Technique for Rendering Scenes with Difficult Specular Transport. ACM Trans. Graph. 31, 4 (2012), 58:1--58:13.

Digital Library

[17]

Henrik Wann Jensen. 1995. Importance driven path tracing using the photon map. In Rendering Techniques' 95. Springer, 326--335.

[18]

James T. Kajiya. 1986. The Rendering Equation. SIGGRAPH Comput. Graph. 20, 4 (1986), 143--150.

Digital Library

[19]

Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. 2018. Neural 3D mesh renderer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3907--3916.

[20]

Csaba Kelemen, László Szirmay-Kalos, György Antal, and Ferenc Csonka. 2002. A simple and robust mutation strategy for the Metropolis light transport algorithm. In Computer Graphics Forum, Vol. 21. Wiley Online Library, 531--540.

[21]

Pramook Khungurn, Daniel Schroeder, Shuang Zhao, Kavita Bala, and Steve Marschner. 2015. Matching real fabrics with micro-appearance models. ACM Trans. Graph. 35, 1 (2015), 1:1--1:26.

Digital Library

[22]

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).

[23]

Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018a. Differentiable Monte Carlo ray tracing through edge sampling. ACM Trans. Graph. 37, 6 (2018), 222:1--222:11.

Digital Library

[24]

Tzu-Mao Li, Jaakko Lehtinen, Ravi Ramamoorthi, Wenzel Jakob, and Frédo Durand. 2015. Anisotropic Gaussian mutations for Metropolis light transport through Hessian-Hamiltonian dynamics. ACM Trans. Graph. 34, 6 (2015), 209:1--209:13.

Digital Library

[25]

Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. 2018b. Learning to reconstruct shape and spatially-varying reflectance from a single image. ACM Trans. Graph. 37, 6 (2018), 269:1--269:11.

Digital Library

[26]

Matthew M Loper and Michael J Black. 2014. OpenDR: an approximate differentiable renderer. In European Conference on Computer Vision. Springer, 154--169.

[27]

Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob. 2019. Reparameterizing discontinuous integrands for differentiable rendering. ACM Trans. Graph. 38, 6 (2019).

Digital Library

[28]

James L McClelland, David E Rumelhart, PDP Research Group, et al. 1986. Parallel distributed processing. Explorations in the microstructure of cognition 2 (1986), 216--271.

[29]

Abhimitra Meka, Maxim Maximov, Michael Zollhoefer, Avishek Chatterjee, Hans-Peter Seidel, Christian Richardt, and Christian Theobalt. 2018. Lime: Live intrinsic material estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6315--6324.

[30]

Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: a retargetable forward and inverse renderer. ACM Transactions on Graphics (TOG) 38, 6 (2019), 203.

Digital Library

[31]

Mark Pauly, Thomas Kollig, and Alexander Keller. 2000. Metropolis light transport for participating media. In Rendering Techniques 2000. Springer, 11--22.

[32]

Ravi Ramamoorthi, Dhruv Mahajan, and Peter Belhumeur. 2007. A first-order analysis of lighting, shading, and shadows. ACM Trans. Graph. 26, 1 (2007), 2:1--2:21.

Digital Library

[33]

Osborne Reynolds. 1903. Papers on mechanical and physical subjects: the sub-mechanics of the universe. Vol. 3. The University Press.

[34]

Soumyadip Sengupta, Angjoo Kanazawa, Carlos D Castillo, and David W Jacobs. 2018. SfSNet: Learning shape, reflectance and illuminance of faces in the wild'. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6296--6305.

[35]

Denis Sumin, Tobias Rittig, Vahid Babaei, Thomas Nindel, Alexander Wilkie, Piotr Didyk, Bernd Bickel, Jaroslav Křivánek, Karol Myszkowski, and Tim Weyrich. 2019. Geometry-aware scattering compensation for 3D printing. ACM Trans. Graph. 38, 4 (2019), 111:1--111:14.

Digital Library

[36]

Chia-Yin Tsai, Aswin C. Sankaranarayanan, and Ioannis Gkioulekas. 2019. Beyond volumetric albedo---a surface optimization framework for non-line-of-sight imaging. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]

Eric Veach. 1997. Robust Monte Carlo methods for light transport simulation. Vol. 1610. Stanford University PhD thesis.

Digital Library

[38]

Eric Veach and Leonidas Guibas. 1995. Bidirectional estimators for light transport. In Photorealistic Rendering Techniques. Springer, 145--167.

[39]

Eric Veach and Leonidas J. Guibas. 1997. Metropolis Light Transport. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '97). ACM Press/Addison-Wesley Publishing Co., 65--76.

[40]

Zdravko Velinov, Marios Papas, Derek Bradley, Paulo Gotardo, Parsa Mirdehghan, Steve Marschner, Jan Novák, and Thabo Beeler. 2018. Appearance Capture and Modeling of Human Teeth. ACM Trans. Graph. 37, 6 (2018), 207:1--207:13.

Digital Library

[41]

Robert Edwin Wengert. 1964. A simple automatic derivative evaluation program. Commun. ACM 7, 8 (1964), 463--464.

Digital Library

[42]

Andrew P. Witkin and Paul S. Heckbert. 1994. Using particles to sample and control implicit surfaces. In Proceedings of the 21st Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '94). ACM, 269--277.

[43]

Jiajun Wu, Joshua B Tenenbaum, and Pushmeet Kohli. 2017. Neural scene de-rendering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 699--707.

[44]

Cheng Zhang, Lifan Wu, Changxi Zheng, Ioannis Gkioulekas, Ravi Ramamoorthi, and Shaung Zhao. 2019. A differential theory of radiative transfer. ACM Trans. Graph. 38, 6 (2019), 227:1--227:16.

Digital Library

[45]

Shuang Zhao, Lifan Wu, Frédo Durand, and Ravi Ramamoorthi. 2016. Downsampling scattering parameters for rendering anisotropic media. ACM Trans. Graph. 35, 6 (2016), 166:1--166:11.

Digital Library

[46]

Changxi Zheng and Doug L. James. 2012. Energy-based self-collision culling for arbitrary mesh deformations. ACM Trans. Graph. 31, 4 (2012), 98:1--98:12.

Digital Library

Cited By

Lai ZZhang CZhang YYuan DSekar VYu MSeneviratne AVeitch D(2024)DEMO: Real-Time Simulation of Wireless Signal Propagation for Dynamic Environment Through GPU-Based Ray TracingProceedings of the ACM SIGCOMM 2024 Conference: Posters and Demos10.1145/3672202.3673725(98-100)Online publication date: 4-Aug-2024
https://dl.acm.org/doi/10.1145/3672202.3673725
Lipp LHahn DEcormier-Nocca PRist FWimmer M(2024)View-Independent Adjoint Light Tracing for Lighting Design OptimizationACM Transactions on Graphics10.1145/366218043:3(1-16)Online publication date: 3-May-2024
https://dl.acm.org/doi/10.1145/3662180
Gu JBack JYoon SMoon B(2024)Target-Aware Image Denoising for Inverse Monte Carlo RenderingACM Transactions on Graphics10.1145/365818243:4(1-11)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658182
Show More Cited By

Index Terms

Path-space differentiable rendering
1. Computing methodologies
  1. Computer graphics
    1. Rendering

Recommendations

Path-space differentiable rendering of participating media

Physics-based differentiable rendering---which focuses on estimating derivatives of radiometric detector responses with respect to arbitrary scene parameters---has a diverse array of applications from solving analysis-by-synthesis problems to training ...
Reparameterizing discontinuous integrands for differentiable rendering

Differentiable rendering has recently opened the door to a number of challenging inverse problems involving photorealistic images, such as computational material design and scattering-aware reconstruction of geometry and materials from photographs. ...
Reconstructing Translucent Objects using Differentiable Rendering
SIGGRAPH '22: ACM SIGGRAPH 2022 Conference Proceedings

Inverse rendering is a powerful approach to modeling objects from photographs, and we extend previous techniques to handle translucent materials that exhibit subsurface scattering. Representing translucency using a heterogeneous bidirectional ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 39, Issue 4

August 2020

1732 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3386569

Editor:
Szymon Rusinkiewicz
Princeton University

Issue’s Table of Contents

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 August 2020

Published in TOG Volume 39, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

76
Total Citations
View Citations
548
Total Downloads

Downloads (Last 12 months)70
Downloads (Last 6 weeks)11

Reflects downloads up to 10 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lai ZZhang CZhang YYuan DSekar VYu MSeneviratne AVeitch D(2024)DEMO: Real-Time Simulation of Wireless Signal Propagation for Dynamic Environment Through GPU-Based Ray TracingProceedings of the ACM SIGCOMM 2024 Conference: Posters and Demos10.1145/3672202.3673725(98-100)Online publication date: 4-Aug-2024
https://dl.acm.org/doi/10.1145/3672202.3673725
Lipp LHahn DEcormier-Nocca PRist FWimmer M(2024)View-Independent Adjoint Light Tracing for Lighting Design OptimizationACM Transactions on Graphics10.1145/366218043:3(1-16)Online publication date: 3-May-2024
https://dl.acm.org/doi/10.1145/3662180
Gu JBack JYoon SMoon B(2024)Target-Aware Image Denoising for Inverse Monte Carlo RenderingACM Transactions on Graphics10.1145/365818243:4(1-11)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658182
Fan ZShi PGuo MFu RGuo YGuo J(2024)Conditional Mixture Path Guiding for Differentiable RenderingACM Transactions on Graphics10.1145/365813343:4(1-11)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658133
Deliot THeitz EBelcour L(2024)Transforming a Non-Differentiable Rasterizer into a Differentiable One with Stochastic Gradient EstimationProceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/36512987:1(1-16)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3651298
Belhe YXu BBangaru SRamamoorthi RLi T(2024)Importance Sampling BRDF DerivativesACM Transactions on Graphics10.1145/364861143:3(1-21)Online publication date: 9-Apr-2024
https://dl.acm.org/doi/10.1145/3648611
Zhou SChang YMukai NSanto HOkura FMatsushita YZhao S(2024)Path-Space Differentiable Rendering of Implicit SurfacesACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657473(1-11)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657473
Ling JYu RXu FDu CZhao S(2024)NeRF as a Non-Distant Environment Emitter in Physics-based Inverse RenderingACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657404(1-12)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657404
Teh AGkioulekas IO'Toole M(2024)Aperture-Aware Lens DesignACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657398(1-10)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657398
Xu BLi TGeorgiev IHedstrom TRamamoorthi R(2024)Residual path integrals for re‐renderingComputer Graphics Forum10.1111/cgf.1515243:4Online publication date: 24-Jul-2024
https://doi.org/10.1111/cgf.15152
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents