research-article

Projective Sampling for Differentiable Rendering of Geometry

Authors:

Nicolas Roussel,

Wenzel JakobAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 42, Issue 6

Article No.: 212, Pages 1 - 14

https://doi.org/10.1145/3618385

Published: 05 December 2023 Publication History

Abstract

Discontinuous visibility changes at object boundaries remain a persistent source of difficulty in the area of differentiable rendering. Left untreated, they bias computed gradients so severely that even basic optimization tasks fail.

Prior path-space methods addressed this bias by decoupling boundaries from the interior, allowing each part to be handled using specialized Monte Carlo sampling strategies. While conceptually powerful, the full potential of this idea remains unrealized since existing methods often fail to adequately sample the boundary proportional to its contribution.

This paper presents theoretical and algorithmic contributions. On the theoretical side, we transform the boundary derivative into a remarkably simple local integral that invites present and future developments.

Building on this result, we propose a new strategy that projects ordinary samples produced during forward rendering onto nearby boundaries. The resulting projections establish a variance-reducing guiding distribution that accelerates convergence of the subsequent differential phase.

We demonstrate the superior efficiency and versatility of our method across a variety of shape representations, including triangle meshes, implicitly defined surfaces, and cylindrical fibers based on Bézier curves.

Supplementary Material

ZIP File (papers_711s4-file4.zip)

supplemental

Download
533.51 KB

MP4 File (papers_711s4-file3.mp4)

supplemental

Download
101.67 MB

References

[1]

Sai Praveen Bangaru, Michaël Gharbi, Fujun Luan, Tzu-Mao Li, Kalyan Sunkavalli, Milos Hasan, Sai Bi, Zexiang Xu, Gilbert Bernstein, and Fredo Durand. 2022. Differentiable rendering of neural sdfs through reparameterization. In SIGGRAPH Asia 2022 Conference Papers. 1--9.

Digital Library

[2]

Sai Praveen Bangaru, Tzu-Mao Li, and Frédo Durand. 2020. Unbiased warped-area sampling for differentiable rendering. ACM Trans. Graph. 39, 6 (2020), 1--18.

Digital Library

[3]

Sai Praveen Bangaru, Jesse Michel, Kevin Mu, Gilbert Bernstein, Tzu-Mao Li, and Jonathan Ragan-Kelley. 2021. Systematically differentiating parametric discontinuities. ACM Trans. Graph. 40, 4 (2021), 1--18.

Digital Library

[4]

Thomas Booth. 2007. Unbiased Monte Carlo Estimation of the Reciprocal of an Integral. Nuclear Science and Engineering 156 (07 2007), 403--407.

[5]

David Bremer and John F. Hughes. 1998. Rapid Approximate Silhouette Rendering of Implicit Surfaces. In Proceesings of Implicit Surfaces 98.

[6]

Forrester Cole, Kyle Genova, Avneesh Sud, Daniel Vlasic, and Zhoutong Zhang. 2021. Differentiable surface rendering via non-differentiable sampling. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 6088--6097.

[7]

Andreas Griewank and Andrea Walther. 2008. Evaluating derivatives: principles and techniques of algorithmic differentiation. Vol. 105. SIAM.

[8]

Wenzel Jakob, Sébastien Speierer, Nicolas Roussel, Merlin Nimier-David, Delio Vicini, Tizian Zeltner, Baptiste Nicolet, Miguel Crespo, Vincent Leroy, and Ziyi Zhang. 2022b. Mitsuba 3 renderer. https://mitsuba-renderer.org.

[9]

Wenzel Jakob, Sébastien Speierer, Nicolas Roussel, and Delio Vicini. 2022a. Dr.Jit: A Just-In-Time Compiler for Differentiable Rendering. Transactions on Graphics (Proceedings of SIGGRAPH) 41, 4 (July 2022).

Digital Library

[10]

Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. 2018. Neural 3D Mesh Renderer. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]

Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. 2020. Modular Primitives for High-Performance Differentiable Rendering. Transactions on Graphics (Proceedings of SIGGRAPH) 39, 6 (2020).

[12]

Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable monte carlo ray tracing through edge sampling. ACM Trans. Graph. 37, 6 (2018), 1--11.

Digital Library

[13]

Tzu-Mao Li, Michal Lukáč, Michaël Gharbi, and Jonathan Ragan-Kelley. 2020. Differentiable Vector Graphics Rasterization for Editing and Learning. ACM Trans. Graph. 39, 6 (nov 2020).

Digital Library

[14]

Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. 2019. Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning. The IEEE International Conference on Computer Vision (ICCV) (Oct. 2019).

[15]

Matthew M. Loper and Michael J. Black. 2014. OpenDR: An Approximate Differentiable Renderer. In European Conference on Computer Vision (ECCV). Springer, 154--169.

[16]

Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob. 2019. Reparameterizing discontinuous integrands for differentiable rendering. ACM Trans. Graph. 38, 6 (2019), 1--14.

Digital Library

[17]

Baptiste Nicolet, Alec Jacobson, and Wenzel Jakob. 2021. Large Steps in Inverse Rendering of Geometry. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia) 40, 6 (Dec. 2021).

[18]

Merlin Nimier-David, Sébastien Speierer, Benoît Ruiz, and Wenzel Jakob. 2020. Radiative Backpropagation: An Adjoint Method for Lightning-Fast Differentiable Rendering. Transactions on Graphics (Proceedings of SIGGRAPH) 39, 4 (July 2020).

Digital Library

[19]

Marios Papas, Thomas Houit, Derek Nowrouzezahrai, Markus Gross, and Wojciech Jarosz. 2012. The Magic Lens: Refractive Steganography. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia) 31, 6 (Nov. 2012).

[20]

Felix Petersen, Bastian Goldluecke, Christian Borgelt, and Oliver Deussen. 2022. GenDR: A Generalized Differentiable Renderer. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 4002--4011.

[21]

Osborne Reynolds. 1903. The sub-mechanics of the universe. Vol. 3. University Press.

[22]

Herman Hansson Söderlund, Alex Evans, and Tomas Akenine-Möller. 2022. Ray Tracing of Signed Distance Function Grids. Journal of Computer Graphics Techniques Vol 11, 3 (2022).

[23]

T. S. Trowbridge and K. P. Reitz. 1975. Average irregularity representation of a rough surface for ray reflection. J. Opt. Soc. Am. 65, 5 (May 1975), 531--536.

[24]

Eric Veach. 1997. Robust Monte Carlo Methods for Light Transport Simulation. Ph. D. Dissertation. Stanford University.

[25]

Eric Veach and Leonidas J. Guibas. 1995. Optimally Combining Sampling Techniques for Monte Carlo Rendering. In Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques. Association for Computing Machinery.

[26]

Delio Vicini, Sébastien Speierer, and Wenzel Jakob. 2021. Path Replay Backpropagation: Differentiating Light Paths using Constant Memory and Linear Time. Transactions on Graphics (Proceedings of SIGGRAPH) 40, 4 (Aug. 2021).

Digital Library

[27]

Delio Vicini, Sébastien Speierer, and Wenzel Jakob. 2022. Differentiable Signed Distance Function Rendering. Transactions on Graphics (Proceedings of SIGGRAPH) 41, 4 (July 2022).

[28]

Kai Yan, Christoph Lassner, Brian Budge, Zhao Dong, and Shuang Zhao. 2022. Efficient estimation of boundary integrals for path-space differentiable rendering. ACM Trans. Graph. 41, 4 (2022), 1--13.

Digital Library

[29]

Yuting Yang, Connelly Barnes, Andrew Adams, and Adam Finkelstein. 2022. A Δ: autodiff for discontinuous programs-applied to shaders. ACM Trans. Graph. 41, 4 (2022), 1--24.

Digital Library

[30]

Cheng Zhang, Bailey Miller, Kan Yan, Ioannis Gkioulekas, and Shuang Zhao. 2020. Path-space differentiable rendering. ACM Transactions on Graphics 39, 4 (2020).

Digital Library

[31]

Yang Zhou, Lifan Wu, Ravi Ramamoorthi, and Ling-Qi Yan. 2021. Vectorization for Fast, Analytic, and Differentiable Visibility. ACM Transactions on Graphics 40, 3 (July 2021).

Digital Library

Cited By

Gao RQi Y(2024)A Brief Review on Differentiable Rendering: Recent Advances and ChallengesElectronics10.3390/electronics1317354613:17(3546)Online publication date: 6-Sep-2024
https://doi.org/10.3390/electronics13173546
Tran DJensen MSantafé-Gabarda PKällberg SFerrero AHannemose MFrisvad J(2024)Digitizing translucent object appearance by validating computed optical propertiesApplied Optics10.1364/AO.52197463:16(4317)Online publication date: 22-May-2024
https://doi.org/10.1364/AO.521974
Fan ZShi PGuo MFu RGuo YGuo J(2024)Conditional Mixture Path Guiding for Differentiable RenderingACM Transactions on Graphics10.1145/365813343:4(1-11)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658133
Show More Cited By

Index Terms

Projective Sampling for Differentiable Rendering of Geometry
1. Computing methodologies
  1. Computer graphics
    1. Rendering

Recommendations

Differentiable Rendering of Parametric Geometry

We propose an efficient method for differentiable rendering of parametric surfaces and curves, which enables their use in inverse graphics problems. Our central observation is that a representative triangle mesh can be extracted from a continuous ...
Unbiased warped-area sampling for differentiable rendering

Differentiable rendering computes derivatives of the light transport equation with respect to arbitrary 3D scene parameters, and enables various applications in inverse rendering and machine learning. We present an unbiased and efficient differentiable ...
Antithetic sampling for Monte Carlo differentiable rendering

Stochastic sampling of light transport paths is key to Monte Carlo forward rendering, and previous studies have led to mature techniques capable of drawing high-contribution light paths in complex scenes. These sampling techniques have also been applied ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 42, Issue 6

December 2023

1565 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3632123

Issue’s Table of Contents

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 December 2023

Published in TOG Volume 42, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
336
Total Downloads

Downloads (Last 12 months)336
Downloads (Last 6 weeks)37

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Gao RQi Y(2024)A Brief Review on Differentiable Rendering: Recent Advances and ChallengesElectronics10.3390/electronics1317354613:17(3546)Online publication date: 6-Sep-2024
https://doi.org/10.3390/electronics13173546
Tran DJensen MSantafé-Gabarda PKällberg SFerrero AHannemose MFrisvad J(2024)Digitizing translucent object appearance by validating computed optical propertiesApplied Optics10.1364/AO.52197463:16(4317)Online publication date: 22-May-2024
https://doi.org/10.1364/AO.521974
Fan ZShi PGuo MFu RGuo YGuo J(2024)Conditional Mixture Path Guiding for Differentiable RenderingACM Transactions on Graphics10.1145/365813343:4(1-11)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658133
Zhou SChang YMukai NSanto HOkura FMatsushita YZhao S(2024)Path-Space Differentiable Rendering of Implicit SurfacesACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657473(1-11)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657473
Tojo KShamir ABickel BUmetani N(2024)Fabricable 3D Wire ArtACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657453(1-11)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657453
Zhao QLong PZhang QQin DLiang HZhang LZhang YYu JXu L(2024)Media2Face: Co-speech Facial Animation Generation With Multi-Modality GuidanceACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657413(1-13)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657413
Hu JHui KLiu ZZhang HFu C(2024)CNS-Edit: 3D Shape Editing via Coupled Neural Shape OptimizationACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657412(1-12)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657412
Teh AGkioulekas IO'Toole M(2024)Aperture-Aware Lens DesignACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657398(1-10)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657398
Tu GXiong FLiang BXu RHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)A Persona-Infused Cross-Task Graph Network for Multimodal Emotion Recognition with Emotion Shift Detection in ConversationsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657944(2266-2270)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657944
Abdallah AKasem MAbdalla MMahmoud MElkasaby MElbendary YJatowt AHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)ArabicaQA: A Comprehensive Dataset for Arabic Question AnsweringProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657889(2049-2059)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657889
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents