research-article

NeLT: Object-Oriented Neural Light Transfer

Authors:

Chuankun Zheng,

Hujun BaoAuthors Info & Claims

ACM Transactions on Graphics, Volume 42, Issue 5

Article No.: 163, Pages 1 - 16

https://doi.org/10.1145/3596491

Published: 29 August 2023 Publication History

Abstract

This article presents object-oriented neural light transfer (NeLT), a novel neural representation of the dynamic light transportation between an object and the environment. Our method disentangles the global illumination of a scene into individual objects’ light transportation represented via neural networks, then composes them explicitly. It therefore enables flexible rendering with dynamic lighting, cameras, materials, and objects. Our rendering features various important global illumination effects, such as diffuse illumination, glossy illumination, dynamic shadowing, and indirect illumination, which completes the capability of existing neural object representation. Experiments show that NeLT does not require path tracing or shading results as input but achieves rendering quality comparable to state-of-the-art rendering frameworks, including the recent deep learning based denoisers.

Supplementary Material

TOG-22-0118-SUPP (tog-22-0118-supp.zip)

Supplementary material

Download
159.12 MB

References

[1]

Kara-Ali Aliev, Artem Sevastopolsky, Maria Kolos, Dmitry Ulyanov, and Victor Lempitsky. 2020. Neural point-based graphics. In Computer Vision—ECCV 2020. Lecture Notes in Computer Science, Vol. 12367. Springer, 696–712.

[2]

Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony Derose, and Fabrice Rousselle. 2017. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Transactions on Graphics 36, 4 (2017), Article 97, 1–14.

Digital Library

[3]

Aner Ben-Artzi, Kevin Egan, Frédo Durand, and Ravi Ramamoorthi. 2008. A precomputed polynomial representation for interactive BRDF editing with global illumination. ACM Transactions on Graphics 27, 2 (2008), 1–13.

Digital Library

[4]

Aner Ben-Artzi, Ryan Overbeck, and Ravi Ramamoorthi. 2006. Real-time BRDF editing in complex lighting. ACM Transactions on Graphics 25, 3 (2006), 945–954.

Digital Library

[5]

Adrian Blumer, Jan Novák, Ralf Habel, Derek Nowrouzezahrai, and Wojciech Jarosz. 2016. Reduced aggregate scattering operators for path tracing. Computer Graphics Forum 35 (2016), 461–473.

[6]

Chakravarty R. Alla Chaitanya, Anton S. Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder. ACM Transactions on Graphics 36, 4 (2017), Article 98, 1–12.

Digital Library

[7]

In-Young Cho, Yuchi Huo, and Sung-Eui Yoon. 2021. Weakly-supervised contrastive learning in path manifold for monte carlo image reconstruction. ACM Trans. Graph. 40, 4 (2021), 38–1.

[8]

Carsten Dachsbacher and Marc Stamminger. 2005. Reflective shadow maps. In Proceedings of the 2005 Symposium on Interactive 3D Graphics and Games. 203–231.

Digital Library

[9]

Carsten Dachsbacher, Marc Stamminger, George Drettakis, and Frédo Durand. 2007. Implicit visibility and antiradiance for interactive global illumination. ACM Transactions on Graphics 26, 3 (2007), 61–es.

Digital Library

[10]

Sayantan Datta, Derek Nowrouzezahrai, Christoph Schied, and Zhao Dong. 2022. Neural shadow mapping. In Proceedings of the ACM SIGGRAPH 2022 Conference (SIGGRAPH’22). 1–9.

Digital Library

[11]

Stavros Diolatzis, Julien Philip, and George Drettakis. 2022. Active exploration for neural global illumination of variable scenes. ACM Transactions on Graphics 41, 5 (2022), Article 171, 18 pages.

Digital Library

[12]

S. M. Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S. Morcos, Marta Garnelo, Avraham Ruderman, et al. 2018. Neural scene representation and rendering. Science 360, 6394 (2018), 1204–1210.

[13]

Hangming Fan, Rui Wang, Yuchi Huo, and Hujun Bao. 2021. Real-time Monte Carlo denoising with weight sharing kernel prediction network. Computer Graphics Forum 40 (2021), 15–27.

[14]

Michaël Gharbi, Tzu-Mao Li, Miika Aittala, Jaakko Lehtinen, and Frédo Durand. 2019. Sample-based Monte Carlo denoising using a kernel-splatting network. ACM Transactions on Graphics 38, 4 (2019), 1–12.

Digital Library

[15]

Jonathan Granskog, Fabrice Rousselle, Marios Papas, and Jan Novák. 2020. Compositional neural scene representations for shading inference. ACM Transactions on Graphics 39, 4 (2020), Article 135, 13 pages.

Digital Library

[16]

Jonathan Granskog, Till N. Schnabel, Fabrice Rousselle, and Jan Novák. 2021. Neural scene graph rendering. ACM Transactions on Graphics 40, 4 (2021), 1–11.

Digital Library

[17]

Jie Guo, Mengtian Li, Quewei Li, Yuting Qiang, Bingyang Hu, Yanwen Guo, and Ling-Qi Yan. 2019. GradNet: Unsupervised deep screened Poisson reconstruction for gradient-domain rendering. ACM Transactions on Graphics 38, 6 (2019), 1–13.

Digital Library

[18]

Michelle Guo, Alireza Fathi, Jiajun Wu, and Thomas Funkhouser. 2020. Object-centric neural scene rendering. arXiv preprint arXiv:2012.08503 (2020).

[19]

David Ha, Andrew M. Dai, and Quoc V. Le. 2017. Hyper networks. In Proceedings of the 5th International Conference on Learning Representations: Conference Track (ICLR’17).

[20]

Saeed Hadadan, Shuhong Chen, and Matthias Zwicker. 2021. Neural radiosity. ACM Transactions on Graphics 40, 6 (2021), 1–11.

Digital Library

[21]

Pedro Hermosilla, Sebastian Maisch, Tobias Ritschel, and Timo Ropinski. 2019. Deep-learning the latent space of light transport. Computer Graphics Forum 38 (2019), 207–217).

[22]

Yuchi Huo, Rui Wang, Ruzahng Zheng, Hualin Xu, Hujun Bao, and Sung-Eui Yoon. 2020. Adaptive incident radiance field sampling and reconstruction using deep reinforcement learning. ACM Transactions on Graphics 39, 1 (2020), 1–17.

Digital Library

[23]

Yuchi Huo and Sung-Eui Yoon. 2021. A survey on deep learning-based Monte Carlo denoising. Computational Visual Media 7 (2021), 169–185.

[24]

Simon Kallweit, Thomas Müller, Brian McWilliams, Markus Gross, and Jan Novák. 2017. Deep scattering: Rendering atmospheric clouds with radiance-predicting neural networks. ACM Transactions on Graphics 36, 6 (2017), 1–11.

Digital Library

[25]

Brian Karis. 2014. High-quality temporal supersampling. In Proceedings of the ACM SIGGRAPH 2014 Conference: Advances in Real-Time Rendering in Games(SIGGRAPH’14).

[26]

Jan Kautz, Jaakko Lehtinen, and Timo Aila. 2004. Hemispherical rasterization for self-shadowing of dynamic objects. In Proceedings of the 15th Eurographics Conference on Rendering Techniques (EGSR’04). 179–184.

[27]

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR’15): Conference Track.

[28]

Janne Kontkanen and Samuli Laine. 2005. Ambient occlusion fields. In Proceedings of the 2005 Symposium on Interactive 3D Graphics and Games. 41–48.

Digital Library

[29]

Anders Wang Kristensen, Tomas Akenine-Möller, and Henrik Wann Jensen. 2005. Precomputed local radiance transfer for real-time lighting design. ACM Transactions on Graphics 24, 3 (2005), 1208–1215.

Digital Library

[30]

Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. 2019. Neural volumes: Learning dynamic renderable volumes from images. ACM Transactions on Graphics 38, 4 (July 2019), Article 65, 14 pages.

Digital Library

[31]

Bradford J. Loos, Lakulish Antani, Kenny Mitchell, Derek Nowrouzezahrai, Wojciech Jarosz, and Peter-Pike Sloan. 2011. Modular radiance transfer. In Proceedings of the 2011 SIGGRAPH Asia Conference. 1–10.

Digital Library

[32]

Bradford J. Loos, Derek Nowrouzezahrai, Wojciech Jarosz, and Peter-Pike Sloan. 2012. Delta radiance transfer. In Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games. 191–196.

Digital Library

[33]

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2021. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM 65, 1 (2021), 99–106.

[34]

Thomas Müller. 2021. Tiny CUDA Neural Network Framework. Retrieved May 18, 2023 from https://github.com/nvlabs/tiny-cuda-nn.

[35]

Thomas Müller, Fabrice Rousselle, Jan Novák, and Alexander Keller. 2021. Real-time neural radiance caching for path tracing. ACM Transactions on Graphics 40, 4 (2021), 1–16.

Digital Library

[36]

Oliver Nalbach, Elena Arabadzhiyska, Dushyant Mehta, H.-P. Seidel, and Tobias Ritschel. 2017. Deep shading: Convolutional neural networks for screen space shading. Computer Graphics Forum 36 (2017), 65–78.

[37]

Michael Niemeyer and Andreas Geiger. 2021. GIRAFFE: Representing scenes as compositional generative neural feature fields. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’21).

[38]

Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. 2021. Neural scene graphs for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2856–2865.

[39]

Steven G. Parker, James Bigler, Andreas Dietrich, Heiko Friedrich, Jared Hoberock, David Luebke, David McAllister, et al. 2010. Optix: A general purpose ray tracing engine. ACM Transactions on Graphics 29, 4 (2010), 1–13.

Digital Library

[40]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, et al. 2019. PyTorch: An imperative style, high-performance deep learning library. In Proceedings of the 33rd International Conference on Neural Information Processing Systems (NIPS’19). 8026–8037.

[41]

Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2020. Physically Based Rendering: From Theory to Implementation (4th ed). MIT Press, Cambridge, MA. https://github.com/mmp/pbrt-v4.

[42]

Konstantinos Rematas and Vittorio Ferrari. 2020. Neural voxel renderer: Learning an accurate and controllable rendering tool. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5417–5427.

[43]

Peiran Ren, Jiaping Wang, Minmin Gong, Stephen Lin, Xin Tong, and Baining Guo. 2013. Global illumination with radiance regression functions. ACM Transactions on Graphics 32, 4 (2013), Article 130, 12 pages.

Digital Library

[44]

Tobias Ritschel, Thorsten Grosch, and Hans-Peter Seidel. 2009. Approximating dynamic global illumination in image space. In Proceedings of the 2009 Symposium on Interactive 3D Graphics and Games. 75–82.

Digital Library

[45]

Paul Sanzenbacher, Lars Mescheder, and Andreas Geiger. 2020. Learning neural light transport. arXiv preprint arXiv:2006.03427 (2020).

[46]

Perumaal Shanmugam and Okan Arikan. 2007. Hardware accelerated ambient occlusion techniques on GPUs. In Proceedings of the 2007 Symposium on Interactive 3D Graphics and Games. 73–80.

Digital Library

[47]

Peter-Pike Sloan, Jan Kautz, and John Snyder. 2002. Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. In Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’02). 527–536.

Digital Library

[48]

Xin Sun, Kun Zhou, Yanyun Chen, Stephen Lin, Jiaoying Shi, and Baining Guo. 2007. Interactive relighting with dynamic BRDFs. ACM Transactions on Graphics 26, 3 (2007), 27–es.

Digital Library

[49]

Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. 2016. Multi-view 3D models from single images with a convolutional network. In Proceedings of the European Conference on Computer Vision. 322–337.

[50]

Delio Vicini, Vladlen Koltun, and Wenzel Jakob. 2019. A learned shape-adaptive subsurface scattering model. ACM Transactions on Graphics 38, 4 (2019), 1–15.

Digital Library

[51]

Thijs Vogels, Fabrice Rousselle, Brian McWilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. 2018. Denoising with kernelprediction and asymmetric loss functions. ACM Transactions on Graphics 37, 4 (2018), 1–15.

Digital Library

[52]

Bruce Walter, Sebastian Fernandez, Adam Arbree, Kavita Bala, Michael Donikian, and Donald P. Greenberg. 2005. Lightcuts: A scalable approach to illumination. ACM Transactions on Graphics 24, 3 (2005), 1098–1107.

[53]

Bruce Walter, Stephen R. Marschner, Hongsong Li, and Kenneth E. Torrance. 2007. Microfacet models for refraction through rough surfaces. In Proceedings of the 18th Eurographics Conference on Rendering Techniques (EGSR’07). 195–206.

[54]

Hanggao Xin, Shaokun Zheng, Kun Xu, and Ling-Qi Yan. 2022. Lightweight bilateral convolutional neural networks for interactive single-bounce diffuse indirect illumination. IEEE Transactions on Visualization and Computer Graphics 28, 4 (2022), 1824–1834.

[55]

Bing Xu, Junfei Zhang, Rui Wang, Kun Xu, Yong-Liang Yang, Chuan Li, and Rui Tang. 2019. Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation. ACM Transactions on Graphics 38, 6 (2019), Article 224, 12 pages.

Digital Library

[56]

Jiaqi Yu, Yongwei Nie, Chengjiang Long, Wenjun Xu, Qing Zhang, and Guiqing Li. 2021. Monte Carlo denoising via auxiliary feature guided self-attention. ACM Transactions on Graphics 40, 6 (2021), Article 273, 13 pages.

Digital Library

[57]

Chuankun Zheng, Ruzhang Zheng, Rui Wang, Shuang Zhao, and Hujun Bao. 2021. A compact representation of measured BRDFs using neural processes. ACM Transactions on Graphics 41, 2 (2021), 1–15.

Digital Library

[58]

Kun Zhou, Yaohua Hu, Stephen Lin, Baining Guo, and Heung-Yeung Shum. 2005. Precomputed shadow fields for dynamic scenes. ACM Transactions on Graphics 24, 3 (2005), 1196–1201.

Digital Library

[59]

Junqiu Zhu, Yaoyi Bai, Zilin Xu, Steve Bako, Edgar Velázquez-Armendáriz, Lu Wang, Pradeep Sen, Miloš Hašan, and Ling-Qi Yan. 2021. Neural complex luminaires: Representation and rendering. ACM Transactions on Graphics 40, 4 (2021), 1–12.

Digital Library

Cited By

Wang ZZhu XYu JZhang TLei ZCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)S2TD-Face: Reconstruct a Detailed 3D Face with Controllable Texture from a Single SketchProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681159(6453-6462)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681159
Zheng WXia HChen RSun LShao MXia SDing ZCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Sketch3D: Style-Consistent Guidance for Sketch-to-3D GenerationProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680641(3617-3626)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680641
Petrov DGoyal PThamizharasan VKim VGadelha MAverkiou MChaudhuri SKalogerakis E(2024)GEM3D: GEnerative Medial Abstractions for 3D Shape SynthesisACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657415(1-11)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657415
Show More Cited By

Index Terms

NeLT: Object-Oriented Neural Light Transfer
1. Computing methodologies
  1. Computer graphics
    1. Rendering
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks

Recommendations

Deferred neural lighting: free-viewpoint relighting from unstructured photographs

We present deferred neural lighting, a novel method for free-viewpoint relighting from unstructured photographs of a scene captured with handheld devices. Our method leverages a scene-dependent neural rendering network for relighting a rough geometric ...
LightFormer: Light-Oriented Global Neural Rendering in Dynamic Scene

The generation of global illumination in real time has been a long-standing challenge in the graphics community, particularly in dynamic scenes with complex illumination. Recent neural rendering techniques have shown great promise by utilizing neural ...
Neural Light Transport for Relighting and View Synthesis

The light transport (LT) of a scene describes how it appears under different lighting conditions from different viewing directions, and complete knowledge of a scene’s LT enables the synthesis of novel views under arbitrary lighting. In this article, we ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 42, Issue 5

October 2023

195 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3607124

Editor:
Carol O'Sullivan
Trinity College Dublin, Ireland

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 August 2023

Online AM: 10 May 2023

Accepted: 22 April 2023

Revised: 17 April 2023

Received: 02 December 2022

Published in TOG Volume 42, Issue 5

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Key R&D Program of Zhejiang Province
NSFC
Fundamental Research Funds for the Central Universities, Zhejiang Lab
Key Research Project of Zhejiang Lab
Information Technology Center and State Key Lab of CAD&CG, Zhejiang University

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
1,772
Total Downloads

Downloads (Last 12 months)996
Downloads (Last 6 weeks)66

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang ZZhu XYu JZhang TLei ZCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)S2TD-Face: Reconstruct a Detailed 3D Face with Controllable Texture from a Single SketchProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681159(6453-6462)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681159
Zheng WXia HChen RSun LShao MXia SDing ZCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Sketch3D: Style-Consistent Guidance for Sketch-to-3D GenerationProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680641(3617-3626)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680641
Petrov DGoyal PThamizharasan VKim VGadelha MAverkiou MChaudhuri SKalogerakis E(2024)GEM3D: GEnerative Medial Abstractions for 3D Shape SynthesisACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657415(1-11)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657415
TG TFrisvad JRamamoorthi RJensen H(2024)NeuPreSS: Compact Neural Precomputed Subsurface Scattering for Distant Lighting of Heterogeneous Translucent ObjectsComputer Graphics Forum10.1111/cgf.1523443:7Online publication date: 18-Oct-2024
https://doi.org/10.1111/cgf.15234
Coomans ADominci EDöring CMueller JHladky JSteinberger M(2024)Real‐time Neural Rendering of Dynamic Light FieldsComputer Graphics Forum10.1111/cgf.1501443:2Online publication date: 23-Apr-2024
https://doi.org/10.1111/cgf.15014
Wang LZhao XSun JZhang YZhang HYu TLiu Y(2023)StyleAvatar: Real-time Photo-realistic Portrait Avatar from a Single VideoACM SIGGRAPH 2023 Conference Proceedings10.1145/3588432.3591517(1-10)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1145/3588432.3591517

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents