research-article

Open access

MCNeRF: Monte Carlo Rendering and Denoising for Real-Time NeRFs

Authors:

Kalyan Sunkavalli,

Manmohan Chandraker,

Sai BiAuthors Info & Claims

SA '23: SIGGRAPH Asia 2023 Conference Papers

Article No.: 98, Pages 1 - 11

https://doi.org/10.1145/3610548.3618221

Published: 11 December 2023 Publication History

All formats PDF

Abstract

The volume rendering step used in Neural Radiance Fields (NeRFs) produces highly photorealistic results, but is inherently slow because it evaluates an MLP at a large number of sample points per ray. Previous work has addressed this by either proposing neural scene representations that are faster to evaluate or by pre-computing (and approximating) scene properties to reduce render times. In this work, we propose MCNeRF, a general Monte Carlo-based rendering algorithm that can speed up any NeRF representation. We show that the NeRF volume rendering integral can be efficiently computed via Monte Carlo integration using an importance sampling scheme based on ray density distributions. This allows us to use a small number of MLP evaluations to estimate pixel radiance. These noisy Monte Carlo estimates can be further denoised using an inexpensive image-space denoiser trained per-scene. We demonstrate that MCNeRF can be used to speed up NeRF representations like TensoRF by 7 × while closely matching their visual quality and without making the scene approximations that real-time NeRF rendering methods usually make.

Supplemental Material

MP4 File

Supplementary video

Download
440.99 MB

References

[1]

Relja Arandjelović and Andrew Zisserman. 2021. Nerf in detail: Learning to sample for view synthesis. arXiv preprint arXiv:2106.05264 (2021).

[2]

Steve Bako, Thijs Vogels, Brian Mcwilliams, Mark Meyer, Jan NováK, Alex Harvill, Pradeep Sen, Tony Derose, and Fabrice Rousselle. 2017. Kernel-Predicting Convolutional Networks for Denoising Monte Carlo Renderings. ACM Trans. Graph. 36, 4, Article 97 (jul 2017), 14 pages.

Digital Library

[3]

Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. 2022. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5470–5479.

[4]

Chakravarty R Alla Chaitanya, Anton S Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1–12.

Digital Library

[5]

Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. 2022b. Tensorf: Tensorial radiance fields. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXII. Springer, 333–350.

[6]

Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. 2022a. Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. arXiv preprint arXiv:2208.00277 (2022).

[7]

Michaël Gharbi, Tzu-Mao Li, Miika Aittala, Jaakko Lehtinen, and Frédo Durand. 2019. Sample-based Monte Carlo denoising using a kernel-splatting network. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1–12.

Digital Library

[8]

Yuan-Chen Guo. 2022. Instant Neural Surface Reconstruction. https://github.com/bennyguo/instant-nsr-pl.

[9]

Miloš Hašan, Jaroslav Křivánek, Bruce Walter, and Kavita Bala. 2009. Virtual spherical lights for many-light rendering of glossy scenes. In ACM SIGGRAPH Asia 2009 papers. 1–6.

[10]

Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. 2021. Baking neural radiance fields for real-time view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 5875–5884.

[11]

Mustafa Işık, Krishna Mullia, Matthew Fisher, Jonathan Eisenmann, and Michaël Gharbi. 2021. Interactive Monte Carlo denoising using affinity of neural features. ACM Transactions on Graphics (TOG) 40, 4 (2021), 1–13.

Digital Library

[12]

Yifan Jiang, Peter Hedman, Ben Mildenhall, Dejia Xu, Jonathan T Barron, Zhangyang Wang, and Tianfan Xue. 2022. AligNeRF: High-Fidelity Neural Radiance Fields via Alignment-Aware Training. arXiv preprint arXiv:2211.09682 (2022).

[13]

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).

[14]

Murat Kurt, László Szirmay-Kalos, and Jaroslav Křivánek. 2010. An anisotropic BRDF model for fitting and Monte Carlo rendering. ACM SIGGRAPH Computer Graphics 44, 1 (2010), 1–15.

Digital Library

[15]

Andreas Kurz, Thomas Neff, Zhaoyang Lv, Michael Zollhöfer, and Markus Steinberger. 2022. AdaNeRF: Adaptive Sampling for Real-Time Rendering of Neural Radiance Fields. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVII. Springer, 254–270.

Digital Library

[16]

Peter Kutz, Ralf Habel, Yining Karl Li, and Jan Novák. 2017. Spectral and decomposition tracking for rendering heterogeneous volumes. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1–16.

Digital Library

[17]

Jason Lawrence, Szymon Rusinkiewicz, and Ravi Ramamoorthi. 2004. Efficient BRDF importance sampling using a factored representation. ACM Transactions on Graphics (ToG) 23, 3 (2004), 496–505.

Digital Library

[18]

Jun S Liu and Jun S Liu. 2001. Monte Carlo strategies in scientific computing. Vol. 75. Springer.

[19]

Nelson Max. 1995. Optical models for direct volume rendering. IEEE Transactions on Visualization and Computer Graphics 1, 2 (1995), 99–108.

Digital Library

[20]

Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. 2019. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1–14.

Digital Library

[21]

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I. 405–421.

[22]

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG) 41, 4 (2022), 1–15.

Digital Library

[23]

Thomas Müller, Markus Gross, and Jan Novák. 2017. Practical path guiding for efficient light-transport simulation. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 91–100.

[24]

Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Joerg H Mueller, Chakravarty R Alla Chaitanya, Anton Kaplanyan, and Markus Steinberger. 2021. DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks. In Computer Graphics Forum, Vol. 40. Wiley Online Library, 45–59.

[25]

Alexander Rath, Pascal Grittmann, Sebastian Herholz, Petr Vévoda, Philipp Slusallek, and Jaroslav Křivánek. 2020. Variance-aware path guiding. ACM Transactions on Graphics (TOG) 39, 4 (2020), 151–1.

Digital Library

[26]

Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. 2021. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 14335–14345.

[27]

Christian Reiser, Rick Szeliski, Dor Verbin, Pratul Srinivasan, Ben Mildenhall, Andreas Geiger, Jon Barron, and Peter Hedman. 2023. Merf: Memory-efficient radiance fields for real-time view synthesis in unbounded scenes. ACM Transactions on Graphics (TOG) 42, 4 (2023), 1–12.

Digital Library

[28]

Christian P Robert, George Casella, and George Casella. 1999. Monte Carlo statistical methods. Vol. 2. Springer.

[29]

Cheng Sun, Min Sun, and Hwann-Tzong Chen. 2022. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5459–5469.

[30]

Petr Vévoda, Ivo Kondapaneni, and Jaroslav Křivánek. 2018. Bayesian online regression for adaptive direct illumination sampling. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1–12.

Digital Library

[31]

Thijs Vogels, Fabrice Rousselle, Brian Mcwilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. 2018. Denoising with Kernel Prediction and Asymmetric Loss Functions. ACM Trans. Graph. 37, 4, Article 124 (jul 2018), 15 pages.

Digital Library

[32]

E Woodcock, T Murphy, P Hemmings, and S Longworth. 1965. Techniques used in the GEM code for Monte Carlo neutronics calculations in reactors and other systems of complex geometry. In Proc. Conf. Applications of Computing Methods to Reactor Problems, Vol. 557. Argonne National Laboratory.

[33]

Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P Srinivasan, Richard Szeliski, Jonathan T Barron, and Ben Mildenhall. 2023. BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis. arXiv preprint arXiv:2302.14859 (2023).

[34]

Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. 2021a. Plenoxels: Radiance fields without neural networks. arXiv preprint arXiv:2112.05131 (2021).

[35]

Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. 2021b. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 5752–5761.

Cited By

Yang ZLiu BSong YYi LXiong YZhang ZYu X(2024)DirectL: Efficient Radiance Fields Rendering for 3D Light Field DisplaysACM Transactions on Graphics10.1145/368789743:6(1-19)Online publication date: 19-Nov-2024
https://dl.acm.org/doi/10.1145/3687897
Trevithick AChan MTakikawa TIqbal UDe Mello SChandraker MRamamoorthi RNagano K(2024)What You See is What You GAN: Rendering Every Pixel for High-Fidelity Geometry in 3D GANs2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.02148(22765-22775)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.02148
Turki HAgrawal VBuló SPorzi LKontschieder PRamanan DZollhöfer MRichardt C(2024)HybridNeRF: Efficient Neural Rendering via Adaptive Volumetric Surfaces2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.01858(19647-19656)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.01858
Show More Cited By

Index Terms

MCNeRF: Monte Carlo Rendering and Denoising for Real-Time NeRFs
1. Computing methodologies
  1. Computer graphics
    1. Rendering

Recommendations

An anisotropic BRDF model for fitting and Monte Carlo rendering
Visual Research, Evaluation and Assessment in the Age of Computer Graphics

In this paper we propose a new physically plausible, anisotropic Bidirectional Reflectance Distribution Function (BRDF) for fitting and for importance sampling in global illumination rendering. We demonstrate that the new model is better in data fitting ...
Double hierarchies for efficient sampling in Monte Carlo rendering
SIGGRAPH '17: ACM SIGGRAPH 2017 Talks

We propose a novel representation of the light field tailored to improve importance sampling for Monte Carlo rendering. The domain of the light field i.e., the product space of spatial positions and directions is hierarchically subdivided into subsets ...
Noiseless GPU rendering of isotropic BRDF surfaces

Illumination at a surface point is formulated as an integral of a BRDF using the incident radiance over the hemisphere domain. A popular method to compute the integral is Monte Carlo integration, in which the surface illumination is computed as the sum ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SA '23: SIGGRAPH Asia 2023 Conference Papers

December 2023

1113 pages

ISBN:9798400703157

DOI:10.1145/3610548

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 December 2023

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

SA '23

Sponsor:

SIGGRAPH

SA '23: SIGGRAPH Asia 2023

December 12 - 15, 2023

NSW, Sydney, Australia

Acceptance Rates

Overall Acceptance Rate 178 of 869 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
856
Total Downloads

Downloads (Last 12 months)856
Downloads (Last 6 weeks)51

Reflects downloads up to 28 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yang ZLiu BSong YYi LXiong YZhang ZYu X(2024)DirectL: Efficient Radiance Fields Rendering for 3D Light Field DisplaysACM Transactions on Graphics10.1145/368789743:6(1-19)Online publication date: 19-Nov-2024
https://dl.acm.org/doi/10.1145/3687897
Trevithick AChan MTakikawa TIqbal UDe Mello SChandraker MRamamoorthi RNagano K(2024)What You See is What You GAN: Rendering Every Pixel for High-Fidelity Geometry in 3D GANs2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.02148(22765-22775)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.02148
Turki HAgrawal VBuló SPorzi LKontschieder PRamanan DZollhöfer MRichardt C(2024)HybridNeRF: Efficient Neural Rendering via Adaptive Volumetric Surfaces2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.01858(19647-19656)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.01858
Attal BVerbin DMildenhall BHedman PBarron JO’Toole MSrinivasan P(2024)Flash Cache: Reducing Bias in Radiance Cache Based Inverse RenderingComputer Vision – ECCV 202410.1007/978-3-031-73390-1_2(20-36)Online publication date: 29-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-73390-1_2

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents