Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article

Deep Adaptive Sampling and Reconstruction Using Analytic Distributions

Published: 30 November 2022 Publication History

Abstract

We propose an adaptive sampling and reconstruction method for offline Monte Carlo rendering. Our method produces sampling maps constrained by a user-defined budget that minimize the expected future denoising error. Compared to other state-of-the-art methods, which produce the necessary training data on the fly by composing pre-rendered images, our method samples from analytic noise distributions instead. These distributions are compact and closely approximate the pixel value distributions stemming from Monte Carlo rendering. Our method can efficiently sample training data by leveraging only a few per-pixel statistics of the target distribution, which provides several benefits over the current state of the art. Most notably, our analytic distributions' modeling accuracy and sampling efficiency increase with sample count, essential for high-quality offline rendering. Although our distributions are approximate, our method supports joint end-to-end training of the sampling and denoising networks. Finally, we propose the addition of a global summary module to our architecture that accumulates valuable information from image regions outside of the network's receptive field. This information discourages sub-optimal decisions based on local information. Our evaluation against other state-of-the-art neural sampling methods demonstrates denoising quality and data efficiency improvements.

References

[1]
Martín Abadi, Ashish Agarwal, Paul Barham, et al. 2015. TensorFlow: Large-scale machine learning on heterogeneous systems. http://tensorflow.org/
[2]
Pontus Andersson, Jim Nilsson, Tomas Akenine-Möller, Magnus Oskarsson, Kalle Åström, and Mark D. Fairchild. 2020. FLIP: A Difference Evaluator for Alternating Images. Proc. ACM Comput. Graph. Interact. Tech. 3, 2, Article 15 (aug 2020), 23 pages.
[3]
Steve Bako, Thijs Vogels, Brian Mcwilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony Derose, and Fabrice Rousselle. 2017. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Trans. Graphics (Proc. SIGGRAPH) 36, 4, Article 97 (July 2017), 14 pages.
[4]
Kavita Bala, Bruce Walter, and Donald P Greenberg. 2003. Combining edges and points for interactive high-quality rendering. ACM Transactions on Graphics (TOG) 22, 3 (2003), 631--640.
[5]
Pablo Bauszat, Martin Eisemann, Elmar Eisemann, and Marcus Magnor. 2015. General and Robust Error Estimation and Reconstruction for Monte Carlo Rendering. Computer Graphics Forum (Proc. of Eurographics EG) 34, 2 (May 2015), 597--608. Eurographics 2015 paper.
[6]
Pablo Bauszat, Martin Eisemann, and Marcus Magnor. 2011. Adaptive Sampling for Geometry-aware Reconstruction Filters. In Proc. Vision, Modeling and Visualization (VMV). Eurographics, 183--190.
[7]
Benedikt Bitterli, Fabrice Rousselle, Bochang Moon, José A. Iglesias-Guitián, David Adler, Kenny Mitchell, Wojciech Jarosz, and Jan Novák. 2016. Nonlinearly Weighted First-order Regression for Denoising Monte Carlo Renderings. Computer Graphics Forum 35, 4 (2016), 107--117. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.12954
[8]
Chakravarty R. Alla Chaitanya, Anton S. Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive Reconstruction of Monte Carlo Image Sequences Using a Recurrent Denoising Autoencoder. ACM Trans. Graph. 36, 4, Article 98 (July 2017), 12 pages.
[9]
Frédo Durand, Nicolas Holzschuch, Cyril Soler, Eric Chan, and François X. Sillion. 2005. A Frequency Analysis of Light Transport. ACM Trans. Graph. 24, 3 (July 2005), 1115--1126.
[10]
Oskar Elek, Manu M. Thomas, and Angus Forbes. 2019. Learning Patterns in Sample Distributions for Monte Carlo Variance Reduction. arXiv:1906.00124 [cs.GR]
[11]
Mikhail Figurnov, Shakir Mohamed, and Andriy Mnih. 2018. Implicit Reparameterization Gradients. In Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2018/file/92c8c96e4c37100777c7190b76d28233-Paper.pdf
[12]
Baining Guo. 1998. Progressive radiance evaluation using directional coherence maps. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques. 255--266.
[13]
Toshiya Hachisuka, Wojciech Jarosz, Richard Peter Weistroffer, Kevin Dale, Greg Humphreys, Matthias Zwicker, and Henrik Wann Jensen. 2008. Multidimensional Adaptive Sampling and Reconstruction for Ray Tracing. ACM Trans. Graph. 27, 3 (Aug. 2008), 1--10.
[14]
Jon Hasselgren, Jacob Munkberg, Marco Salvi, Anjul Patney, and Aaron Lefohn. 2020. Neural Temporal Adaptive Sampling and Denoising. Computer Graphics Forum (2020).
[15]
Mustafa Işık, Krishna Mullia, Matthew Fisher, Jonathan Eisenmann, and Michaël Gharbi. 2021. Interactive Monte Carlo Denoising using Affinity of Neural Features. ACM Transactions on Graphics (TOG) 40, 4, Article 37 (2021), 13 pages.
[16]
Wenzel Jakob. 2010. Mitsuba renderer. http://www.mitsuba-renderer.org.
[17]
Nima Khademi Kalantari, Steve Bako, and Pradeep Sen. 2015. A Machine Learning Approach for Filtering Monte Carlo Noise. ACM Trans. Graph. 34, 4, Article 122 (July 2015), 12 pages.
[18]
Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013).
[19]
David Kirk and James Arvo. 1991. Unbiased Sampling Techniques for Image Synthesis. In Proceedings of the 18th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '91). Association for Computing Machinery, New York, NY, USA, 153--156.
[20]
Alexandr Kuznetsov, Nima Khademi Kalantari, and Ravi Ramamoorthi. 2018. Deep Adaptive Sampling for Low Sample Count Rendering. Computer Graphics Forum 37 (07 2018), 35--44.
[21]
Tzu-Mao Li, Yu-Ting Wu, and Yung-Yu Chuang. 2012a. SURE-Based Optimization for Adaptive Sampling and Reconstruction. ACM Trans. Graph. 31, 6, Article 194 (Nov. 2012), 9 pages.
[22]
Tzu-Mao Li, Yu-Ting Wu, and Yung-Yu Chuang. 2012b. SURE-based optimization for adaptive sampling and reconstruction. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 31, 6, Article 194 (Nov. 2012), 9 pages.
[23]
Don P. Mitchell. 1987. Generating Antialiased Images at Low Sampling Densities. SIGGRAPH Comput. Graph. 21, 4 (Aug. 1987), 65--72.
[24]
Bochang Moon, Nathan Carr, and Sung-Eui Yoon. 2014. Adaptive Rendering Based on Weighted Local Regression. ACM Trans. Graph. 33, 5, Article 170 (Sept. 2014), 14 pages.
[25]
Bochang Moon, Jose A. Iglesias-Guitian, Sung-Eui Yoon, and Kenny Mitchell. 2015. Adaptive Rendering with Linear Predictions. ACM Trans. Graph. 34, 4, Article 121 (July 2015), 11 pages.
[26]
Bochang Moon, Steven McDonagh, Kenny Mitchell, and Markus Gross. 2016. Adaptive Polynomial Rendering. ACM Trans. Graph. 35, 4, Article 40 (July 2016), 10 pages.
[27]
Ryan S. Overbeck, Craig Donner, and Ravi Ramamoorthi. 2009. Adaptive Wavelet Rendering. In ACM SIGGRAPH Asia 2009 Papers (Yokohama, Japan) (SIGGRAPH Asia '09). Association for Computing Machinery, New York, NY, USA, Article 140, 12 pages.
[28]
Fabrice Rousselle, Claude Knaus, and Matthias Zwicker. 2011. Adaptive Sampling and Reconstruction Using Greedy Error Minimization. ACM Trans. Graph. 30, 6 (Dec. 2011), 1--12.
[29]
Fabrice Rousselle, Claude Knaus, and Matthias Zwicker. 2012. Adaptive Rendering with Non-Local Means Filtering. ACM Trans. Graph. 31, 6, Article 195 (Nov. 2012), 11 pages.
[30]
Fabrice Rousselle, Marco Manzi, and Matthias Zwicker. 2013. Robust Denoising using Feature and Color Information. Computer Graphics Forum 32 (10 2013).
[31]
Rasmus Tamstorf and Henrik Jensen. 1997. Adaptive Sampling and Bias Estimation in Path Tracing. Eurographics Workshop on Rendering Techniques 97 (07 1997).
[32]
Thijs Vogels, Fabrice Rousselle, Brian McWilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. 2018. Denoising with Kernel Prediction and Asymmetric Loss Functions. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2018) 37, 4, Article 124 (2018), 124:1--124:15 pages.
[33]
Tiange Xiang, Hongliang Yuan, Haozhi Huang, and Yujin Shi. 2021. Two-Stage Monte Carlo Denoising with Adaptive Sampling and Kernel Pool. arXiv:2103.16115 [cs.CV]
[34]
Xianyao Zhang, Marco Manzi, Thijs Vogels, Henrik Dahlberg, Markus Gross, and Marios Papas. 2021. Deep Compositional Denoising for High-quality Monte Carlo Rendering. Computer Graphics Forum 40, 4 (2021), 1--13. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14337
[35]
Xianyao Zhang, Melvin Ott, Marco Manzi, Markus Gross, and Marios Papas. 2022. Automatic Feature Selection for Denoising Volumetric Renderings. Computer Graphics Forum 41, 4 (2022), 63--77. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14587

Cited By

View all
  • (2024)Practical Error Estimation for Denoised Monte Carlo Image SynthesisACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657511(1-10)Online publication date: 13-Jul-2024
  • (2023)RL-based stateful neural adaptive sampling and denoising for real-time path tracingProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3669020(66390-66407)Online publication date: 10-Dec-2023
  • (2023)Input-Dependent Uncorrelated Weighting for Monte Carlo DenoisingSIGGRAPH Asia 2023 Conference Papers10.1145/3610548.3618177(1-10)Online publication date: 10-Dec-2023
  • Show More Cited By

Index Terms

  1. Deep Adaptive Sampling and Reconstruction Using Analytic Distributions

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Transactions on Graphics
      ACM Transactions on Graphics  Volume 41, Issue 6
      December 2022
      1428 pages
      ISSN:0730-0301
      EISSN:1557-7368
      DOI:10.1145/3550454
      Issue’s Table of Contents
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 30 November 2022
      Published in TOG Volume 41, Issue 6

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. adaptive sampling
      2. denoising
      3. path tracing

      Qualifiers

      • Research-article

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)101
      • Downloads (Last 6 weeks)14
      Reflects downloads up to 23 Dec 2024

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Practical Error Estimation for Denoised Monte Carlo Image SynthesisACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657511(1-10)Online publication date: 13-Jul-2024
      • (2023)RL-based stateful neural adaptive sampling and denoising for real-time path tracingProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3669020(66390-66407)Online publication date: 10-Dec-2023
      • (2023)Input-Dependent Uncorrelated Weighting for Monte Carlo DenoisingSIGGRAPH Asia 2023 Conference Papers10.1145/3610548.3618177(1-10)Online publication date: 10-Dec-2023
      • (2023)Denoising-Aware Adaptive Sampling for Monte Carlo Ray TracingACM SIGGRAPH 2023 Conference Proceedings10.1145/3588432.3591537(1-11)Online publication date: 23-Jul-2023
      • (2023)Perspectives and Final RemarksDeep Learning for Fluid Simulation and Animation10.1007/978-3-031-42333-8_9(137-145)Online publication date: 11-Aug-2023

      View Options

      Login options

      Full Access

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media