research-article

Open access

Temporally Stable Real-Time Joint Neural Denoising and Supersampling

Authors:

Manu Mathew Thomas,

Christoph Peters,

Karthik Vaidyanathan,

Angus G. ForbesAuthors Info & Claims

Proceedings of the ACM on Computer Graphics and Interactive Techniques, Volume 5, Issue 3

Article No.: 21, Pages 1 - 22

https://doi.org/10.1145/3543870

Published: 27 July 2022 Publication History

Abstract

Recent advances in ray tracing hardware bring real-time path tracing into reach, and ray traced soft shadows, glossy reflections, and diffuse global illumination are now common features in games. Nonetheless, ray budgets are still limited. This results in undersampling, which manifests as aliasing and noise. Prior work addresses these issues separately. While temporal supersampling methods based on neural networks have gained a wide use in modern games due to their better robustness, neural denoising remains challenging because of its higher computational cost.

We introduce a novel neural network architecture for real-time rendering that combines supersampling and denoising, thus lowering the cost compared to two separate networks. This is achieved by sharing a single low-precision feature extractor with multiple higher-precision filter stages. To reduce cost further, our network takes low-resolution inputs and reconstructs a high-resolution denoised supersampled output. Our technique produces temporally stable high-fidelity results that significantly outperform state-of-the-art real-time statistical or analytical denoisers combined with TAA or neural upsampling to the target resolution.

Supplementary Material

thomas (thomas.zip)

Supplemental movie, appendix, image and software files for, Temporally Stable Real-Time Joint Neural Denoising and Supersampling

Download
458.15 MB

References

[1]

Steve Bako, Thijs Vogels, Brian Mcwilliams, Mark Meyer, Jan NováK, Alex Harvill, Pradeep Sen, Tony Derose, and Fabrice Rousselle. 2017. Kernel-Predicting Convolutional Networks for Denoising Monte Carlo Renderings. ACM Trans. Graph. 36, 4, Article 97 (jul 2017), 14 pages. https://doi.org/10.1145/3072959.3073708

Digital Library

[2]

Chakravarty R. Alla Chaitanya, Anton S. Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive Reconstruction of Monte Carlo Image Sequences Using a Recurrent Denoising Autoencoder. ACM Trans. Graph. 36, 4, Article 98 (jul 2017), 12 pages. https://doi.org/10.1145/3072959.3073601

Digital Library

[3]

Hisham Chowdhury, Kawiak, Rense Robert, de Boer, Gabriel Ferreira, and Lucas Xavier. 2022. Intel XeSS - an AI based Super Sampling solution for Real-time Rendering. (2022). In Game Developers Conference.

[4]

Michael Crawshaw. 2020. Multi-task learning with deep neural networks: A survey. arXiv preprint arXiv:2009.09796 (2020).

[5]

Holger Dammertz, Daniel Sewtz, Johannes Hanika, and Hendrik P. A. Lensch. 2010. Edge-Avoiding À-Trous Wavelet Transform for Fast Global Illumination Filtering. In Proceedings of the Conference on High Performance Graphics (Saarbrucken, Germany) (HPG '10). Eurographics Association, 67--75.

[6]

Epic Games. 2018. Unreal Engine 4.19: Screen percentage with temporal upsample. https://docs.unrealengine.com/4.27/enUS/RenderingAndGraphics/ScreenPercentage/ Hangming Fan, Rui Wang, Yuchi Huo, and Hujun Bao. 2021. Real-time Monte Carlo Denoising with Weight Sharing Kernel Prediction Network. Computer Graphics Forum (2021). https://doi.org/10.1111/cgf.14338

[7]

Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. 2018. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811.12231 (2018).

[8]

Jon Hasselgren, Jacob Munkberg, Marco Salvi, Anjul Patney, and Aaron Lefohn. 2020a. Neural Temporal Adaptive Sampling and Denoising. Computer Graphics Forum (2020). https://doi.org/10.1111/cgf.13919

[9]

Jon Hasselgren, Jacob Munkberg, Marco Salvi, Anjul Patney, and Aaron Lefohn. 2020b. Neural temporal adaptive sampling and denoising. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 147--155.

[10]

Mustafa Işık, Krishna Mullia, Matthew Fisher, Jonathan Eisenmann, and Michaël Gharbi. 2021. Interactive Monte Carlo Denoising Using Affinity of Neural Features. ACM Trans. Graph. 40, 4, Article 37 (jul 2021), 13 pages. https://doi.org/10.1145/3450626.3459793

Digital Library

[11]

Intel Corporation. 2019. Intel Open Image Denoise. https://www.openimagedenoise.org/

[12]

Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual Losses for Real-Time Style Transfer and Super-Resolution. In Computer Vision - ECCV 2016, Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (Eds.). Springer International Publishing, Cham, 694--711.

[13]

Nima Khademi Kalantari, Steve Bako, and Pradeep Sen. 2015. A Machine Learning Approach for Filtering Monte Carlo Noise. ACM Trans. Graph. 34, 4, Article 122 (jul 2015), 12 pages. https://doi.org/10.1145/2766977

Digital Library

[14]

Brian Karis. 2014. High quality temporal supersampling. (2014). In ACM SIGGRAPH Courses: Advances in RealTime Rendering in Games.

[15]

Alex Kendall, Yarin Gal, and Roberto Cipolla. 2018. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7482--7491.

[16]

Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. 2019. Panoptic feature pyramid networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6399--6408.

[17]

Edward Liu. 2020. DLSS 2.0 - Image Reconstruction for Real-Time Rendering with Deep learning. (2020). In Game Developers Conference.

[18]

Edward Liu, Ignacio Llamas, Juan Cañada, and Patrick Kelly. 2019. Cinematic Rendering in UE4 with Real-Time Ray Tracing and Denoising. In Ray Tracing Gems, Eric Haines and Tomas Akenine-Möller (Eds.). Apress Media, 289--319. https://doi.org/10.1007/978-1-4842-4427-2

[19]

Soham Uday Mehta, Brandon Wang, and Ravi Ramamoorthi. 2012. Axis-Aligned Filtering for Interactive Sampled Soft Shadows. ACM Trans. Graph. 31, 6, Article 163 (nov 2012), 10 pages. https://doi.org/10.1145/2366145.2366182

Digital Library

[20]

Soham Uday Mehta, Brandon Wang, Ravi Ramamoorthi, and Fredo Durand. 2013. Axis-Aligned Filtering for Interactive Physically-Based Diffuse Indirect Lighting. ACM Trans. Graph. 32, 4, Article 96 (jul 2013), 12 pages. https://doi.org/10.1145/2461912.2461947

Digital Library

[21]

Xiaoxu Meng, Quan Zheng, Amitabh Varshney, Gurprit Singh, and Matthias Zwicker. 2020. Real-time Monte Carlo Denoising with the Neural Bilateral Grid. In Eurographics Symposium on Rendering - DL-only Track, Carsten Dachsbacher and Matt Pharr (Eds.). The Eurographics Association. https://doi.org/10.2312/sr.20201133

[22]

Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M. Susskind. 2021. Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 10912--10922.

[23]

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015a. U-Net: Convolutional Networks for Biomedical Image Segmentation. arXiv:1505.04597 [cs.CV]

[24]

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015b. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234--241.

[25]

Christoph Schied, Anton Kaplanyan, Chris Wyman, Anjul Patney, Chakravarty R Alla Chaitanya, John Burgess, Shiqiu Liu, Carsten Dachsbacher, Aaron Lefohn, and Marco Salvi. 2017. Spatiotemporal variance-guided filtering: real-time reconstruction for path-traced global illumination. In Proceedings of High Performance Graphics. 1--12.

Digital Library

[26]

Christoph Schied, Christoph Peters, and Carsten Dachsbacher. 2018. Gradient estimation for real-time adaptive temporal filtering. Proceedings of the ACM on Computer Graphics and Interactive Techniques 1, 2 (2018), 1--16.

Digital Library

[27]

Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).

[28]

Akella Ravi Tej, Shirsendu Sukanta Halder, Arunav Pratap Shandeelya, and Vinod Pankajakshan. 2020. Enhancing perceptual loss with adversarial feature matching for super-resolution. In 2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 1--8.

[29]

Manu Mathew Thomas, Karthik Vaidyanathan, Gabor Liktor, and Angus G. Forbes. 2020. A Reduced-Precision Network for Image Reconstruction. ACM Trans. Graph. 39, 6, Article 231 (nov 2020), 12 pages. https://doi.org/10.1145/3414685.3417786

Digital Library

[30]

Thijs Vogels, Fabrice Rousselle, Brian Mcwilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. 2018. Denoising with Kernel Prediction and Asymmetric Loss Functions. ACM Trans. Graph. 37, 4, Article 124 (jul 2018), 15 pages. https://doi.org/10.1145/3197517.3201388

Digital Library

[31]

Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. 2018. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European conference on computer vision (ECCV) workshops. 0--0.

[32]

Sebastian Weiss, Mengyu Chu, Nils Thürey, and Rüdiger Westermann. 2021. Volumetric Isosurface Rendering with Deep Learning-Based Super-Resolution. IEEE Transactions on Visualization and Computer Graphics 27 (2021), 3064--3078.

[33]

Lei Xiao, Salah Nouri, Matt Chapman, Alexander Fix, Douglas Lanman, and Anton Kaplanyan. 2020. Neural supersampling for real-time rendering. ACM Transactions on Graphics (TOG) 39, 4 (2020), 142--1.

Digital Library

[34]

Bing Xu, Junfei Zhang, Rui Wang, Kun Xu, Yong-Liang Yang, Chuan Li, and Rui Tang. 2019. Adversarial Monte Carlo Denoising with Conditioned Auxiliary Feature Modulation. ACM Trans. Graph. 38, 6, Article 224 (nov 2019), 12 pages. https://doi.org/10.1145/3355089.3356547

Digital Library

[35]

Ling-Qi Yan, Soham Uday Mehta, Ravi Ramamoorthi, and Fredo Durand. 2015. Fast 4D Sheared Filtering for Interactive Rendering of Distribution Effects. ACM Transactions on Graphics 35, 1 (2015), 7.

Digital Library

[36]

Lei Yang, Shiqiu Liu, and Marco Salvi. 2020. A Survey of Temporal Antialiasing Techniques. Computer Graphics Forum 39 (2020).

[37]

Zheng Zeng, Shiqiu Liu, Jinglei Yang, Lu Wang, and Ling-Qi Yan. 2021. Temporally Reliable Motion Vectors for Real-time Ray Tracing. Computer Graphics Forum 40, 2 (2021). https://doi.org/10.1111/cgf.142616

[38]

Yu Zhang and Qiang Yang. 2021. A Survey on Multi-Task Learning. IEEE Transactions on Knowledge and Data Engineering (2021), 1--1. https://doi.org/10.1109/TKDE.2021.3070203

Cited By

Kandlbinder LDittebrandt ASchipek ADachsbacher C(2024)Optimizing Path Termination for Radiance Caching Through Explicit Variance TradingProceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/36753817:3(1-19)Online publication date: 9-Aug-2024
https://dl.acm.org/doi/10.1145/3675381
Dolp RHanika JDachsbacher C(2024)A Fast GPU Schedule For À-Trous Wavelet-Based DenoisersProceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/36512997:1(1-18)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3651299
Harrer MFranke LFink LStamminger MWeyrich T(2023)Inovis: Instant Novel-View SynthesisSIGGRAPH Asia 2023 Conference Papers10.1145/3610548.3618216(1-12)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.1145/3610548.3618216
Show More Cited By

Index Terms

Temporally Stable Real-Time Joint Neural Denoising and Supersampling
1. Computer systems organization
  1. Architectures
    1. Other architectures
      1. Neural networks
2. Computing methodologies
  1. Computer graphics
    1. Rendering

Recommendations

Level-of-detail representation of bidirectional texture functions for real-time rendering
I3D '05: Proceedings of the 2005 symposium on Interactive 3D graphics and games

This paper presents a new technique for rendering bidirectional texture functions (BTFs) at different levels of detail (LODs). Our method first decomposes each BTF image into multiple subbands with a Laplacian pyramid. Each vector of Laplacian ...
Display supersampling

Supersampling is widely used by graphics hardware to render anti-aliased images. In conventional supersampling, multiple scene samples are computationally combined to produce a single screen pixel. We consider a novel imaging paradigm that we call ...
Exploring a Boeing 777: Ray Tracing Large-Scale CAD Data

Unlike rasterization, which draws and shades triangles individually, ray tracing simulates propagation of light between surfaces in a scene and enables advanced effects such as transparency, shadows, and ambient occlusion--effects not easily ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Computer Graphics and Interactive Techniques

Proceedings of the ACM on Computer Graphics and Interactive Techniques Volume 5, Issue 3

July 2022

198 pages

EISSN:2577-6193

DOI:10.1145/3552302

Issue’s Table of Contents

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 July 2022

Published in PACMCGIT Volume 5, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
1,859
Total Downloads

Downloads (Last 12 months)735
Downloads (Last 6 weeks)114

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kandlbinder LDittebrandt ASchipek ADachsbacher C(2024)Optimizing Path Termination for Radiance Caching Through Explicit Variance TradingProceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/36753817:3(1-19)Online publication date: 9-Aug-2024
https://dl.acm.org/doi/10.1145/3675381
Dolp RHanika JDachsbacher C(2024)A Fast GPU Schedule For À-Trous Wavelet-Based DenoisersProceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/36512997:1(1-18)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3651299
Harrer MFranke LFink LStamminger MWeyrich T(2023)Inovis: Instant Novel-View SynthesisSIGGRAPH Asia 2023 Conference Papers10.1145/3610548.3618216(1-12)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.1145/3610548.3618216
Balint MWolski KMyszkowski KSeidel HMantiuk R(2023)Neural Partitioning Pyramids for Denoising Monte Carlo RenderingsACM SIGGRAPH 2023 Conference Proceedings10.1145/3588432.3591562(1-11)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1145/3588432.3591562
Firmino AFrisvad JJensen H(2023)Denoising-Aware Adaptive Sampling for Monte Carlo Ray TracingACM SIGGRAPH 2023 Conference Proceedings10.1145/3588432.3591537(1-11)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1145/3588432.3591537
Chen YLu YZhang XXie N(2023)Interactive neural cascade denoising for 1-spp Monte Carlo imagesThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-023-02951-639:8(3197-3210)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1007/s00371-023-02951-6

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents