research-article

Public Access

Deep inverse rendering for high-resolution SVBRDF estimation from an arbitrary number of images

Authors:

Xin TongAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 38, Issue 4

Article No.: 134, Pages 1 - 15

https://doi.org/10.1145/3306346.3323042

Published: 12 July 2019 Publication History

Abstract

In this paper we present a unified deep inverse rendering framework for estimating the spatially-varying appearance properties of a planar exemplar from an arbitrary number of input photographs, ranging from just a single photograph to many photographs. The precision of the estimated appearance scales from plausible when the input photographs fails to capture all the reflectance information, to accurate for large input sets. A key distinguishing feature of our framework is that it directly optimizes for the appearance parameters in a latent embedded space of spatially-varying appearance, such that no handcrafted heuristics are needed to regularize the optimization. This latent embedding is learned through a fully convolutional auto-encoder that has been designed to regularize the optimization. Our framework not only supports an arbitrary number of input photographs, but also at high resolution. We demonstrate and evaluate our deep inverse rendering solution on a wide variety of publicly available datasets.

References

[1]

Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. https://www.tensorflow.org/

[2]

Miika Aittala, Timo Aila, and Jaakko Lehtinen. 2016. Reflectance Modeling by Neural Texture Synthesis. ACM Trans. Graph. 35, 4, Article 65 (July 2016).

Digital Library

[3]

Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. 2015. Two-shot SVBRDF Capture for Stationary Materials. ACM Trans. Graph. 34, 4, Article 110 (July 2015).

Digital Library

[4]

Dan A. Calian, Jean-François Lalonde, Paulo Gotardo, Tomas Simon, Iain Matthews, and Kenny Mitchell. 2018. From Faces to Outdoor Light Probes. Computer Graphics Forum 37, 2 (2018), 51--61.

[5]

Inchang Choi, Daniel S. Jeon, Giljoo Nam, Diego Gutierrez, and Min H. Kim. 2017. High-Quality Hyperspectral Reconstruction Using a Spectral Prior. ACM Trans. Graph. 36, 6, Article 218 (2017).

Digital Library

[6]

Valentin Deschaintre, Miika Aittala, Frédo Durand, George Drettakis, and Adrien Bousseau. 2018. Single-Image SVBRDF Capture with a Rendering-Aware Deep Network. ACM Trans. Graph. 37, 128 (aug 2018).

Digital Library

[7]

Yue Dong, Guojun Chen, Pieter Peers, Jiawan Zhang, and Xin Tong. 2014. Appearance-from-motion: Recovering Spatially Varying Surface Reflectance Under Unknown Lighting. ACM Trans. Graph. 33, 6, Article 193 (2014).

Digital Library

[8]

Julie Dorsey, Holly Rushmeier, and Franois Sillion. 2008. Digital Modeling of Material Appearance. Morgan Kaufmann Publishers Inc.

Digital Library

[9]

Leon Gatys, Alexander S Ecker, and Matthias Bethge. 2015. Texture synthesis using convolutional neural networks. In NIPS. 262--270.

Digital Library

[10]

Geoffrey Hinton and Ruslan Salakhutdinov. 2006. Reducing the Dimensionality of Data with Neural Networks. Science 313, 5786 (2006), 504 -- 507.

[11]

Zhuo Hui, Kalyan Sunkavalli, Joon-Young Lee, Sunil Hadap, and Aswin Sankaranarayanan. 2017. Reflectance Capture using Univariate Sampling of BRDFs. In ICCV.

[12]

Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In ICML. 448--456.

Digital Library

[13]

Kaizhang Kang, Zimin Chen, Jiaping Wang, Kun Zhou, and Hongzhi Wu. 2018. Efficient Reflectance Capture Using an Autoencoder. ACM Trans. Graph. 37, 4, Article 127 (July 2018).

Digital Library

[14]

Kihwan Kim, Jinwei Gu, Stephen Tyree, Pavlo Molchanov, Matthias Nießner, and Jan Kautz. 2017. A Lightweight Approach for On-the-Fly Reflectance Estimation. In ICCV.

[15]

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR.

[16]

Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2017. Modeling Surface Appearance from a Single Photograph Using Self-augmented Convolutional Neural Networks. ACM Trans. Graph. 36, 4, Article 45 (July 2017), 11 pages.

Digital Library

[17]

Zhengqin Li, Kalyan Sunkavalli, and Manmohan Krishna Chandraker. 2018a. Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Image. ECCV.

[18]

Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. 2018b. Learning to Reconstruct Shape and Spatially-varying Reflectance from a Single Image. ACM Trans. Graph. 37, 6 (2018), 126.

Digital Library

[19]

Gianpaolo Palma, Marco Callieri, Matteo Dellepiane, and Roberto Scopigno. 2012. A Statistical Method for SVBRDF Approximation from Video Sequences in General Lighting Conditions. Comput. Graph. Forum 31, 4 (2012), 1491--1500.

Digital Library

[20]

Jérémy Riviere, Pieter Peers, and Abhijeet Ghosh. 2016. Mobile Surface Reflectometry. Comput. Graph. Forum 35, 1 (2016), 191--202.

Digital Library

[21]

Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing High-Dimensional Data Using t-SNE. (2008), 2579--2605.

[22]

Bruce Walter, Stephen R. Marschner, Hongsong Li, and Kenneth E. Torrance. 2007. Microfacet Models for Refraction through Rough Surfaces. In Rendering Techniques. 195--206.

Digital Library

[23]

Michael Weinmann and Richard Klein. 2015. Advances in Geometry and Reflectance Acquisition. In ACM SIGGRAPH Asia, Course Notes.

Digital Library

[24]

Rui Xia, Yue Dong, Pieter Peers, and Xin Tong. 2016. Recovering Shape and Spatially-Varying Surface Reflectance under Unknown Illumination. ACM Trans. Graph. 35, 6 (December 2016).

Digital Library

[25]

Zexiang Xu, Jannik Boll Nielsen, Jiyang Yu, Henrik Wann Jensen, and Ravi Ramamoorthi. 2016. Minimal BRDF Sampling for Two-shot Near-field Reflectance Acquisition. ACM Trans. Graph. 35, 6, Article 188 (Nov. 2016).

Digital Library

[26]

Zexiang Xu, Kalyan Sunkavalli, Sunil Hadap, and Ravi Ramamoorthi. 2018. Deep image-based relighting from optimal sparse samples. ACM Trans. Graph. 37, 4 (2018), 126.

Digital Library

[27]

Wenjie Ye, Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2018. Single Photograph Surface Appearance Modeling with Self-Augmented CNNs and Inexact Supervision. Comput. Graph. Forum 37, 7 (Oct 2018).

[28]

Zhiming Zhou, Guojun Chen, Yue Dong, David Wipf, Yong Yu, John Snyder, and Xin Tong. 2016. Sparse-as-Possible SVBRDF Acquisition. ACM Trans. Graph. 35 (November 2016).

Digital Library

Cited By

Chitnis SSole AChandran S(2025)Spectral Bidirectional Reflectance Distribution Function SimplificationJournal of Imaging10.3390/jimaging1101001811:1(18)Online publication date: 11-Jan-2025
https://doi.org/10.3390/jimaging11010018
Righetto LKhademizadeh MGiachetti APonchio FGigilashvili DBettio FGobbetti E(2024)Efficient and User-Friendly Visualization of Neural Relightable Images for Cultural Heritage ApplicationsJournal on Computing and Cultural Heritage 10.1145/369039017:4(1-24)Online publication date: 7-Dec-2024
https://dl.acm.org/doi/10.1145/3690390
Vecchio GMartin RRoullier AKaiser ARouffet RDeschaintre VBoubekeur T(2024)ControlMat: A Controlled Generative Approach to Material CaptureACM Transactions on Graphics10.1145/368883043:5(1-17)Online publication date: 27-Aug-2024
https://dl.acm.org/doi/10.1145/3688830
Show More Cited By

Index Terms

Deep inverse rendering for high-resolution SVBRDF estimation from an arbitrary number of images
1. Computing methodologies
  1. Computer graphics
    1. Image manipulation
      1. Image processing
    2. Rendering
      1. Reflectance modeling

Recommendations

Single-image SVBRDF capture with a rendering-aware deep network

Texture, highlights, and shading are some of many visual cues that allow humans to perceive material appearance in single pictures. Yet, recovering spatially-varying bi-directional reflectance distribution functions (SVBRDFs) from a single image based ...
Two-shot SVBRDF capture for stationary materials

Material appearance acquisition usually makes a trade-off between acquisition effort and richness of reflectance representation. In this paper, we instead aim for both a light-weight acquisition procedure and a rich reflectance representation ...
High-resolution SVBRDF estimation based on deep inverse rendering from two-shot images
Abstract
The surface of a large number of objects in the real world shows the characteristics of stationary material. These appearances usually frequently exhibit the same or similar reflectance properties in different locations, while recent methods based ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 38, Issue 4

August 2019

1480 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3306346

Editor:
Olga Sorkine-Hornung
ETH Zurich

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 July 2019

Published in TOG Volume 38, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSF
Google
Activision
Nvidia
National Natural Science Foundation of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

141
Total Citations
View Citations
1,727
Total Downloads

Downloads (Last 12 months)201
Downloads (Last 6 weeks)41

Reflects downloads up to 20 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chitnis SSole AChandran S(2025)Spectral Bidirectional Reflectance Distribution Function SimplificationJournal of Imaging10.3390/jimaging1101001811:1(18)Online publication date: 11-Jan-2025
https://doi.org/10.3390/jimaging11010018
Righetto LKhademizadeh MGiachetti APonchio FGigilashvili DBettio FGobbetti E(2024)Efficient and User-Friendly Visualization of Neural Relightable Images for Cultural Heritage ApplicationsJournal on Computing and Cultural Heritage 10.1145/369039017:4(1-24)Online publication date: 7-Dec-2024
https://dl.acm.org/doi/10.1145/3690390
Vecchio GMartin RRoullier AKaiser ARouffet RDeschaintre VBoubekeur T(2024)ControlMat: A Controlled Generative Approach to Material CaptureACM Transactions on Graphics10.1145/368883043:5(1-17)Online publication date: 27-Aug-2024
https://dl.acm.org/doi/10.1145/3688830
Li BHu YGuerrero PHasan MShi LDeschaintre VMatusik W(2024)Procedural Material Generation with Reinforcement LearningACM Transactions on Graphics10.1145/368797943:6(1-14)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687979
Wang LZhang LGao FKang YZhang J(2024)NFPLight: Deep SVBRDF Estimation via the Combination of Near and Far Field Point LightingACM Transactions on Graphics10.1145/368797843:6(1-11)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687978
Liu CRitschel T(2024)Neural Differential Appearance EquationsACM Transactions on Graphics10.1145/368790043:6(1-17)Online publication date: 19-Nov-2024
https://doi.org/10.1145/3687900
Tsai YAbu Bakar M(2024)HMK-CTA: A Hierarchical Multidimensional Representation for Visual DatasetsProceedings of the 50th Graphics Interface Conference10.1145/3670947.3670954(1-10)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3670947.3670954
Li ZGan RLuo CWang YLiu JZhu ZLi QYin XZhang MZhang ZPeng JCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)MaterialSeg3D: Segmenting Dense Materials from 2D Priors for 3D AssetsProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680757(370-379)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680757
Wang YGuo PZhang HHuang D(2024)Self-augmented modeling surface appearance based on ResNetProceedings of the International Conference on Computer Vision and Deep Learning10.1145/3653781.3653816(1-7)Online publication date: 19-Jan-2024
https://dl.acm.org/doi/10.1145/3653781.3653816
Kavoosighafi BHajisharif SMiandji EBaravdish GCao WUnger J(2024)Deep SVBRDF Acquisition and Modelling: A SurveyComputer Graphics Forum10.1111/cgf.1519943:6Online publication date: 16-Sep-2024
https://doi.org/10.1111/cgf.15199
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents