research-article

Relighting humans: occlusion-aware inverse rendering for full-body human images

Authors:

Yoshihiro Kanamori,

Yuki EndoAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 37, Issue 6

Article No.: 270, Pages 1 - 11

https://doi.org/10.1145/3272127.3275104

Published: 04 December 2018 Publication History

Abstract

Relighting of human images has various applications in image synthesis. For relighting, we must infer albedo, shape, and illumination from a human portrait. Previous techniques rely on human faces for this inference, based on spherical harmonics (SH) lighting. However, because they often ignore light occlusion, inferred shapes are biased and relit images are unnaturally bright particularly at hollowed regions such as armpits, crotches, or garment wrinkles. This paper introduces the first attempt to infer light occlusion in the SH formulation directly. Based on supervised learning using convolutional neural networks (CNNs), we infer not only an albedo map, illumination but also a light transport map that encodes occlusion as nine SH coefficients per pixel. The main difficulty in this inference is the lack of training datasets compared to unlimited variations of human portraits. Surprisingly, geometric information including occlusion can be inferred plausibly even with a small dataset of synthesized human figures, by carefully preparing the dataset so that the CNNs can exploit the data coherency. Our method accomplishes more realistic relighting than the occlusion-ignored formulation.

Supplementary Material

MP4 File (a270-kanamori.mp4)

Download
78.47 MB

References

[1]

Miika Aittala, Timo Aila, and Jaakko Lehtinen. 2016. Reflectance modeling by neural texture synthesis. ACM Trans. Graph. 35, 4 (2016), 65:1--65:13.

Digital Library

[2]

Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. 2005. SCAPE: shape completion and animation of people. ACM Trans. Graph. 24, 3 (2005), 408--416.

Digital Library

[3]

Alexandru O. Balan, Leonid Sigal, Michael J. Black, James E. Davis, and Horst W. Haussecker. 2007. Detailed Human Shape and Pose from Images. In 2007 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2007).

[4]

Jonathan T. Barron and Jitendra Malik. 2015. Shape, Illumination, and Reflectance from Shading. IEEE Trans. Pattern Anal. Mach. Intell. 37, 8 (2015), 1670--1687.

Digital Library

[5]

H. G. Barrow and J. M. Tenenbaum. 1978. Recovering intrinsic scene characteristics from images. Comp. Vis. Sys. (1978).

[6]

Anil S. Baslamisli, Hoang-An Le, and Theo Gevers. 2018. CNN based Learning using Reflection and Retinex Models for Intrinsic Image Decomposition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018).

[7]

R. Basri and D. W. Jacobs. 2003. Lambertian reflectance and linear subspaces. IEEE Transactions on Pattern Analysis and Machine Intelligence 25, 2 (Feb 2003), 218--233.

Digital Library

[8]

Sean Bell, Kavita Bala, and Noah Snavely. 2014. Intrinsic images in the wild. ACM Trans. Graph. 33, 4 (2014), 159:1--159:12.

Digital Library

[9]

Volker Blanz and Thomas Vetter. 1999. A Morphable Model for the Synthesis of 3D Faces. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 1999). 187--194.

Digital Library

[10]

Nicolas Bonneel, Balazs Kovacs, Sylvain Paris, and Kavita Bala. 2017. Intrinsic Decompositions for Image Editing. Comput. Graph. Forum 36, 2 (2017), 593--609.

Digital Library

[11]

Menglei Chai, Linjie Luo, Kalyan Sunkavalli, Nathan Carr, Sunil Hadap, and Kun Zhou. 2015. High-quality hair modeling from a single portrait photo. ACM Trans. Graph. 34, 6 (2015), 204:1--204:10.

Digital Library

[12]

Manmohan Krishna Chandraker and Ravi Ramamoorthi. 2011. What an image reveals about material reflectance. In IEEE International Conference on Computer Vision (ICCV 2011). 1076--1083.

Digital Library

[13]

R. Danerek, Endri Dibra, A. Cengiz Öztireli, Remo Ziegler, and Markus H. Gross. 2017. DeepGarment: 3D Garment Shape Estimation from a Single Image. Comput. Graph. Forum 36, 2 (2017), 269--280.

Digital Library

[14]

Paul Debevec. 2004. Light Probe Image Gallery. (2004). http://www.pauldebevec.com/Probes/.

[15]

Paul E. Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. 2000. Acquiring the reflectance field of a human face. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 2000). 145--156.

Digital Library

[16]

Yuki Endo, Yoshihiro Kanamori, and Jun Mitani. 2017. Deep reverse tone mapping. ACM Trans. Graph. 36, 6 (2017), 177:1--177:10.

Digital Library

[17]

Marc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and Jean-François Lalonde. 2017. Learning to predict indoor illumination from a single image. ACM Trans. Graph. 36, 6 (2017), 176:1--176:14.

Digital Library

[18]

Pablo Garrido, Levi Valgaerts, Chenglei Wu, and Christian Theobalt. 2013. Reconstructing detailed dynamic face geometry from monocular video. ACM Trans. Graph. 32, 6 (2013), 158:1--158:10.

Digital Library

[19]

Stamatios Georgoulis, Konstantinos Rematas, Tobias Ritschel, Efstratios Gavves, Mario Fritz, Luc Van Gool, and Tinne Tuytelaars. 2018. Reflectance and Natural Illumination from Single-Material Specular Objects Using Deep Learning. IEEE Trans. Pattern Anal. Mach. Intell. 40, 8 (2018), 1932--1947.

[20]

Peng Guan, Alexander Weiss, Alexandru O. Balan, and Michael J. Black. 2009. Estimating human shape and pose from a single image. In IEEE 12th International Conference on Computer Vision (ICCV 2009). 1381--1388.

[21]

Yannick Hold-Geoffroy, Kalyan Sunkavalli, Sunil Hadap, Emiliano Gambaretto, and Jean-François Lalonde. 2017. Deep Outdoor Illumination Estimation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017). 2373--2382.

[22]

Berthold K. P. Horn. 1989. Shape from Shading. MIT Press, Chapter Obtaining Shape from Shading Information, 123--171.

Digital Library

[23]

Carlo Innamorati, Tobias Ritschel, Tim Weyrich, and Niloy J. Mitra. 2017. Decomposing Single Images for Layered Photo Retouching. Comput. Graph. Forum 36, 4 (2017), 15--25.

Digital Library

[24]

Micah K. Johnson and Edward H. Adelson. 2011. Shape estimation in natural illumination. In The 24th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2011). 2553--2560.

Digital Library

[25]

Ira Kemelmacher-Shlizerman and Ronen Basri. 2011. 3D Face Reconstruction from a Single Image Using a Single Reference Face Shape. IEEE Trans. Pattern Anal. Mach. Intell. 33, 2 (2011), 394--405.

Digital Library

[26]

Natasha Kholgade, Tomas Simon, Alexei A. Efros, and Yaser Sheikh. 2014. 3D object manipulation in a single photograph using stock 3D models. ACM Trans. Graph. 33, 4 (2014), 127:1--127:12.

Digital Library

[27]

Edwin H. Land and John J. McCann. 1971. Lightness and Retinex Theory. J. Opt. Soc. Am. 61, 1 (Jan 1971), 1--11.

[28]

Guannan Li, Chenglei Wu, Carsten Stoll, Yebin Liu, Kiran Varanasi, Qionghai Dai, and Christian Theobalt. 2013. Capturing Relightable Human Performances under General Uncontrolled Illumination. Comput. Graph. Forum 32, 2 (2013), 275--284.

[29]

Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2017. Modeling surface appearance from a single photograph using self-augmented convolutional neural networks. ACM Trans. Graph. 36, 4 (2017), 45:1--45:11.

Digital Library

[30]

Jorge Lopez-Moreno, Elena Garces, Sunil Hadap, Erik Reinhard, and Diego Gutierrez. 2013. Multiple Light Source Estimation in a Single Image. Comput. Graph. Forum 32, 8 (2013), 170--182.

[31]

Zhaoliang Lun, Matheus Gadelha, Evangelos Kalogerakis, Subhransu Maji, and Rui Wang. 2017. 3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks. In 2017 International Conference on 3D Vision (3DV 2017).

[32]

Takuya Narihira, Michael Maire, and Stella X. Yu. 2015. Direct Intrinsics: Learning Albedo-Shading Decomposition by Convolutional Regression. In 2015 IEEE International Conference on Computer Vision (ICCV 2015). 2992.

Digital Library

[33]

Geoffrey Oxholm and Ko Nishino. 2012. Shape and Reflectance from Natural Illumination. In 12th European Conference on Computer Vision (ECCV 2012), Proceedings, Part I. 528--541.

Digital Library

[34]

Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. 2009. A 3D Face Model for Pose and Illumination Invariant Face Recognition. In Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS 2009). 296--301.

Digital Library

[35]

Ravi Ramamoorthi and Pat Hanrahan. 2001. An efficient representation for irradiance environment maps. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2001. 497--500.

Digital Library

[36]

Andreas Schneider, Sandro Schönborn, Bernhard Egger, Lavrenti Frobeen, and Thomas Vetter. 2017. Efficient Global Illumination for Morphable Models. In IEEE International Conference on Computer Vision (ICCV 2017). 3885--3893.

[37]

Soumyadip Sengupta, Angjoo Kanazawa, Carlos D. Castillo, and David W. Jacobs. 2018. SfSNet: Learning Shape, Reflectance and Illuminance of Faces 'in the Wild'. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018).

[38]

Jian Shi, Yue Dong, Hao Su, and Stella X. Yu. 2017. Learning Non-Lambertian Object Intrinsics Across ShapeNet Categories. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017). 5844--5853.

[39]

Zhixin Shu, Sunil Hadap, Eli Shechtman, Kalyan Sunkavalli, Sylvain Paris, and Dimitris Samaras. 2017a. Portrait Lighting Transfer Using a Mass Transport Approach. ACM Trans. Graph. 36, 4, Article 145a (Oct. 2017).

Digital Library

[40]

Zhixin Shu, Ersin Yumer, Sunil Hadap, Kalyan Sunkavalli, Eli Shechtman, and Dimitris Samaras. 2017b. Neural Face Editing with Intrinsic Image Disentangling. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017). 5444--5453.

[41]

Peter-Pike J. Sloan, Jan Kautz, and John Snyder. 2002. Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. ACM Trans. Graph. 21, 3 (2002), 527--536.

Digital Library

[42]

Ayush Tewari, Michael Zollhöfer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Pérez, and Christian Theobalt. 2017. MoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction. In IEEE International Conference on Computer Vision (ICCV 2017). 3735--3744.

[43]

Su Xue, Aseem Agarwala, Julie Dorsey, and Holly E. Rushmeier. 2012. Understanding and improving the realism of image composites. ACM Trans. Graph. 31, 4 (2012), 84:1--84:10.

Digital Library

[44]

Shugo Yamaguchi, Shunsuke Saito, Koki Nagano, Yajie Zhao, Weikai Chen, Kyle Olszewski, Shigeo Morishima, and Hao Li. 2018. High-fidelity Facial Reflectance and Geometry Inference from an Unconstrained Image. ACM Trans. Graph. 37, 4, Article 162 (July 2018), 162:1--162:14 pages.

Digital Library

[45]

Chao Zhang, Sergi Pujades, Michael J. Black, and Gerard Pons-Moll. 2017. Detailed, Accurate, Human Shape Estimation from Clothed 3D Scan Sequences. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017). 5484--5493.

[46]

Bin Zhou, Xiaowu Chen, Qiang Fu, Kan Guo, and Ping Tan. 2013. Garment Modeling from a Single Image. Comput. Graph. Forum 32, 7 (2013), 85--91.

[47]

Sergey Zhukov, Andrei Iones, and Grigorij Kronin. 1998. An Ambient Light Illumination Model. In Rendering Techniques '98, Proceedings of the Eurographics Workshop. 45--56.

Cited By

Xie RHuang KCho IYang SChen WBao HZheng WLi RHuo Y(2024)ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human RenderingACM Transactions on Graphics10.1145/3678002Online publication date: 15-Jul-2024
https://doi.org/10.1145/3678002
Zeng CDong YPeers PKong YWu HTong X(2024)DiLightNet: Fine-grained Lighting Control for Diffusion-based Image GenerationACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657396(1-12)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657396
Poirier‐Ginter YGauthier APhillip JLalonde JDrettakis G(2024)A Diffusion Approach to Radiance Field Relighting using Multi‐Illumination SynthesisComputer Graphics Forum10.1111/cgf.1514743:4Online publication date: 24-Jul-2024
https://doi.org/10.1111/cgf.15147
Show More Cited By

Index Terms

Relighting humans: occlusion-aware inverse rendering for full-body human images
1. Computing methodologies
  1. Computer graphics
    1. Image manipulation
      1. Image-based rendering
    2. Rendering

Recommendations

Self-supervised Outdoor Scene Relighting
Computer Vision – ECCV 2020
Abstract
Outdoor scene relighting is a challenging problem that requires good understanding of the scene geometry, illumination and albedo. Current techniques are completely supervised, requiring high quality synthetic renderings to train a solution. Such ...
SOL-NeRF: Sunlight Modeling for Outdoor Scene Decomposition and Relighting
SA '23: SIGGRAPH Asia 2023 Conference Papers

Outdoor scenes often involve large-scale geometry and complex unknown lighting conditions, making it difficult to decompose them into geometry, reflectance and illumination. Recently researchers made attempts to decompose outdoor scenes using Neural ...
Image-based rendering of diffuse, specular and glossy surfaces from a single image
SIGGRAPH '01: Proceedings of the 28th annual conference on Computer graphics and interactive techniques

In this paper, we present a new method to recover an approximation of the bidirectional reflectance distribution function (BRDF) of the surfaces present in a real scene. This is done from a single photograph and a 3D geometric model of the scene. The ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 37, Issue 6

December 2018

1401 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3272127

Editor:
Takeo Igarashi
The University of Tokyo, Japan

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 December 2018

Published in TOG Volume 37, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

47
Total Citations
View Citations
315
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)2

Reflects downloads up to 03 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Xie RHuang KCho IYang SChen WBao HZheng WLi RHuo Y(2024)ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human RenderingACM Transactions on Graphics10.1145/3678002Online publication date: 15-Jul-2024
https://doi.org/10.1145/3678002
Zeng CDong YPeers PKong YWu HTong X(2024)DiLightNet: Fine-grained Lighting Control for Diffusion-based Image GenerationACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657396(1-12)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657396
Poirier‐Ginter YGauthier APhillip JLalonde JDrettakis G(2024)A Diffusion Approach to Radiance Field Relighting using Multi‐Illumination SynthesisComputer Graphics Forum10.1111/cgf.1514743:4Online publication date: 24-Jul-2024
https://doi.org/10.1111/cgf.15147
Hu ZNsampi NWang XWang Q(2024)SIDNet: Learning Shading-Aware Illumination Descriptor for Image HarmonizationIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2024.33524138:2(1290-1302)Online publication date: Apr-2024
https://doi.org/10.1109/TETCI.2024.3352413
Hu ZLi JWang XWang Q(2024)Spatially-Varying Illumination-Aware Indoor HarmonizationInternational Journal of Computer Vision10.1007/s11263-024-01994-z132:7(2473-2492)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1007/s11263-024-01994-z
Bieron JTong XPeers P(2023)Single Image Neural Material RelightingACM SIGGRAPH 2023 Conference Proceedings10.1145/3588432.3591515(1-11)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1145/3588432.3591515
Zeng CChen GDong YPeers PWu HTong X(2023)Relighting Neural Radiance Fields with Shadow and Highlight HintsACM SIGGRAPH 2023 Conference Proceedings10.1145/3588432.3591482(1-11)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1145/3588432.3591482
Li ZSong LChen ZDu XChen LYuan JXu YEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Relit-NeuLF: Efficient Relighting and Novel View Synthesis via Neural 4D Light FieldProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612160(7007-7016)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612160
Yang XTaketomi TKanamori Y(2023)Makeup Extraction of 3D Representation via Illumination‐Aware Image DecompositionComputer Graphics Forum10.1111/cgf.1476242:2(293-307)Online publication date: 23-May-2023
https://doi.org/10.1111/cgf.14762
Weir JZhao JChalmers ARhee T(2023)De-lighting Human Images Using Region-Specific Data Augmentation2023 38th International Conference on Image and Vision Computing New Zealand (IVCNZ)10.1109/IVCNZ61134.2023.10344007(1-6)Online publication date: 29-Nov-2023
https://doi.org/10.1109/IVCNZ61134.2023.10344007
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents