research-article

Open access

Model-Based Deep Portrait Relighting

Authors:

Frederik David Schreiber,

Peter EisertAuthors Info & Claims

CVMP '22: Proceedings of the 19th ACM SIGGRAPH European Conference on Visual Media Production

Article No.: 7, Pages 1 - 9

https://doi.org/10.1145/3565516.3565526

Published: 01 December 2022 Publication History

All formats PDF

Abstract

Like most computer vision problems the relighting of portrait face images is more and more being entirely formulated as a deep learning problem. However, data-driven approaches need a detailed and exhaustive database to work on and the creation of ground truth data is tedious and oftentimes technically complex. At the same time, networks get bigger and deeper. Knowledge about the problem statement, scene structure, and physical laws are often neglected. In this paper, we propose to encompass prior knowledge for relighting directly in the network learning process, adding model-based building blocks to the training. Thereby, we improve the learning speed and effectiveness of the network, thus performing better even with a restricted dataset. We demonstrate through an ablation study that the proposed model-based building blocks improve the network’s training and enhance the generated images compared with the naive approach.

References

[1]

Sai Bi, Stephen Lombardi, Shunsuke Saito, Tomas Simon, Shih-En Wei, Kevyn Mcphail, Ravi Ramamoorthi, Yaser Sheikh, and Jason Saragih. 2021. Deep relightable appearance models for animatable faces. ACM Transactions on Graphics (TOG) 40, 4 (2021), 1–15.

Digital Library

[2]

Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. 2017. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587(2017).

[3]

Haiwen Feng, Timo Bolkart, Joachim Tesch, Michael Black, and Victoria Abrevaya. 2022. On Fairness in Face Albedo Estimation. In ACM SIGGRAPH 2022 Talks (Vancouver, BC, Canada) (SIGGRAPH ’22). Association for Computing Machinery, New York, NY, USA, Article 4, 2 pages. https://doi.org/10.1145/3532836.3536281

Digital Library

[4]

Ralph Gross, Iain Matthews, Jeffrey Cohn, Takeo Kanade, and Simon Baker. 2010. Multi-PIE. Image Vision Comput. 28, 5 (may 2010), 807–813. https://doi.org/10.1016/j.imavis.2009.08.002

Digital Library

[5]

Andrew Hou, Michel Sarkis, Ning Bi, Yiying Tong, and Xiaoming Liu. 2022. Face Relighting with Geometrically Consistent Shadows. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4217–4226.

[6]

Andrew Hou, Ze Zhang, Michel Sarkis, Ning Bi, Yiying Tong, and Xiaoming Liu. 2021. Towards High Fidelity Face Relighting with Realistic Shadows. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 14714–14723. https://doi.org/10.1109/CVPR46437.2021.01448

[7]

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image-To-Image Translation With Conditional Adversarial Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]

Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2018. Progressive Growing of GANs for Improved Quality, Stability, and Variation. In International Conference on Learning Representations. https://openreview.net/forum?id=Hk99zCeAb

[9]

Tero Karras, Samuli Laine, and Timo Aila. 2019. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 4401–4410.

[10]

Davis E. King. 2009. Dlib-ml: A Machine Learning Toolkit. Journal of Machine Learning Research 10 (2009), 1755–1758.

Digital Library

[11]

Davis E King. 2015. Max-margin object detection. arXiv preprint arXiv:1502.00046(2015).

[12]

Thomas Nestmeyer, Jean-François Lalonde, Iain Matthews, and Andreas Lehrmann. 2020. Learning physics-guided face relighting under directional light. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5124–5133.

[13]

Rohit Pandey, Sergio Orts-Escolano, Chloe LeGendre, Christian Haene, Sofien Bouaziz, Christoph Rhemann, Paul Debevec, and Sean Fanello. 2021. Total Relighting: Learning to Relight Portraits for Background Replacement. ACM Transactions on Graphics (Proceedings SIGGRAPH) 40, 4 (August 2021). https://doi.org/10.1145/3450626.3459872

Digital Library

[14]

Christoph Rhemann, Graham Fyffe, Jay Busch, Jonathan T. Barron, Paul Debevec, Ravi Ramamoorthi, Tiancheng Sun, Xueming Yu, Yun-Ta Tsai, and Zexiang Xu. 2019. Single Image Portrait Relighting. In SIGGRAPH.

[15]

O. Ronneberger, P.Fischer, and T. Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI)(LNCS, Vol. 9351). Springer, 234–241. http://lmb.informatik.uni-freiburg.de/Publications/2015/RFB15a(available on arXiv:1505.04597 [cs.CV]).

[16]

Leonid I. Rudin, Stanley Osher, and Emad Fatemi. 1992. Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena 60, 1 (1992), 259–268. https://doi.org/10.1016/0167-2789(92)90242-F

Digital Library

[17]

Soumyadip Sengupta, Angjoo Kanazawa, Carlos D Castillo, and David W Jacobs. 2018. Sfsnet: Learning shape, reflectance and illuminance of facesin the wild’. In Proceedings of the IEEE conference on computer vision and pattern recognition. 6296–6305.

[18]

Yun-Ta Tsai and Rohit Panday. 2020. Portrait Light: Enhancing Portrait Lighting with Machine Learning. Retrieved July 29, 2022 from https://ai.googleblog.com/2020/12/portrait-light-enhancing-portrait.html

[19]

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600–612.

Digital Library

[20]

Hao Zhou, Sunil Hadap, Kalyan Sunkavalli, and David W Jacobs. 2019. Deep single-image portrait relighting. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 7194–7202.

Cited By

Lin AGhosh A(2024)Optimal OLAT Alignment for Image Based Relighting with Color-Multiplexed OLAT SequenceProceedings of 21st ACM SIGGRAPH Conference on Visual Media Production10.1145/3697294.3697297(1-7)Online publication date: 18-Nov-2024
https://dl.acm.org/doi/10.1145/3697294.3697297

Index Terms

Model-Based Deep Portrait Relighting
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
        Scene understanding
  2. Computer graphics
    1. Image manipulation
      1. Image-based rendering

Recommendations

Total relighting: learning to relight portraits for background replacement

We propose a novel system for portrait relighting and background replacement, which maintains high-frequency boundary details and accurately synthesizes the subject's appearance as lit by novel illumination, thereby producing realistic composite images ...
Single image portrait relighting via explicit multiple reflectance channel modeling

Portrait relighting aims to render a face image under different lighting conditions. Existing methods do not explicitly consider some challenging lighting effects such as specular and shadow, and thus may fail in handling extreme lighting conditions. In ...
Single image portrait relighting

Lighting plays a central role in conveying the essence and depth of the subject in a portrait photograph. Professional photographers will carefully control the lighting in their studio to manipulate the appearance of their subject, while consumer ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CVMP '22: Proceedings of the 19th ACM SIGGRAPH European Conference on Visual Media Production

December 2022

97 pages

ISBN:9781450399395

DOI:10.1145/3565516

Editors:
Marco Volino
University of Surrey, UK
,
Rafał Mantiuk
University of Cambridge, UK
,
Armin Mustafa
University of Surrey, UK
,
Yulia Gryaditskaya
University of Surrey, UK

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 December 2022

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Bundesministerium für Bildung und Forschung

Conference

CVMP '22

Sponsor:

SIGGRAPH

CVMP '22: European Conference on Visual Media Production

December 1 - 2, 2022

London, United Kingdom

Acceptance Rates

Overall Acceptance Rate 40 of 67 submissions, 60%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
556
Total Downloads

Downloads (Last 12 months)251
Downloads (Last 6 weeks)25

Reflects downloads up to 22 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Lin AGhosh A(2024)Optimal OLAT Alignment for Image Based Relighting with Color-Multiplexed OLAT SequenceProceedings of 21st ACM SIGGRAPH Conference on Visual Media Production10.1145/3697294.3697297(1-7)Online publication date: 18-Nov-2024
https://dl.acm.org/doi/10.1145/3697294.3697297

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten