research-article

Learning a Generalized Physical Face Model From Data

Authors:

Prashanth Chandran,

Barbara Solenthaler,

Eftychios Sifakis,

Derek BradleyAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 43, Issue 4

Article No.: 94, Pages 1 - 14

https://doi.org/10.1145/3658189

Published: 19 July 2024 Publication History

Abstract

Physically-based simulation is a powerful approach for 3D facial animation as the resulting deformations are governed by physical constraints, allowing to easily resolve self-collisions, respond to external forces and perform realistic anatomy edits. Today's methods are data-driven, where the actuations for finite elements are inferred from captured skin geometry. Unfortunately, these approaches have not been widely adopted due to the complexity of initializing the material space and learning the deformation model for each character separately, which often requires a skilled artist followed by lengthy network training. In this work, we aim to make physics-based facial animation more accessible by proposing a generalized physical face model that we learn from a large 3D face dataset. Once trained, our model can be quickly fit to any unseen identity and produce a ready-to-animate physical face model automatically. Fitting is as easy as providing a single 3D face scan, or even a single face image. After fitting, we offer intuitive animation controls, as well as the ability to retarget animations across characters. All the while, the resulting animations allow for physical effects like collision avoidance, gravity, paralysis, bone reshaping and more.

References

[1]

Michael Bao, Matthew Cong, Stéphane Grabli, and Ronald Fedkiw. 2018. High-Quality Face Capture Using Anatomical Muscles. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (12 2018). http://arxiv.org/abs/1812.02836

[2]

Volker Blanz and Thomas Vetter. 1999. A morphable model for the synthesis of 3D faces. In Siggraph, Vol. 99. 187--194.

[3]

Sofien Bouaziz, Sebastian Martin, Tiantian Liu, Ladislav Kavan, and Mark Pauly. 2014. Projective dynamics: fusing constraint projections for fast simulation. ACM Transactions on Graphics 33, 4 (7 2014), 1--11.

Digital Library

[4]

Zenghao Chai, Haoxian Zhang, Jing Ren, Di Kang, Zhengzhuo Xu, Xuefei Zhe, Chun Yuan, and Linchao Bao. 2022. REALY: Rethinking the Evaluation of 3D Face Reconstruction. In Proceedings of the European Conference on Computer Vision (ECCV).

Digital Library

[5]

Prashanth Chandran, Derek Bradley, Markus Gross, and Thabo Beeler. 2020. Semantic Deep Face Models. In 2020 International Conference on 3D Vision (3DV). IEEE, 345--354.

[6]

Prashanth Chandran, Loïc Ciccone, Markus Gross, and Derek Bradley. 2022. Local Anatomically-Constrained Facial Performance Retargeting. ACM Trans. Graph. 41, 4, Article 168 (jul 2022).

Digital Library

[7]

Prashanth Chandran and Gaspard Zoss. 2023. Anatomically Constrained Implicit Face Models. arXiv:2312.07538 [cs.GR]

[8]

Prashanth Chandran, Gaspard Zoss, Paulo Gotardo, and Derek Bradley. 2023. Continuous Landmark Detection With 3D Queries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16858--16867.

[9]

Byungkuk Choi, Haekwang Eom, Benjamin Mouscadet, Stephen Cullingford, Kurt Ma, Stefanie Gassel, Suzi Kim, Andrew Moffat, Millicent Maier, Marco Revelant, Joe Letteri, and Karan Singh. 2022. Animatomy: An Animator-Centric, Anatomically Inspired System for 3D Facial Modeling, Animation and Transfer. In SIGGRAPH Asia 2022 Conference Papers. Association for Computing Machinery, Article 16, 9 pages.

[10]

Hang Dai, Nick Pears, William Smith, and Christian Duncan. 2019. Statistical Modeling of Craniofacial Shape and Texture. International Journal of Computer Vision (2019).

[11]

Radek Danecek, Michael J. Black, and Timo Bolkart. 2022. EMOCA: Emotion Driven Monocular Face Capture and Animation. In Conference on Computer Vision and Pattern Recognition (CVPR). 20311--20322.

[12]

Bernhard Egger, William A. P. Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, Christian Theobalt, Volker Blanz, and Thomas Vetter. 2020. 3D Morphable Face Models---Past, Present, and Future. ACM Transactions on Graphics 39, 5 (10 2020), 1--38.

Digital Library

[13]

Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart. 2021. Learning an Animatable Detailed 3D Face Model from In-the-Wild Images. ACM Transactions on Graphics (ToG), Proc. SIGGRAPH 40, 4 (Aug. 2021), 88:1--88:13.

[14]

Panagiotis P. Filntisis, George Retsinas, Foivos Paraperas-Papantoniou, Athanasios Katsamanis, Anastasios Roussos, and Petros Maragos. 2022. Visual Speech-Aware Perceptual 3D Facial Expression Reconstruction from Videos. arXiv preprint arXiv:2207.11094 (2022).

[15]

Baris Gecer, Stylianos Ploumpis, Irene Kotsia, and Stefanos Zafeiriou. 2019. GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]

Baris Gecer, Stylianos Ploumpis, Irene Kotsia, and Stefanos P Zafeiriou. 2021. Fast-GANFIT: Generative Adversarial Network for High Fidelity 3D Face Reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).

[17]

Aurel Gruber, Marco Fratarcangeli, Gaspard Zoss, Roman Cattaneo, Thabo Beeler, Markus Gross, and Derek Bradley. 2020. Interactive Sculpting of Digital Faces Using an Anatomical Modeling Paradigm. Computer Graphics Forum (2020), 93--102.

[18]

Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 (2016).

[19]

Alexandru-Eugen Ichim, Petr Kadleček, Ladislav Kavan, and Mark Pauly. 2017. Phace: Physics-based Face Modeling and Animation. ACM Transactions on Graphics 36, 4 (7 2017), 1--14.

Digital Library

[20]

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7--9, 2015, Conference Track Proceedings (12 2015). http://arxiv.org/abs/1412.6980

[21]

Tobias Kirschstein, Shenhan Qian, Simon Giebenhain, Tim Walter, and Matthias Nießner. 2023. Nersemble: Multi-view radiance field reconstruction of human heads. ACM Transactions on Graphics (TOG) 42, 4 (2023), 1--14.

Digital Library

[22]

Gergely Klár, Andrew Moffat, Ken Museth, and Eftychios Sifakis. 2020. Shape Targeting: A Versatile Active Elasticity Constitutive Model. In ACM SIGGRAPH 2020 Talks (Virtual Event, USA) (SIGGRAPH '20). Association for Computing Machinery, New York, NY, USA, Article 59, 2 pages.

Digital Library

[23]

Bo Li, Lingchen Yang, and Barbara Solenthaler. 2023. Efficient Incremental Potential Contact for Actuated Face Simulation. In SIGGRAPH Asia 2023 Technical Communications. 1--4.

[24]

Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. 2017. Learning a model of facial shape and expression from 4D scans. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia) 36, 6 (2017).

Digital Library

[25]

Hsueh-Ti Derek Liu, Francis Williams, Alec Jacobson, Sanja Fidler, and Or Litany. 2022. Learning Smooth Neural Functions via Lipschitz Regularization. arXiv preprint arXiv:2202.08345 (2022).

[26]

Araceli Morales, Gemma Piella, and Federico M. Sukno. 2020. Survey on 3D face reconstruction from uncalibrated images. Comput. Sci. Rev. 40 (2020), 100400.

Digital Library

[27]

Christopher Otto, Prashanth Chandran, Gaspard Zoss, Markus Gross, Paulo Gotardo, and Derek Bradley. 2023. A Perceptual Shape Loss for Monocular 3D Face Reconstruction. arXiv:2310.19580 [cs.CV]

[28]

Zesong Qiu, Yuwei Li, Dongming He, Qixuan Zhang, Longwen Zhang, Yinghao Zhang, Jingya Wang, Lan Xu, Xudong Wang, Yuyao Zhang, et al. 2022. SCULPTOR: Skeleton-consistent face creation using a learned parametric generator. ACM Transactions on Graphics (TOG) 41, 6 (2022), 1--17.

Digital Library

[29]

Maziar Raissi, Paris Perdikaris, and George E Karniadakis. 2019. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics 378 (2019), 686--707.

[30]

Eftychios Sifakis, Igor Neverov, and Ronald Fedkiw. 2005. Automatic determination of facial muscle activations from sparse motion capture marker data. ACM Transactions on Graphics 24, 3 (7 2005), 417--425.

Digital Library

[31]

Vincent Sitzmann, Julien N P Martel, Alexander W Bergman, David B Lindell, and Gordon Wetzstein. 2020. Implicit Neural Representations with Periodic Activation Functions. In NeurIPS 2020, Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). https://proceedings.neurips.cc/paper/2020/hash/53c04118df112c13a8c34b38343b9c10-Abstract.html

[32]

Sangeetha Grama Srinivasan, Qisi Wang, Junior Rojas, Gergely Klár, Ladislav Kavan, and Eftychios Sifakis. 2021. Learning active quasistatic physics-based models from data. ACM Transactions on Graphics 40, 4 (8 2021), 1--14.

Digital Library

[33]

Nicolas Wagner, Mario Botsch, and Ulrich Schwanecke. 2023. SoftDECA: Computationally Efficient Physics-Based Facial Animations. In Proceedings of the 16th ACM SIGGRAPH Conference on Motion, Interaction and Games (MIG '23). Association for Computing Machinery, New York, NY, USA, Article 11, 11 pages.

Digital Library

[34]

Chenglei Wu, Derek Bradley, Markus Gross, and Thabo Beeler. 2016. An anatomically-constrained local deformation model for monocular face capture. ACM Transactions on Graphics 35, 4 (7 2016), 1--12.

Digital Library

[35]

Ming Xu and James Yang. 2015. Human facial soft tissue thickness and mechanical properties: a literature review. In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Vol. 57045. American Society of Mechanical Engineers, V01AT02A045.

[36]

Haotian Yang, Hao Zhu, Yanru Wang, Mingkai Huang, Qiu Shen, Ruigang Yang, and Xun Cao. 2020a. FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]

Haotian Yang, Hao Zhu, Yanru Wang, Mingkai Huang, Qiu Shen, Ruigang Yang, and Xun Cao. 2020b. Facescape: a large-scale high quality 3d face dataset and detailed riggable 3d face prediction. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition. 601--610.

[38]

Lingchen Yang, Byungsoo Kim, Gaspard Zoss, Baran Gözcü, Markus Gross, and Barbara Solenthaler. 2022. Implicit neural representation for physics-driven actuated soft bodies. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1--10.

Digital Library

[39]

Lingchen Yang, Gaspard Zoss, Prashanth Chandran, Paulo Gotardo, Markus Gross, Barbara Solenthaler, Eftychios Sifakis, and Derek Bradley. 2023. An Implicit Physical Face Model Driven by Expression and Style. In SIGGRAPH Asia 2023 Conference Papers. Article 106, 12 pages.

Digital Library

[40]

Tianke Zhang, Xuangeng Chu, Yunfei Liu, Lijian Lin, Zhendong Yang, Zhengzhuo Xu, Chengkun Cao, Fei Yu, Changyin Zhou, Chun Yuan, and Yu Li. 2023. Accurate 3D Face Reconstruction with Facial Component Tokens. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 9033--9042.

[41]

Mingwu Zheng, Hongyu Yang, Di Huang, and Liming Chen. 2022. Imface: A nonlinear 3d morphable face model with implicit neural representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 20343--20352.

[42]

Wojciech Zielonka, Timo Bolkart, and Justus Thies. 2022. Towards Metrical Reconstruction of Human Faces. In European Conference on Computer Vision.

Cited By

Hou JWeng DZhao ZLi YZhou J(2024)Neutral Facial Rigging from Limited Spatiotemporal MeshesElectronics10.3390/electronics1313244513:13(2445)Online publication date: 21-Jun-2024
https://doi.org/10.3390/electronics13132445
Büchner TPenzel NGuntinas-Lichius ODenzler J(2024)Facing Asymmetry - Uncovering the Causal Link Between Facial Symmetry and Expression Classifiers Using Synthetic InterventionsComputer Vision – ACCV 202410.1007/978-981-96-0911-6_26(443-464)Online publication date: 8-Dec-2024
https://doi.org/10.1007/978-981-96-0911-6_26

Recommendations

Data-Driven Expressive 3D Facial Animation Synthesis for Digital Humans
SA '23: SIGGRAPH Asia 2023 Doctoral Consortium

This doctoral research focuses on generating expressive 3D facial animation for digital humans by studying and employing data-driven techniques. Face is the first point of interest during human interaction, and it is not any different for interacting ...
An Implicit Physical Face Model Driven by Expression and Style
SA '23: SIGGRAPH Asia 2023 Conference Papers

3D facial animation is often produced by manipulating facial deformation models (or rigs), that are traditionally parameterized by expression controls. A key component that is usually overlooked is expression “style", as in, how a particular expression ...
FaceXHuBERT: Text-less Speech-driven E(X)pressive 3D Facial Animation Synthesis Using Self-Supervised Speech Representation Learning
ICMI '23: Proceedings of the 25th International Conference on Multimodal Interaction

This paper presents FaceXHuBERT, a text-less speech-driven 3D facial animation generation method that generates facial cues driven by an emotional expressiveness condition. In addition, it can handle audio recorded in a variety of situations (e.g. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 43, Issue 4

July 2024

1774 pages

EISSN:1557-7368

DOI:10.1145/3675116

Issue’s Table of Contents

Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 July 2024

Published in TOG Volume 43, Issue 4

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
172
Total Downloads

Downloads (Last 12 months)172
Downloads (Last 6 weeks)19

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Hou JWeng DZhao ZLi YZhou J(2024)Neutral Facial Rigging from Limited Spatiotemporal MeshesElectronics10.3390/electronics1313244513:13(2445)Online publication date: 21-Jun-2024
https://doi.org/10.3390/electronics13132445
Büchner TPenzel NGuntinas-Lichius ODenzler J(2024)Facing Asymmetry - Uncovering the Causal Link Between Facial Symmetry and Expression Classifiers Using Synthetic InterventionsComputer Vision – ACCV 202410.1007/978-981-96-0911-6_26(443-464)Online publication date: 8-Dec-2024
https://doi.org/10.1007/978-981-96-0911-6_26

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents