research-article

Animatomy: an Animator-centric, Anatomically Inspired System for 3D Facial Modeling, Animation and Transfer

Authors:

Benjamin Mouscadet,

Stephen Cullingford,

Stefanie Gassel,

Millicent Maier,

Marco Revelant,

Karan SinghAuthors Info & Claims

SA '22: SIGGRAPH Asia 2022 Conference Papers

Article No.: 16, Pages 1 - 9

https://doi.org/10.1145/3550469.3555398

Published: 30 November 2022 Publication History

Abstract

We present Animatomy, a novel anatomic+animator centric representation of the human face. Present FACS-based systems are plagued with problems of face muscle separation, coverage, opposition, and redundancy. We, therefore, propose a collection of muscle fiber curves as an anatomic basis, whose contraction and relaxation provide us with a fine-grained parameterization of human facial expression. We build an end-to-end modular deformation architecture using this representation that enables: automatic optimization of the parameters of a specific face from high-quality dynamic facial scans; face animation driven by performance capture, keyframes, or dynamic simulation; interactive and direct manipulation of facial expression; and animation transfer from an actor to a character. We validate our facial system by showing compelling animated results, applications, and a quantitative comparison of our facial reconstruction to ground truth performance capture. Our system is being intensively used by a large creative team on Avatar: The Way of Water. We report feedback from these users as qualitative evaluation of our system.

References

[1]

3Dflow. 2022. 3DF Zephyr. http://3dflow.net/3df-zephyr-photogrammetry-software/

[2]

Rinat Abdrashitov, Fanny Chevalier, and Karan Singh. 2020. Interactive Exploration and Refinement of Facial Expression Using Manifold Learning. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology.

Digital Library

[3]

Stephen W. Bailey, Dalton Omens, Paul Dilorenzo, and James F. O’Brien. 2020. Fast and Deep Facial Deformations. ACM Trans. Graph. 39(2020).

[4]

Michael Bao, Matthew Cong, Stephane Grabli, and Ronald Fedkiw. 2019. High-Quality Face Capture Using Anatomical Muscles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]

Vincent Barrielle, Nicolas Stoiber, and Cédric Cagniart. 2016. BlendForces: A Dynamic Framework for Facial Animation. Computer Graphics Forum 35 (2016).

[6]

Thabo Beeler and Derek Bradley. 2014. Rigid Stabilization of Facial Expressions. ACM Trans. Graph. 33(2014).

[7]

James Booth, Anastasios Roussos, Stefanos Zafeiriou, Allan Ponniah, and David Dunaway. 2016. A 3D Morphable Model Learnt from 10,000 Faces. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]

Ozan Cetinaslan, Verónica Orvalho, and J.P. Lewis. 2015. Sketch-Based Controllers for Blendshape Facial Animation. In EG 2015 - Short Papers.

[9]

Prashanth Chandran, Derek Bradley, Markus Gross, and Thabo Beeler. 2020. Semantic Deep Face Models. In 2020 International Conference on 3D Vision (3DV).

[10]

Prashanth Chandran, Loïc Ciccone, Markus Gross, and Derek Bradley. 2022a. Local Anatomically-Constrained Facial Performance Retargeting. ACM Trans. Graph. 41(2022).

[11]

Prashanth Chandran, Gaspard Zoss, Markus Gross, Paulo Gotardo, and Derek Bradley. 2022b. Shape Transformers: Topology-Independent 3D Shape Models Using Transformers. Computer Graphics Forum 41 (2022).

[12]

Hang Dai, Nick Pears, William Smith, and Christian Duncan. 2020. Statistical Modeling of Craniofacial Shape and Texture. International Journal of Computer Vision 128 (2020).

Digital Library

[13]

Sven De Greef, Peter Claes, Dirk Vandermeulen, Wouter Mollemans, Paul Suetens, and Guy Willems. 2006. Large-scale in-vivo Caucasian facial soft tissue thickness database for craniofacial reconstruction. Forensic science international 159 (2006).

[14]

Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. 2000. Acquiring the Reflectance Field of a Human Face. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques.

Digital Library

[15]

Zhigang Deng and Junyong Noh. 2008. Computer Facial Animation: A Survey. Springer London.

[16]

Bernhard Egger, William A. P. Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, Christian Theobalt, Volker Blanz, and Thomas Vetter. 2020. 3D Morphable Face Models—Past, Present, and Future. ACM Trans. Graph. 39(2020).

[17]

Paul Ekman and Wallace V. Friesen. 1978. Facial Action Coding System: A Technique for the Measurement of Facial Movement. (1978).

[18]

Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart. 2021. Learning an Animatable Detailed 3D Face Model from In-the-Wild Images. ACM Trans. Graph. 40(2021).

[19]

Philip-William Grassal, Malte Prinzler, Titus Leistner, Carsten Rother, Matthias Nießner, and Justus Thies. 2021. Neural Head Avatars from Monocular RGB Videos.

[20]

Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or. 2019. MeshCNN: A Network with an Edge. ACM Trans. Graph. 38(2019).

[21]

C.H. Hjortsjö and W.F. Salisbury. 1970. Man’s face and mimic language.

[22]

Alexandru-Eugen Ichim, Petr Kadleček, Ladislav Kavan, and Mark Pauly. 2017. Phace: Physics-Based Face Modeling and Animation. ACM Trans. Graph. 36(2017).

Digital Library

[23]

Zi-Hang Jiang, Qianyi Wu, Keyu Chen, and Juyong Zhang. 2019. Disentangled Representation Learning for 3D Face Shape. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]

Ira Kemelmacher-Shlizerman and Steven M. Seitz. 2011. Face Reconstruction in the Wild. In 2011 International Conference on Computer Vision.

[25]

Joonho Kim and Karan Singh. 2021. Optimizing UI Layouts for Deformable Face-Rig Manipulation. ACM Trans. Graph. 40(2021).

[26]

Seonghyeon Kim, Sunjin Jung, Kwanggyoon Seo, Roger Blanco i Ribera, and Junyong Noh. 2021. Deep Learning-Based Unsupervised Human Facial Retargeting. Computer Graphics Forum 40 (2021).

[27]

Steve Lesser, Alexey Stomakhin, Gilles Daviet, Joel Wretborn, John Edholm, Noh-Hoon Lee, Eston Schweickart, Xiao Zhai, Sean Flynn, and Andrew Moffat. 2022. Loki: a unified multiphysics simulation framework for production. ACM Trans. Graph. 41(2022).

[28]

J.P. Lewis, Ken Anjyo, Taehyun Rhee, Mengjie Zhang, Fred Pighin, and Zhigang Deng. 2014. Practice and Theory of Blendshape Facial Models. In Eurographics 2014 - State of the Art Reports.

[29]

J.P. Lewis and Ken-ichi Anjyo. 2010. Direct Manipulation Blendshapes. IEEE Computer Graphics and Applications 30 (2010).

Digital Library

[30]

Hao Li, Jihun Yu, Yuting Ye, and Chris Bregler. 2013. Realtime Facial Animation with On-the-Fly Correctives. ACM Trans. Graph. 32(2013).

[31]

Ruilong Li, Karl Bladin, Yajie Zhao, Chinmay Chinara, Owen Ingraham, Pengda Xiang, Xinglei Ren, Pratusha Prasad, Bipin Kishore, Jun Xing, and Hao Li. 2020. Learning Formation of Physically-Based Face Attributes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[32]

Tianye Li, Timo Bolkart, Michael J. Black, Hao Li, and Javier Romero. 2017. Learning a Model of Facial Shape and Expression from 4D Scans. ACM Trans. Graph. 36(2017).

[33]

Stephen Lombardi, Jason Saragih, Tomas Simon, and Yaser Sheikh. 2018. Deep Appearance Models for Face Rendering. ACM Trans. Graph. 37(2018).

[34]

Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. 2015. SMPL: A Skinned Multi-Person Linear Model. ACM Trans. Graph. 34(2015).

[35]

Huiwen Luo, Koki Nagano, Han-Wei Kung, Qingguo Xu, Zejian Wang, Lingyu Wei, Liwen Hu, and Hao Li. 2021. Normalized Avatar Synthesis Using StyleGAN and Perceptual Refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[36]

Karl F. MacDorman, Robert D. Green, Chin-Chang Ho, and Clinton T. Koch. 2009. Too real for comfort? Uncanny responses to computer generated faces. Computers in Human Behavior 25 (2009).

[37]

Verónica Orvalho, Pedro Bastos, Frederic Parke, Bruno Oliveira, and Xenxo Alvarez. 2012. A Facial Rigging Survey. In Eurographics 2012 - State of the Art Reports.

[38]

Verónica Costa Orvalho, Ernesto Zacur, and Antonio Susin. 2008. Transferring the Rig and Animations from a Character to Different Face Models. Computer Graphics Forum 27 (2008).

[39]

Frederic I. Parke and Keith Waters. 2008. Computer Facial Animation(2nd ed.). AK Peters Ltd.

[40]

Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. 2009. A 3D Face Model for Pose and Illumination Invariant Face Recognition. In 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance.

Digital Library

[41]

R3DS. 2022. Wrap3. https://www.russian3dscanner.com/

[42]

Bay Raitt. 2004. The Making of Gollum. Presentation at U. Southern California Institute for Creative Technologies’s Frontiers of Facial Animation Workshop.

[43]

Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J. Black. 2018. Generating 3D Faces using Convolutional Mesh Autoencoders. In Proceedings of the European Conference on Computer Vision (ECCV).

Digital Library

[44]

Roger Blanco i Ribera, Eduard Zell, J. P. Lewis, Junyong Noh, and Mario Botsch. 2017. Facial Retargeting with Automatic Range of Motion Alignment. ACM Trans. Graph. 36(2017).

[45]

EH Rothauser. 1969. IEEE Recommended Practice for Speech Quality Measurements. IEEE Transactions on Audio and Electroacoustics 17 (1969).

[46]

Gabriel Schwartz, Shih-En Wei, Te-Li Wang, Stephen Lombardi, Tomas Simon, Jason Saragih, and Yaser Sheikh. 2020. The Eyes Have It: An Integrated Eye and Face Model for Photorealistic Facial Animation. ACM Trans. Graph. 39(2020).

[47]

Yeongho Seol, J.P. Lewis, Jaewoo Seo, Byungkuk Choi, Ken Anjyo, and Junyong Noh. 2012. Spacetime Expression Cloning for Blendshapes. ACM Trans. Graph. 31(2012).

[48]

Yeongho Seol, Wan-Chun Ma, and J.P. Lewis. 2016. Creating an Actor-Specific Facial Rig from Performance Capture. In Proceedings of the 2016 Symposium on Digital Production(DigiPro ’16).

[49]

Mike Seymour. 2019. FACS at 40: Facial Action Coding System Panel. In ACM SIGGRAPH 2019 Panels(SIGGRAPH ’19).

Digital Library

[50]

Eftychios Sifakis, Igor Neverov, and Ronald Fedkiw. 2005. Automatic Determination of Facial Muscle Activations from Sparse Motion Capture Marker Data. In ACM SIGGRAPH 2005 Papers(SIGGRAPH ’05).

Digital Library

[51]

Karan Singh and Eugene Fiume. 1998. Wires: A Geometric Deformation Technique. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques(SIGGRAPH ’98).

Digital Library

[52]

Steven L. Song, Weiqi Shi, and Michael Reed. 2020. Accurate Face Rig Approximation with Deep Differential Subspace Reconstruction. ACM Trans. Graph. 39(2020).

[53]

Sangeetha Grama Srinivasan, Qisi Wang, Junior Rojas, Gergely Klár, Ladislav Kavan, and Eftychios Sifakis. 2021. Learning Active Quasistatic Physics-Based Models from Data. ACM Trans. Graph. 40(2021).

[54]

Supasorn Suwajanakorn, Ira Kemelmacher-Shlizerman, and Steven M. Seitz. 2014. Total Moving Face Reconstruction. In Computer Vision - ECCV 2014.

[55]

Supasorn Suwajanakorn, Steven M. Seitz, and Ira Kemelmacher-Shlizerman. 2015. What Makes Tom Hanks Look Like Tom Hanks. In 2015 IEEE International Conference on Computer Vision (ICCV).

[56]

Chenglei Wu, Derek Bradley, Markus Gross, and Thabo Beeler. 2016. An Anatomically-Constrained Local Deformation Model for Monocular Face Capture. ACM Trans. Graph. 35(2016).

[57]

Uldis Zarins. 2018. Anatomy of Facial Expressions. Exonicus, Incorporated.

[58]

Gaspard Zoss, Derek Bradley, Pascal Bérard, and Thabo Beeler. 2018. An Empirical Rig for Jaw Animation. ACM Trans. Graph. 37(2018).

Cited By

Pan YAgrawal RSingh K(2024)S3: Speech, Script and Scene driven Head and Eye AnimationACM Transactions on Graphics10.1145/365817243:4(1-12)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658172
Kavan LDoublestein JPrazak MCioffi MRoble D(2024)Compressed Skinning for Facial BlendshapesACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657477(1-9)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657477
Kyriakou Tde la Campa Crespo MPanayiotou AChrysanthou YCharalambous PAristidou A(2024)Virtual Instrument Performances (VIP): A Comprehensive ReviewComputer Graphics Forum10.1111/cgf.1506543:2Online publication date: 30-Apr-2024
https://doi.org/10.1111/cgf.15065
Show More Cited By

Index Terms

Animatomy: an Animator-centric, Anatomically Inspired System for 3D Facial Modeling, Animation and Transfer
1. Computing methodologies
  1. Computer graphics

Recommendations

Performance-driven animation of hand-drawn cartoon faces

We present a novel performance-driven approach to animating cartoon faces starting from pure 2D drawings. A 3D approximate facial model automatically built from front and side view master frames of character drawings is introduced to enable the animated ...
Neural Face Rigging for Animating and Retargeting Facial Meshes in the Wild
SIGGRAPH '23: ACM SIGGRAPH 2023 Conference Proceedings

We propose an end-to-end deep-learning approach for automatic rigging and retargeting of 3D models of human faces in the wild. Our approach, called Neural Face Rigging (NFR), holds three key properties: (i) NFR’s expression space maintains human-...
Low Bandwidth Video Streaming using FACS, Facial Expression and Animation Techniques
GRAPP 2016: Proceedings of the 11th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications: Volume 1: GRAPP

In this paper we describe an easy to use real-time 3D facial expression and animation system that takes the creation of individual facial expressions to the atomic level. That is instead of generating and recording known facial expressions we propose a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SA '22: SIGGRAPH Asia 2022 Conference Papers

November 2022

482 pages

ISBN:9781450394703

DOI:10.1145/3550469

Editors:
Soon Ki Jung
Kyungpook National University, South Korea
,
Jehee Lee
Seoul National University, South Korea
,
Adam Bargteil
University of Maryland Baltimore County, USA

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 November 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

SA '22

Sponsor:

SIGGRAPH

SA '22: SIGGRAPH Asia 2022

December 6 - 9, 2022

Daegu, Republic of Korea

Acceptance Rates

Overall Acceptance Rate 178 of 869 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
836
Total Downloads

Downloads (Last 12 months)177
Downloads (Last 6 weeks)12

Reflects downloads up to 06 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Pan YAgrawal RSingh K(2024)S3: Speech, Script and Scene driven Head and Eye AnimationACM Transactions on Graphics10.1145/365817243:4(1-12)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658172
Kavan LDoublestein JPrazak MCioffi MRoble D(2024)Compressed Skinning for Facial BlendshapesACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657477(1-9)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657477
Kyriakou Tde la Campa Crespo MPanayiotou AChrysanthou YCharalambous PAristidou A(2024)Virtual Instrument Performances (VIP): A Comprehensive ReviewComputer Graphics Forum10.1111/cgf.1506543:2Online publication date: 30-Apr-2024
https://doi.org/10.1111/cgf.15065
Azari BLim A(2024)EmoStyle: One-Shot Facial Expression Editing Using Continuous Emotion Parameters2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00626(6373-6382)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00626
Zhang YSu RYu JLi R(2024)3D facial modeling, animation, and rendering for digital humans: A surveyNeurocomputing10.1016/j.neucom.2024.128168598(128168)Online publication date: Sep-2024
https://doi.org/10.1016/j.neucom.2024.128168
Zhu CJoslin C(2024)A review of motion retargeting techniques for 3D character facial animationComputers & Graphics10.1016/j.cag.2024.104037123(104037)Online publication date: Oct-2024
https://doi.org/10.1016/j.cag.2024.104037
Wagner NSchwanecke UBotsch M(2024)AnaConDaR: Anatomically-Constrained Data-Adaptive Facial RetargetingComputers & Graphics10.1016/j.cag.2024.103988122(103988)Online publication date: Aug-2024
https://doi.org/10.1016/j.cag.2024.103988
Nicolas WUlrich SMario B(2024)SparseSoftDECA — Efficient high-resolution physics-based facial animation from sparse landmarksComputers & Graphics10.1016/j.cag.2024.103903119(103903)Online publication date: Apr-2024
https://doi.org/10.1016/j.cag.2024.103903
Wagner NBotsch MSchwanecke U(2023)SoftDECA: Computationally Efficient Physics-Based Facial AnimationsProceedings of the 16th ACM SIGGRAPH Conference on Motion, Interaction and Games10.1145/3623264.3624439(1-11)Online publication date: 15-Nov-2023
https://dl.acm.org/doi/10.1145/3623264.3624439
Zhang LZhao ZCong XZhang QGu SGao YZheng RYang WXu LYu J(2023)HACK: Learning a Parametric Head and Neck Model for High-fidelity AnimationACM Transactions on Graphics10.1145/359209342:4(1-20)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592093
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents