research-article

3D shape regression for real-time facial animation

Authors:

Stephen Lin, and

Kun ZhouAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 32, Issue 4

Article No.: 41, Pages 1 - 10

https://doi.org/10.1145/2461912.2462012

Published: 21 July 2013 Publication History

Abstract

We present a real-time performance-driven facial animation system based on 3D shape regression. In this system, the 3D positions of facial landmark points are inferred by a regressor from 2D video frames of an ordinary web camera. From these 3D points, the pose and expressions of the face are recovered by fitting a user-specific blendshape model to them. The main technical contribution of this work is the 3D regression algorithm that learns an accurate, user-specific face alignment model from an easily acquired set of training data, generated from images of the user performing a sequence of predefined facial poses and expressions. Experiments show that our system can accurately recover 3D face shapes even for fast motions, non-frontal faces, and exaggerated expressions. In addition, some capacity to handle partial occlusions and changing lighting conditions is demonstrated.

Supplementary Material

ZIP File (a41-cao.zip)

Supplemental material.

Download
129.38 MB

MP4 File (tp096.mp4)

Download
21.38 MB

References

[1]

Beeler, T., Bickel, B., Beardsley, P., Sumner, R., and Gross, M. 2010. High-quality single-shot capture of facial geometry. ACM Trans. Graph. 29, 4, 40:1--40:9.

Digital Library

[2]

Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., and Gross, M. 2011. High-quality passive facial performance capture using anchor frames. ACM Trans. Graph. 30, 4, 75:1--75:10.

Digital Library

[3]

Besl, P., and McKay, H. 1992. A method for registration of 3-d shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14, 2, 239--256.

Digital Library

[4]

Bingham, E., and Mannila, H. 2001. Random projection in dimensionality reduction: Applications to image and text data. In Knowledge Discovery and Data Mining, 245--250.

Digital Library

[5]

Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In Proceedings of SIGGRAPH, 187--194.

Digital Library

[6]

Bradley, D., Heidrich, W., Popa, T., and Sheffer, A. 2010. High resolution passive facial performance capture. ACM Trans. Graph. 29, 4, 41:1--41:10.

Digital Library

[7]

Byrd, R. H., Lu, P., Nocedal, J., and Zhu, C. 1995. A limited memory algorithm for bound constrained optimization. SIAM J. Sci. Comput. 16, 5 (Sept.), 1190--1208.

Digital Library

[8]

Cao, X., Wei, Y., Wen, F., and Sun, J. 2012. Face alignment by explicit shape regression. In IEEE CVPR, 2887--2894.

Digital Library

[9]

Cao, C., Weng, Y., Zhou, S., Tong, Y., and Zhou, K. 2013. FaceWarehouse: a 3D Facial Expression Database for Visual Computing. IEEE TVCG, under revision.

[10]

Castelan, M., Smith, W. A., and Hancock, E. R. 2007. A coupled statistical model for face shape recovery from brightness images. IEEE Trans. Image Processing 16, 4, 1139--1151.

Digital Library

[11]

Chai, J.-X., Xiao, J., and Hodgins, J. 2003. Vision-based control of 3d facial animation. In Symp. Comp. Anim., 193--206.

Digital Library

[12]

Cootes, T. F., Ionita, M. C., Lindner, C., and Sauer, P. 2012. Robust and accurate shape model fitting using random forest regression voting. In ECCV, VII:278--291.

Digital Library

[13]

DeCarlo, D., and Metaxas, D. 2000. Optical flow constraints on deformable models with applications to face tracking. Int. Journal of Computer Vision 38, 2, 99--127.

Digital Library

[14]

Dementhon, D. F., and Davis, L. S. 1995. Model-based object pose in 25 lines of code. Int. J. Comput. Vision 15, 1--2, 123--141.

Digital Library

[15]

Dollar, P., Welinder, P., and Perona, P. 2010. Cascaded pose regression. In IEEE CVPR, 1078--1085.

[16]

Ekman, P., and Friesen, W. 1978. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press.

[17]

Essa, I., Basu, S., Darrell, T., and Pentland, A. 1996. Modeling, tracking and interactive animation of faces and heads: Using input from video. In Computer Animation, 68--79.

Digital Library

[18]

Huang, D., and la Torre, F. D. 2012. Facial action transfer with personalized bilinear regression. In ECCV, II:144--158.

Digital Library

[19]

Huang, H., Chai, J., Tong, X., and Wu, H.-T. 2011. Leveraging motion capture and 3d scanning for high-fidelity facial performance acquisition. ACM Trans. Graph. 30, 4, 74:1--74:10.

Digital Library

[20]

Kholgade, N., Matthews, I., and Sheikh, Y. 2011. Content retargeting using parameter-parallel facial layers. In Symp. Computer Animation, 195--204.

Digital Library

[21]

Lewis, J. P., and Anjyo, K. 2010. Direct manipulation blendshapes. IEEE CG&A 30, 4, 42--50.

Digital Library

[22]

Li, H., Weise, T., and Pauly, M. 2010. Example-based facial rigging. ACM Trans. Graph. 29, 4, 32:1--32:6.

Digital Library

[23]

Matthews, I., Xiao, J., and Baker, S. 2007. 2D vs. 3D deformable face models: Representational power, construction, and real-time fitting. Int. J. Computer Vision 75, 1, 93--113.

Digital Library

[24]

Pighin, F., Hecker, J., Lischinski, D., Szeliski, R., and Salesin, D. H. 1998. Synthesizing realistic facial expressions from photographs. In Proceedings of SIGGRAPH, 75--84.

Digital Library

[25]

Pighin, F., Szeliski, R., and Salesin, D. 1999. Resynthesizing facial animation through 3d model-based tracking. In Int. Conf. Computer Vision, 143--150.

[26]

Saragih, J., Lucey, S., and Cohn, J. 2011. Real-time avatar animation from a single image. In AFGR, 213--220.

[27]

Seo, J., Irving, G., Lewis, J. P., and Noh, J. 2011. Compression and direct manipulation of complex blendshape models. ACM Trans. Graph. 30, 6.

Digital Library

[28]

Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. ACM Trans. Graph. 24, 3, 426--433.

Digital Library

[29]

Weise, T., Li, H., Gool, L. V., and Pauly, M. 2009. Face/off: Live facial puppetry. In Symp. Computer Animation, 7--16.

Digital Library

[30]

Weise, T., Bouaziz, S., Li, H., and Pauly, M. 2011. Realtime performance-based facial animation. ACM Trans. Graph. 30, 4 (July), 77:1--77:10.

Digital Library

[31]

Williams, L. 1990. Performance driven facial animation. In Proceedings of SIGGRAPH, 235--242.

Digital Library

[32]

Xiao, J., Chai, J., and Kanade, T. 2006. A closed-form solution to non-rigid shape and motion recovery. Int. J. Computer Vision 67, 2, 233--246.

Digital Library

[33]

Yang, F., Wang, J., Shechtman, E., Bourdev, L., and Metaxas, D. 2011. Expression flow for 3D-aware face component transfer. ACM Trans. Graph. 30, 4, 60:1--60:10.

Digital Library

[34]

Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: high resolution capture for modeling and animation. ACM Trans. Graph. 23, 3, 548--558.

Digital Library

[35]

Zhang, Z. 2000. A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22, 11, 1330--1334.

Digital Library

Cited By

Murala D(2024)METAEDUCATION: State-of-the-Art Methodology for Empowering Feature EducationIEEE Access10.1109/ACCESS.2024.339190312(57992-58020)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3391903
Kwak JKo H(2024)4D Facial Avatar Reconstruction From Monocular Video via Efficient and Controllable Neural Radiance FieldsIEEE Access10.1109/ACCESS.2024.335505212(15675-15683)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3355052
Zhang YSu RYu JLi R(2024)3D facial modeling, animation, and rendering for digital humans: A surveyNeurocomputing10.1016/j.neucom.2024.128168(128168)Online publication date: Jul-2024
https://doi.org/10.1016/j.neucom.2024.128168
Show More Cited By

Index Terms

3D shape regression for real-time facial animation
1. Computing methodologies
  1. Computer graphics
    1. Animation
    2. Graphics systems and interfaces
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Graphics input devices

Recommendations

Displaced dynamic expression regression for real-time facial tracking and animation

We present a fully automatic approach to real-time facial tracking and animation with a single video camera. Our approach does not need any calibration for each individual user. It learns a generic regressor from public image datasets, which can be ...
Read More
Pose-Robust Facial Expression Recognition Using View-Based 2D + 3D AAM

This paper proposes a pose-robust face tracking and facial expression recognition method using a view-based 2D 3D active appearance model (AAM) that extends the 2D 3D AAM to the view-based approach, where one independent face model is used for a ...
Read More
Real-time facial expression recognition using STAAM and layered GDA classifier

This paper proposes a real-time person independent facial expression recognition in two parts: one is a model fitting part using a proposed stereo active appearance model (STAAM) and another is a person independent facial expression recognition using a ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 32, Issue 4

July 2013

1215 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2461912

Issue’s Table of Contents

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 July 2013

Published in TOG Volume 32, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

284
Total Citations
View Citations
2,986
Total Downloads

Downloads (Last 12 months)115
Downloads (Last 6 weeks)28

Other Metrics

View Author Metrics

Citations

Cited By

Murala D(2024)METAEDUCATION: State-of-the-Art Methodology for Empowering Feature EducationIEEE Access10.1109/ACCESS.2024.339190312(57992-58020)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3391903
Kwak JKo H(2024)4D Facial Avatar Reconstruction From Monocular Video via Efficient and Controllable Neural Radiance FieldsIEEE Access10.1109/ACCESS.2024.335505212(15675-15683)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3355052
Zhang YSu RYu JLi R(2024)3D facial modeling, animation, and rendering for digital humans: A surveyNeurocomputing10.1016/j.neucom.2024.128168(128168)Online publication date: Jul-2024
https://doi.org/10.1016/j.neucom.2024.128168
Ji XLiao ZDong LTang YLi GMao M(2024)3D facial animation driven by speech-video dual-modal signalsComplex & Intelligent Systems10.1007/s40747-024-01481-5Online publication date: 23-May-2024
https://doi.org/10.1007/s40747-024-01481-5
Larey AAsraf OKelder AWilf IKruzel ODaniel N(2024)Facial Expression Retargeting from a Single CharacterAI Technologies and Virtual Reality10.1007/978-981-99-9018-4_16(217-233)Online publication date: 20-Mar-2024
https://doi.org/10.1007/978-981-99-9018-4_16
Huang ZWu X(2024)PR3D: Precise and realistic 3D face reconstruction from a single imageComputer Animation and Virtual Worlds10.1002/cav.225435:3Online publication date: 30-May-2024
https://doi.org/10.1002/cav.2254
Guo LZhu HLu YWu MCao XWilliams BChen YNeville J(2023)RAFaReProceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v37i1.25149(719-727)Online publication date: 7-Feb-2023
https://dl.acm.org/doi/10.1609/aaai.v37i1.25149
Cai YLi XLiu FLiu JLiu KLiu ZShao X(2023)Enhancing polarization 3D facial imaging: overcoming azimuth ambiguity without extra depth devicesOptics Express10.1364/OE.50507431:26(43891)Online publication date: 12-Dec-2023
https://doi.org/10.1364/OE.505074
Xu YZhang HWang LZhao XHuang HQi GLiu Y(2023)LatentAvatar: Learning Latent Expression Code for Expressive Neural Head AvatarACM SIGGRAPH 2023 Conference Proceedings10.1145/3588432.3591545(1-10)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1145/3588432.3591545
Zhu HYang HGuo LZhang YWang YHuang MWu MShen QYang RCao X(2023)FaceScape: 3D Facial Dataset and Benchmark for Single-View 3D Face ReconstructionIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.330733845:12(14528-14545)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1109/TPAMI.2023.3307338
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents