research-article

Facial performance enhancement using dynamic shape space analysis

Authors:

Amit H. Bermano,

Derek Nowrouzezahrai,

Olga Sorkine-Hornung,

Hanspeter Pfister,

Robert W. Sumner,

Markus GrossAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 33, Issue 2

Article No.: 13, Pages 1 - 12

https://doi.org/10.1145/2546276

Published: 08 April 2014 Publication History

Abstract

The facial performance of an individual is inherently rich in subtle deformation and timing details. Although these subtleties make the performance realistic and compelling, they often elude both motion capture and hand animation. We present a technique for adding fine-scale details and expressiveness to low-resolution art-directed facial performances, such as those created manually using a rig, via marker-based capture, by fitting a morphable model to a video, or through Kinect reconstruction using recent faceshift technology. We employ a high-resolution facial performance capture system to acquire a representative performance of an individual in which he or she explores the full range of facial expressiveness. From the captured data, our system extracts an expressiveness model that encodes subtle spatial and temporal deformation details specific to that particular individual. Once this model has been built, these details can be transferred to low-resolution art-directed performances. We demonstrate results on various forms of input; after our enhancement, the resulting animations exhibit the same nuances and fine spatial details as the captured performance, with optional temporal enhancement to match the dynamics of the actor. Finally, we show that our technique outperforms the current state-of-the-art in example-based facial animation.

Supplementary Material

JPG File (a13-sidebyside.jpg)

Download
17.43 KB

bermano (bermano.zip)

Supplemental movie and image files for, Facial performance enhancement using dynamic shape space analysis

Download
114.97 MB

MP4 File (a13-sidebyside.mp4)

Download
29.66 MB

References

[1]

O. Alexander, M. Rogers, W. Lambeth, J.-Y. Chiang, W.-C. Ma, C.-C. Wang, and P. Debevec. 2010. The digital emily project: Achieving a photoreal digital actor. IEEE Comput. Graph. Appl. 30, 4, 20--31.

Digital Library

[2]

E. D. Andersen and K. D. Andersen. 2000. The mosek interior point optimizer for linear programming: An implementation of the homogeneous algorithm. In High Performance Optimization. Kluwer Academic Publishers, 197--232.

[3]

I. Baran, D. Vlasic, E. Grinspun, and J. Popovic. 2009. Semantic deformation transfer. ACM Trans. Graph. 28, 3, 36:1--36:6.

Digital Library

[4]

T. Beeler. B. Bickel, R. Sumner, P. Beardsley, and M. Gross. 2010. High-quality single-shot capture of facial geometry. ACM Trans. Graph. 29, 4.

Digital Library

[5]

T. Beeler, F. Hahn, D. Bradley, B. Bickel, P. Beardsley, C. Gotsman, R. W. Sumner, and M. Gross. 2011. High-quality passive facial performance capture using anchor frames. ACM Trans. Graph. 30, 75:1--75:10.

Digital Library

[6]

B. Bickel, M. Botsch, R. Angst, W. Matusik, M. Otaduy, H. Pfister, and M. Gross. 2007. Multi-scale capture of facial geometry and motion. ACM Trans. Graph. 26, 3.

Digital Library

[7]

B. Bickel, M. Lang, M. Botsch, M. A. Otaduy, and M. Gross. 2008. Pose-space animation and transfer of facial details. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 57--66.

Digital Library

[8]

V. Basso, C. Poggio, T. Blanz, and T. Vetter. 2003. Reanimating faces in images and video. Comput. Graph. Forum 22, 3, 641--650.

[9]

G. Borshukov, D. Piponi, O. Larsen J. Lewis, and C. Tempelaar-Lietz. 2003. Universal capture -- Image-based facial animation for “the matrix reloaded”. In Proceedings of the ACM SIGGRAPH Sketches and Applications Conference.

Digital Library

[10]

M. Botsch, R. Sumner, M. Pauly, and M. Gross. 2006. Deformation transfer for detail-preserving surface editing. In Proceedings of the Workshop on Vision, Modeling and Visualization. 357--364.

[11]

D. Bradley, W. Heidrich, T. Popa, and A. Sheffer. 2010. High resolution passive facial performance capture. ACM Trans. Graph. 29, 4.

Digital Library

[12]

M. Brand. 1999. Voice puppetry. In Proceedings of the 26^th Annual Conference on Computer Graphics and Interactive Techniques. 21--28.

Digital Library

[13]

C. Bregler, M. Covell, and M. Slaney. 1997. Video rewrite: Driving visual speech with audio. In Proceedings of the Annual Conference on Computer Graphics (SIGGRAPH'97). 353--360.

Digital Library

[14]

I. Buck, A. Finkelstein, C. Jacobs, A. Klein, D. H. Salesin, J. Seims, R. Szeliski, and K. Toyama. 2000. Performance-driven hand-drawn animation. In Proceedings of the 1^st International Symposium on NonPhotorealistic Animation and Rendering (NPAR'00). 101--108.

Digital Library

[15]

Y. Cao, P. Faloutsos, E. Kohler, and F. Pighin. 2004. Realtime speech motion synthesis from recorded motions. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 345--353.

Digital Library

[16]

J. Chai and J. K. Hodgins. 2007. Constraint-based motion optimization using a statistical dynamic model. ACM Trans. Graph. 26, 3, 8:1--8:9.

Digital Library

[17]

J.-X. Chai, J. Xiao, and J. Hodgins. 2003. Vision-based control of 3d facial animation. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 193--206.

Digital Library

[18]

E. Chuang and C. Bregler. 2002. Performance driven facial animation using blendshape interpolation. Tech. rep. CS-TR-2002-02, Department of Computer Science, Stanford University.

[19]

K. Dale, K. Sunkavalli, M. K. Johnson, D. Vlasic, W. Matusik, and H. Pfister 2011. Video face replacement. In Proceedings of the SIGGRAPH Asia Conference (SA'11). 130:1--130:10.

Digital Library

[20]

D. Decarlo and D. Metaxas. 1996. The integration of optical flow and deformable models with applications to human face shape and motion estimation. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR'96). 231--238.

Digital Library

[21]

Z. Deng, J. Lewis, and U. Neumann. 2005. Synthesizing speech animation by learning compact speech co-articulation models. In Proceedings of the Computer Graphics International (CGI'05). 19--25.

Digital Library

[22]

M. Desbrun, M. Meyer, P. Schroder, and A. H. Barr. 1999. Implicit fairing of irregular meshes using diffusion and curvature flow. In Proceedings of the Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'99). ACM Press/Addison-Wesley, 317--324.

Digital Library

[23]

P. Ekman and W. Friesen. 1978. The Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press.

[24]

I. Essa, S. Basu, T. Darrell, and A. Pentland. 1996. Modeling, tracking and interactive animation of faces and heads: Using input from video. In Proceedings of the Conference on Computer Animation (CA'96). 68--79.

Digital Library

[25]

T. Ezzat, G. Geiger, and T. Poggio. 2002. Trainable videorealistic speech animation. ACM Trans. Graph. 21, 3, 388-398.

Digital Library

[26]

W.-W. Feng, B.-U. Kim, and Y. Yu. 2008. Real-time data-driven deformation using kernel canonical correlation analysis. ACM Trans. Graph. 27, 3, 91:1--91:9.

Digital Library

[27]

A. Golovinskiy, W. Matusik, H. Pfister, S. Rusinkiewicz, and T. Funkhouser. 2006. A statistical model for synthesis of detailed facial geometry. ACM Trans. Graph. 25, 3, 1025--1034.

Digital Library

[28]

B. K. P. Horn. 1987. Closed-form solution of absolute orientation using unit quaternions. J. Optical Soc. Amer. A 4, 4, 629--642.

[29]

H. Huang, J. Chai, X. Tong, and H.-T. Wu. 2011a. Leveraging motion capture and 3d scanning for high-fidelity facial performance acquisition. ACM Trans. Graph. 30, 4, 74:1--74:10.

Digital Library

[30]

H. Huang, L. Zhao, K. Yin, Y. Qi, Y. Yu, and X. Tong. 2011b. Controllable hand deformation from sparse examples with rich details. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. ACM Press, New York, 73--82.

Digital Library

[31]

A. Jones, A. Gardner, M. Bolas, I. Mcdowall, and P. Debevec. 2006. Performance geometry capture for spatially varying relighting. In Proceedings of the 3^rd European Conference on Visual Media Production (CVMP'06).

[32]

S. Kshirsagar and N. M. Thalmann. 2003. Visyllable based speech animation. Comput. Graph. Forum 22, 3.

[33]

J. Lewis, M. Cordner, and N. Fong. 2000. Pose space deformation: A unified approach to shape interpolation and skeleton-driven deformation. In Proceedings of the 27^th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'00). 165--172.

Digital Library

[34]

J. Lewis, J. Mooser, Z. Deng, and U. Neumann. 2005. Reducing blendshape interference by selected motion attenuation. In Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D'05).

Digital Library

[35]

H. Li, P. Roivainen, and R. Forchheimer. 1993. 3-D motion estimation in model-based facial image coding. IEEE Trans. Pattern Anal. Mach. Intell. 15, 6, 545--555.

Digital Library

[36]

H. LI, R. W. Sumner, and M. Pauly. 2008. Global correspondence optimization for non-rigid registration of depth scans. Comput. Graph. Forum 27, 5.

Digital Library

[37]

J. Ma, R. Cole, B. Pellom, W. Ward, and B. Wise. 2004. Accurate automatic visible speech synthesis of arbitrary 3d model based on concatenation of diviseme motion capture data. Comput. Anim. Virtual Worlds 15, 1--17.

Digital Library

[38]

W.-C. Ma, T. Hawkins, P. Peers, C.-F. Chabert, M. Weiss, and P. Debevec. 2007. Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. In Proceedings of the 18^th Eurographics Conference on Rendering Techniques (EGSR'07). 183--194.

Digital Library

[39]

W.-C. Ma, A. Jones, J.-Y. Chiang, T. Hawkins, S. Frederiksen, P. Peers, M. Vukovic, M. Ouhyoung, and P. Debevec. 2008. Facial performance synthesis using deformation-driven polynomial displacement maps. ACM Trans. Graph. 27, 5.

Digital Library

[40]

X. Ma, B. H. Le, and Z. Deng. 2009. Style learning and transferring for facial animation editing. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA'09). 123--132.

Digital Library

[41]

J.-Y. Noh, and U. Neumann. 2001. Expression cloning. In Proceedings of the Annual Conference on Computer Graphics (SIGGRAPH'01). 277--288.

Digital Library

[42]

F. I. Parke. 1974. A parametric model for human faces. Ph.D. thesis, University of Utah.

Digital Library

[43]

F. H. Pighin, R. Szeliski, and D. Salesin. 1999. Resynthesizing facial animation through 3d model-based tracking. In Proceedings of the 7^th IEEE International Conference on Computer Vision (ICCV'99). 143--150.

[44]

H. Pyun, Y. Kim, W. Chae, H. Kang, and S. Shin. 2003. An example-based approach for facial expression cloning. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 167--176.

Digital Library

[45]

Y. Seol, J. Seo, P. H. Kim, J. P. Lewis, and J. Noh. 2011. Artist friendly facial animation retargeting. ACM Trans. Graph. 30, 6.

Digital Library

[46]

E. Sifakis, I. Neverov, and R. Fedkiw. 2005. Automatic determination of facial muscle activations from sparse motion capture marker data. ACM Trans. Graph. 24, 3, 417--425.

Digital Library

[47]

O. Sorkine, D. Cohen-Or, Y. Lipman, M. Alexa, C. Rossl, and H.-P. Seidel. 2004. Laplacian surface editing. In Proceedings of the Eurographics/ACM SIGGRAPH Symposium on Geometry Processing (SGP'04). ACM Press, New York, 179--188.

Digital Library

[48]

B. Sumner and J. Popovic. 2004. Deformation transfer for triangle meshes. ACM Trans. Graph. 23, 3, 399--405.

Digital Library

[49]

K. Takayama, R. Schmidt, K. Singh, T. Igarashi, T. Boubekeur, and O. Sorkine. 2011. Geobrush: Interactive mesh geometry cloning. Comput. Graph. Forum 30, 2, 613--622.

[50]

J. R. Tena, F. D. L. Torre, and I. Matthews. 2011. Interactive region-based linear 3d face models. ACM Trans. Graph. 30, 4.

Digital Library

[51]

D. Terzopoulus and K. Waters. 1993. Analysis and synthesis of facial image sequences using physical and anatomical models. IEEE Trans. Pattern Anal. Mach. Intell. 14, 569--579.

Digital Library

[52]

K. Venkataraman, S. Lodha, and R. Raghavan. 2005. A kinematic-variational model for animating skin with wrinkles. Comput. Graph. 29, 5, 756--770.

Digital Library

[53]

D. Vlasic, M. Brand, H. Pfister, and J. Popovic. 2005. Face transfer with multilinear models. ACM Trans. Graph. 24, 3, 426--433.

Digital Library

[54]

Y. Wang, X. Huang, C.-S. Lee, S. Zhang, Z. Li, D. Samaras, D. Metaxas, A. Elgammal, and P. Huang. 2004. High resolution acquisition, learning and transfer of dynamic 3-d facial expressions. Comput. Graph. Forum 23, 3, 677--686.

[55]

K. Waters. 1987. A muscle model for animating three-dimensional facial expression. In Proceedings of the 14^th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'87). 17--24.

Digital Library

[56]

T. Weise, S. Bouaziz, H. Li, and M. Pauly. 2011. Realtime performance-based facial animation. ACM Trans. Graph. 30, 4.

Digital Library

[57]

A. Wenger, A. Gardner, C. Tchou, J. Unger, T. Hawkins, and P. Debevec. 2005. Performance relighting and reflectance transformation with time-multiplexed illumination. ACM Trans. Graph. 24, 3, 756--764.

Digital Library

[58]

T. Weyrich, W. Matusik, H. Pfister, B. Bickel, C. Donner, C. Tu, J. Mcandless, J. Lee, A. Ngan, H. W. Jensen, and M. Gross. 2006. Analysis of human faces using a measurement-based skin reflectance model. ACM Trans. Graph. 25, 3, 1013--1024.

Digital Library

[59]

L. Williams. 1990. Performance-driven facial animation. In Proceedings of the 17^th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'90). 235--242.

Digital Library

[60]

C. Wilson, A. Ghosh, P. Peers, J.-Y. Chiang, J. Busch, and P. Debevec 2010. Temporal upsampling of performance geometry using photometric alignment. Trans. Graph. 29, 2.

Digital Library

[61]

Y. Wu, P. Kalra, and N. Magnenat-Thalmann. 1996. Simulation of static and dynamic wrinkles of skin. In Proceedings of the Conference on Computer Animation (CA'96). 90--97.

Digital Library

[62]

L. Zhang, N. Snavely, B. Curless, and S. M. Seitz. 2004. Spacetime faces: High resolution capture for modeling and animation. ACM Trans. Graph. 23, 3, 548--558.

Digital Library

[63]

S. Zhang and P. Huang. 2006. High-resolution, real-time three-dimensional shape measurement. Optical Engin. 45, 12.

[64]

Y. Zhang and T. Sim. 2005. Realistic and efficient wrinkle simulation using an anatomy-based face model with adaptive refinement. In Proceedings of the Computer Graphics International (CGI'05). 3--10.

Digital Library

Cited By

Cong MLan LFedkiw R(2024)Local Geometric Indexing of High Resolution Data for Facial Reconstruction From Sparse MarkersIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.328949530:8(5289-5298)Online publication date: Aug-2024
https://doi.org/10.1109/TVCG.2023.3289495
SUDO MYOKOYAMA K(2022)Effect of Shot Composition, Continuity Assurance Method and Shot Size on Subjective Perception of Continuity during Transitions while Viewing Videos映像のショット構成・連続性担保の手法・ショットサイズの違いがトランジション時の主観的な連続性評価に及ぼす影響についてThe Japanese Journal of Ergonomics10.5100/jje.58.22358:5(223-231)Online publication date: 15-Oct-2022
https://doi.org/10.5100/jje.58.223
Zell EMcDonnell R(2022)Compact Facial Landmark Layouts for Performance CaptureComputer Graphics Forum10.1111/cgf.1446341:2(121-133)Online publication date: 24-May-2022
https://doi.org/10.1111/cgf.14463
Show More Cited By

Index Terms

Facial performance enhancement using dynamic shape space analysis
1. Computing methodologies
  1. Computer graphics
    1. Animation

Recommendations

Facial retargeting with automatic range of motion alignment

While facial capturing focuses on accurate reconstruction of an actor's performance, facial animation retargeting has the goal to transfer the animation to another character, such that the semantic meaning of the animation remains. Because of the ...
Interactive editing of performance-based facial animation
SA '19: SIGGRAPH Asia 2019 Technical Briefs

While performance-based facial animation efficiently produces realistic animation, it still needs additional editing after automatic solving and retargeting. We review why additional editing is required and present a set of interactive editing solutions ...
Facial animation retargeting and control based on a human appearance space

Expressive facial animations are essential to enhance the realism and the credibility of virtual characters. Parameter-based animation methods offer a precise control over facial configurations while performance-based animation benefits from the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 33, Issue 2

March 2014

135 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2603314

Issue’s Table of Contents

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 April 2014

Accepted: 01 August 2013

Revised: 01 July 2013

Received: 01 November 2012

Published in TOG Volume 33, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

31
Total Citations
View Citations
1,107
Total Downloads

Downloads (Last 12 months)17
Downloads (Last 6 weeks)0

Reflects downloads up to 27 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Cong MLan LFedkiw R(2024)Local Geometric Indexing of High Resolution Data for Facial Reconstruction From Sparse MarkersIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.328949530:8(5289-5298)Online publication date: Aug-2024
https://doi.org/10.1109/TVCG.2023.3289495
SUDO MYOKOYAMA K(2022)Effect of Shot Composition, Continuity Assurance Method and Shot Size on Subjective Perception of Continuity during Transitions while Viewing Videos映像のショット構成・連続性担保の手法・ショットサイズの違いがトランジション時の主観的な連続性評価に及ぼす影響についてThe Japanese Journal of Ergonomics10.5100/jje.58.22358:5(223-231)Online publication date: 15-Oct-2022
https://doi.org/10.5100/jje.58.223
Zell EMcDonnell R(2022)Compact Facial Landmark Layouts for Performance CaptureComputer Graphics Forum10.1111/cgf.1446341:2(121-133)Online publication date: 24-May-2022
https://doi.org/10.1111/cgf.14463
Ling JWang ZLu MWang QQian CXu F(2022)Structure-Aware Editable Morphable Model for 3D Facial Detail Animation and ManipulationComputer Vision – ECCV 202210.1007/978-3-031-20062-5_15(249-267)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1007/978-3-031-20062-5_15
Ma LDeng Z(2020)Real-time Face Video Swapping From A Single PortraitSymposium on Interactive 3D Graphics and Games10.1145/3384382.3384519(1-10)Online publication date: 5-May-2020
https://dl.acm.org/doi/10.1145/3384382.3384519
Khan NAkram AMahmood AAshraf SMurtaza K(2020)Masked Linear Regression for Learning Local Receptive Fields for Facial Expression SynthesisInternational Journal of Computer Vision10.1007/s11263-019-01256-3128:5(1433-1454)Online publication date: 1-May-2020
https://dl.acm.org/doi/10.1007/s11263-019-01256-3
Ma LDeng ZAndrews STatarchuk N(2019)Real-time hierarchical facial performance captureProceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games10.1145/3306131.3317016(1-10)Online publication date: 21-May-2019
https://dl.acm.org/doi/10.1145/3306131.3317016
Zollhöfer MThies JGarrido PBradley DBeeler TPérez PStamminger MNießner MTheobalt C(2018)State of the Art on Monocular 3D Face Reconstruction, Tracking, and ApplicationsComputer Graphics Forum10.1111/cgf.1338237:2(523-550)Online publication date: 22-May-2018
https://doi.org/10.1111/cgf.13382
Lamarre MLewis JDanvoye E(2018)Face Stabilization by Mode Pursuit for Avatar Construction2018 International Conference on Image and Vision Computing New Zealand (IVCNZ)10.1109/IVCNZ.2018.8634727(1-6)Online publication date: Nov-2018
https://doi.org/10.1109/IVCNZ.2018.8634727
Moser LHendler DRoble D(2017)MasqueradeACM SIGGRAPH 2017 Talks10.1145/3084363.3085086(1-2)Online publication date: 30-Jul-2017
https://dl.acm.org/doi/10.1145/3084363.3085086
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents