Article

Synthesizing realistic facial expressions from photographs

Authors:

Frederic Pighin,

Dani Lischinski,

Richard Szeliski,

David H. SalesinAuthors Info & Claims

SIGGRAPH '06: ACM SIGGRAPH 2006 Courses

Pages 19 - es

https://doi.org/10.1145/1185657.1185859

Published: 30 July 2006 Publication History

Abstract

We present new techniques for creating photorealistic textured 3D facial models from photographs of a human subject, and for creating smooth transitions between different facial expressions by morphing between these different models. Starting from several uncalibrated views of a human subject, we employ a user-assisted technique to recover the camera poses corresponding to the views as well as the 3D coordinates of a sparse set of chosen locations on the subject's face. A scattered data interpolation technique is then used to deform a generic face mesh to fit the particular geometry of the subject's face. Having recovered the camera poses and the facial geometry, we extract from the input images one or more texture maps for the model. This process is repeated for several facial expressions of a particular subject. To generate transitions between these facial expressions we use 3D shape morphing between the corresponding face models, while at the same time blending the corresponding textures. Using our technique, we have been able to generate highly realistic face models and natural looking animations.

References

[1]

Takaaki Akimoto, Yasuhito Suenaga, and Richard S. Wallace. Automatic Creation of 3D Facial Models. IEEE Computer Graphics and Applications, 13(5):16--22, September 1993.

Digital Library

[2]

Alias j Wavefront, Toronto, Ontario. Alias V7.0, 1995.

[3]

Thaddeus Beier and Shawn Neely. Feature-based Image Metamorphosis. In SIGGRAPH 92 Conference Proceedings, pages 35--42. ACM SIGGRAPH, July 1992.

Digital Library

[4]

Philippe Bergeron and Pierre Lachapelle. Controlling Facial Expressions and Body Movements in the Computer-Generated Animated Short "Tony De Peltrie". In SIGGRAPH 85 Advanced Computer Animation seminar notes. July 1985.

[5]

Christoph Bregler, Michele Covell, and Malcolm Slaney. Video Rewrite: Driving Visual Speech with Audio. In SIGGRAPH 97Conference Proceedings, pages 353--360. ACM SIGGRAPH, August 1997.

Digital Library

[6]

David T. Chen, Andrei State, and David Banks. Interactive Shape Metamorphosis. In 1995 Symposium on Interactive 3D Graphics, pages 43--44. ACM SIGGRAPH, April 1995.

Digital Library

[7]

Chang S. Choi, Kiyoharu, Hiroshi Harashima, and Tsuyoshi Takebe. Analysis and Synthesis of Facial Image Sequences in Model-Based Image Coding. In IEEE Transactions on Circuits and Systems for Video Technology, volume 4, pages 257 -- 275. June 1994.

Digital Library

[8]

Cyberware Laboratory, Inc, Monterey, California. 4020/RGB 3D Scanner with Color Digitizer, 1990.

[9]

Paul E. Debevec, Camillo J. Taylor, and Jitendra Malik. Modeling and Rendering Architecture from Photographs: A Hybrid Geometry- and Image-Based Approach. In SIGGRAPH 96 Conference Proceedings, pages 11--20. ACM SIGGRAPH, August 1996.

Digital Library

[10]

Eben Ostby, Pixar Animation Studios. Personal communication, January 1997.

[11]

Paul Ekman and Wallace V. Friesen. Unmasking the Face. A guide to recognizing emotions fron facial clues. Prentice-Hall, Inc., Englewood Cliffs, New Jersey, 1975.

[12]

Paul Ekman and Wallace V. Friesen. Manual for the Facial Action Coding System. Consulting Psychologists Press, Inc., Palo Alto, California, 1978.

[13]

Irfan Essa, Sumit Basu, Trevor Darrell, and Alex Pentland. Modeling, Tracking and Interactive Animation of Faces and Heads Using Input from Video. In Computer Animation Conference, pages 68--79. June 1996.

Digital Library

[14]

Gary Faigin. The Artist's Complete Guide to Facial Expression. Watson-Guptill Publications, New York, 1990.

[15]

Olivier Faugeras. Three-Dimensional Computer Vision: A Geometric Viewpoint. MIT Press, Cambridge, Massachusetts, 1993.

Digital Library

[16]

G. Golub and C. F. Van Loan. Matrix Computation, third edition. The John Hopkins University Press, Baltimore and London, 1996.

Digital Library

[17]

Brian Guenter, Cindy Grimm, Daniel Wood, Henrique Malvar, and Frederic Pighin. Making Faces. In SIGGRAPH 98 Conference Proceedings. ACM SIGGRAPH, July 1998.

Digital Library

[18]

Pat Hanrahan and Wolfgang Krueger. Reflection from Layered Surfaces Due to Subsurface Scattering. In SIGGRAPH 93 Conference Proceedings, volume 27, pages 165--174. ACM SIGGRAPH, August 1993.

Digital Library

[19]

Bright Star Technologies Inc. Beginning Reading Software. Sierra On-Line, Inc., 1993.

[20]

Horace H. S. Ip and Lijun Yin. Constructing a 3D Individualized Head Model from Two Orthogonal Views. The Visual Computer, 12:254--266, 1996.

[21]

Gregory Ward J., Francis M. Rubinstein, and Robert D. Clear. A Ray Tracing Solution for Diffuse Interreflection. In SIGGRAPH 88 Conference Proceedings, volume 22, pages 85--92. August 1988.

Digital Library

[22]

James R. Kent,Wayne E. Carlson, and Richard E. Parent. Shape Transformation for Polyhedral Objects. In SIGGRAPH 92 Proceedings Conference, volume 26, pages 47--54. ACM SIGGRAPH, July 1992.

Digital Library

[23]

Rolf M. Koch, Markus H. Gross, Friedrich R. Carls, Daniel F. von Buren, George Fankhauser, and Yoav I. H. Parish. Simulating Facial Surgery Using Finite ElementMethods. In SIGGRAPH 96 Conference Proceedings, pages 421--428. ACM SIGGRAPH, August 1996.

Digital Library

[24]

Tsuneya Kurihara and Kiyoshi Arai. A Transformation Method for Modeling and Animation of the Human Face from Photographs. In Nadia Magnenat Thalmann and Daniel Thalmann, editors, Computer Animation 91, pages 45--58. Springer-Verlag, Tokyo, 1991.

[25]

A. Lanitis, C. J. Taylor, and T. F. Cootes. A Unified Approach for Coding and Interpreting Face Images. In Fifth International Conference on Computer Vision (ICCV 95), pages 368--373. Cambridge,Massachusetts, June 1995.

Digital Library

[26]

C. L. Lawson and R. J. Hansen. Solving Least Squares Problems. Prentice-Hall, Englewood Cliffs, 1974.

[27]

Seung-Yong Lee, Kyung-Yong Chwa, Sung Yong Shin, and George Wolberg. Image Metamorphosis Using Snakes and Free-Form Deformations. In SIGGRAPH 95 Conference Proceedings, pages 439--448. ACM SIGGRAPH, August 1995.

Digital Library

[28]

Seung-Yong Lee, George Wolberg, Kyung-Yong Chwa, and Sung Yong Shin. Image Metamorphosis with Scattered Feature Constraints. IEEE Transactions on Visualization and Computer Graphics, 2(4), December 1996.

Digital Library

[29]

Yuencheng Lee, Demetri Terzopoulos, and Keith Waters. Realistic Modeling for Facial Animation. In SIGGRAPH 95 Conference Proceedings, pages 55--62. ACM SIGGRAPH, August 1995.

Digital Library

[30]

Yuencheng C. Lee, Demetri Terzopoulos, and Keith Waters. Constructing Physics-Based Facial Models of Individuals. In Proceedings of Graphics Interface 93, pages 1--8. May 1993.

[31]

Francis H. Moffitt and Edward M. Mikhail. Photogrammetry. Harper & Row, New York, 3 edition, 1980.

[32]

Shree K. Nayar, Katsushi Ikeuchi, and Takeo Kanade. Shape from Interreflections. International Journal of Computer Vision, 6:173--195, 1991.

Digital Library

[33]

Gregory M. Nielson. Scattered Data Modeling. IEEE Computer Graphics and Applications, 13(1):60--70, January 1993.

Digital Library

[34]

Frederic I. Parke. Computer Generated Animation of Faces. Proceedings ACM annual conference., August 1972.

Digital Library

[35]

Frederic I. Parke. A Parametric Model for Human Faces. PhD thesis, University of Utah, Salt Lake City, Utah, December 1974. UTEC-CSc-75-047.

Digital Library

[36]

Frederic I. Parke and Keith Waters. Computer Facial Animation. AK Peters, Wellesley, Massachusetts, 1996.

Digital Library

[37]

W. H. Press, B. P. Flannery, S. A. Teukolsky, andW. T. Vetterling. Numerical Recipes in C: The Art of Scientific Computing. Cambridge University Press, Cambridge, England, second edition, 1992.

Digital Library

[38]

Kari Pulli, Michael Cohen, Tom Duchamp, Hugues Hoppe, Linda Shapiro, and Werner Stuetzle. View-based rendering: Visualizing real objects from scanned range and color data. In Proc. 8th Eurographics Workshop on Rendering. June 1997.

Digital Library

[39]

Steven M. Seitz and Charles R. Dyer. View Morphing. In SIGGRAPH 96 Conference Proceedings, Annual Conference Series, pages 21--30. ACM SIGGRAPH, August 1996.

Digital Library

[40]

Chester C. Slama, editor. Manual of Photogrammetry. American Society of Photogrammetry, Falls Church, Virginia, fourth edition, 1980.

[41]

Richard Szeliski and Sing Bing Kang. Recovering 3D Shape and Motion from Image Streams using Nonlinear Least Squares. Journal of Visual Communication and Image Representation, 5(1):10--28, March 1994.

[42]

Richard Szeliski and Heung-Yeung Shum. Creating Full View Panoramic Image Mosaics and Texture-Mapped Models. In SIGGRAPH 97 Conference Proceedings, pages 251--258. ACM SIGGRAPH, August 1997.

Digital Library

[43]

Demetri Terzopoulos and KeithWaters. Physically-based Facial Modeling, Analysis, and Animation. Journal of Visualization and Computer Animation, 1(4):73--80, March 1990.

[44]

Kristinn R. Thorisson. Gandalf: An Embodied Humanoid Capable of Real-Time Multimodal Dialogue with People. In First ACM International Conference on Autonomous Agents. 1997.

Digital Library

[45]

Michael W. Vannier, Jeffrey F. Marsh, and James O. Warren. Threedimentional Computer Graphics for Craniofacial Surgical Planning and Evaluation. In SIGGRAPH 83 Conference Proceedings, volume 17, pages 263--273. ACM SIGGRAPH, August 1983.

Digital Library

[46]

Keith Waters. A Muscle Model for Animating Three-Dimensional Facial Expression. In SIGGRAPH 87 Conference Proceedings), volume 21, pages 17--24. ACM SIGGRAPH, July 1987.

Digital Library

[47]

Lance Williams. Performance-Driven Facial Animation. In SIGGRAPH 90 Conference Proceedings, volume 24, pages 235--242. August 1990.

Digital Library

[48]

Z. Zhang, K. Isono, and S.Akamatsu. Euclidean Structure from Uncalibrated Images Using Fuzzy Domain Knowledge: Application to Facial Images Synthesis. In Proc. International Conference on Computer Vision (ICCV'98). January 1998.

Digital Library

Cited By

Li JCao CSchwartz GKhirodkar RRichardt CSimon TSheikh YSaito S(2024)URAvatar: Universal Relightable Gaussian Codec AvatarsSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687653(1-11)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3680528.3687653
Li QWang WXu CSun ZYang M(2024)Learning Disentangled Representation for One-Shot Progressive Face SwappingIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.340433446:12(8348-8364)Online publication date: Dec-2024
https://doi.org/10.1109/TPAMI.2024.3404334
Liu YLi QDeng QSun ZYang M(2023)GAN-Based Facial Attribute ManipulationIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.329886845:12(14590-14610)Online publication date: Dec-2023
https://doi.org/10.1109/TPAMI.2023.3298868
Show More Cited By

Index Terms

Synthesizing realistic facial expressions from photographs

Recommendations

Synthesizing realistic facial expressions from photographs
SIGGRAPH '98: Proceedings of the 25th annual conference on Computer graphics and interactive techniques
Synthesizing realistic facial expressions from photographs
SIGGRAPH '05: ACM SIGGRAPH 2005 Courses

We present new techniques for creating photorealistic textured 3D facial models from photographs of a human subject, and for creating smooth transitions between different facial expressions by morphing between these different models. Starting from ...
Synthesis of Facial Expressions in Photographs: Characteristics, Approaches, and Challenges

The synthesis of facial expressions has applicationsin areas such as interactive games, biometrics systems, and training of people with disorders, among others. Although this is an area relatively well explored in the literature, there are no recent ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGGRAPH '06: ACM SIGGRAPH 2006 Courses

July 2006

83 pages

ISBN:1595933646

DOI:10.1145/1185657

Conference Chair:
John Finnegan
Purdue University
,
Program Chair:
Dave Shreiner

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 July 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Conference

SIGGRAPH06

Sponsor:

SIGGRAPH

SIGGRAPH06: Special Interest Group on Computer Graphics and Interactive Techniques Conference

July 30 - August 3, 2006

Massachusetts, Boston

Acceptance Rates

Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

46
Total Citations
View Citations
321
Total Downloads

Downloads (Last 12 months)36
Downloads (Last 6 weeks)4

Reflects downloads up to 24 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li JCao CSchwartz GKhirodkar RRichardt CSimon TSheikh YSaito S(2024)URAvatar: Universal Relightable Gaussian Codec AvatarsSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687653(1-11)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3680528.3687653
Li QWang WXu CSun ZYang M(2024)Learning Disentangled Representation for One-Shot Progressive Face SwappingIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.340433446:12(8348-8364)Online publication date: Dec-2024
https://doi.org/10.1109/TPAMI.2024.3404334
Liu YLi QDeng QSun ZYang M(2023)GAN-Based Facial Attribute ManipulationIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.329886845:12(14590-14610)Online publication date: Dec-2023
https://doi.org/10.1109/TPAMI.2023.3298868
Rawal NKoert DTuran CKersting KPeters JStock-Homburg R(2022)ExGenNet: Learning to Generate Robotic Facial Expression Using Facial Expression RecognitionFrontiers in Robotics and AI10.3389/frobt.2021.7303178Online publication date: 4-Jan-2022
https://doi.org/10.3389/frobt.2021.730317
Han JKim JSuh JKim H(2022)Customizing blendshapes to capture facial detailsThe Journal of Supercomputing10.1007/s11227-022-04885-779:6(6347-6372)Online publication date: 3-Nov-2022
https://doi.org/10.1007/s11227-022-04885-7
Zhang YLing Y(2021)Interactive Narrative Facial Expression Animation Generation by Intuitive Curve Drawing2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW52623.2021.00088(406-409)Online publication date: Mar-2021
https://doi.org/10.1109/VRW52623.2021.00088
Cakir DAritürk MYücel Ö(2021)An Overview of Fake Face Detection ApproachesProgress in Intelligent Decision Science10.1007/978-3-030-66501-2_16(197-211)Online publication date: 30-Jan-2021
https://doi.org/10.1007/978-3-030-66501-2_16
Cai LNitta TYokota SObata TOkada EKawaguchi H(2021)Targeting brain regions of interest in functional near‐infrared spectroscopy—Scalp‐cortex correlation using subject‐specific light propagation modelsHuman Brain Mapping10.1002/hbm.2536742:7(1969-1986)Online publication date: 23-Feb-2021
https://doi.org/10.1002/hbm.25367
Carrigan EZell EGuiard CMcDonnell R(2020)Expression Packing: As‐Few‐As‐Possible Training Expressions for Blendshape TransferComputer Graphics Forum10.1111/cgf.1392539:2(219-233)Online publication date: 13-Jul-2020
https://doi.org/10.1111/cgf.13925
de Bem RGhosh AAjanthan TMiksik OBoukhayma ASiddharth NTorr P(2020)DGPose: Deep Generative Models for Human Body AnalysisInternational Journal of Computer Vision10.1007/s11263-020-01306-1Online publication date: 24-Apr-2020
https://doi.org/10.1007/s11263-020-01306-1
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten