research-article

Data-Driven 3D Neck Modeling and Animation

Authors:

Chengwei Zheng,

Baining GuoAuthors Info & Claims

IEEE Transactions on Visualization and Computer Graphics, Volume 27, Issue 7

Pages 3226 - 3237

https://doi.org/10.1109/TVCG.2020.2967036

Published: 01 July 2021 Publication History

Abstract

In this article, we present a data-driven approach for modeling and animation of 3D necks. Our method is based on a new neck animation model that decomposes the neck animation into local deformation caused by larynx motion and global deformation driven by head poses, facial expressions, and speech. A skinning model is introduced for modeling local deformation and underlying larynx motions, while the global neck deformation caused by each factor is modeled by its corrective blendshape set, respectively. Based on this neck model, we introduce a regression method to drive the larynx motion and neck deformation from speech. Both the neck model and the speech regressor are learned from a dataset of 3D neck animation sequences captured from different identities. Our neck model significantly improves the realism of facial animation and allows users to easily create plausible neck animations from speech and facial expressions. We verify our neck model and demonstrate its advantages in 3D neck tracking and animation.

References

[1]

T. Beeler, et al., “High-quality passive facial performance capture using anchor frames,”ACM Trans. Graph., vol. 30, no. 4, 2011, Art. no.

[2]

H. Huang, J. Chai, X. Tong, and H.-T. Wu, “Leveraging motion capture and 3D scanning for high-fidelity facial performance acquisition,”ACM Trans. Graph., vol. 30, no. 4, 2011, Art. no.

[3]

F. Xu, J. Chai, Y. Liu, and X. Tong, “Controllable high-fidelity facial performance transfer,”ACM Trans. Graph., vol. 33, no. 4, 2014, Art. no.

[4]

M. Sela, E. Richardson, and R. Kimmel, “Unrestricted facial geometry reconstruction using image-to-image translation,” in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 1585–1594.

[5]

A. Tewari, et al., “Self-supervised multi-level face model learning for monocular reconstruction at over 250 Hz,” in Proc. Comput. Vis. Pattern Recognit., 2018, pp. 2549–2559.

[6]

C. Cao, H. Wu, Y. Weng, T. Shao, and K. Zhou, “Real-time facial animation with image-based dynamic avatars,”ACM Trans. Graph., vol. 35, no. 4, 2016, Art. no.

[7]

L. Hu, et al., “Avatar digitization from a single image for real-time rendering,”ACM Trans. Graph., vol. 36, no. 6, 2017, Art. no.

[8]

M. Chai, L. Wang, Y. Weng, X. Jin, and K. Zhou, “Dynamic hair manipulation in images and videos,”ACM Trans. Graph., vol. 32, no. 4, 2013, Art. no.

[9]

M. Chai, C. Zheng, and K. Zhou, “A reduced model for interactive hairs,”ACM Trans. Graph., vol. 33, no. 4, 2014, Art. no.

[10]

L. Hu, C. Ma, L. Luo, and H. Li, “Single-view hair modeling using a hairstyle database,”ACM Trans. Graph., vol. 34, no. 4, 2015, Art. no.

[11]

P. Bérard, D. Bradley, M. Nitti, T. Beeler, and M. H. Gross, “High-quality capture of eyes,”ACM Trans. Graph., vol. 33, no. 6, 2014, Art. no.

[12]

C. Wang, F. Shi, S. Xia, and J. Chai, “Realtime 3D eye gaze animation using a single RGB camera,”ACM Trans. Graph., vol. 35, no. 4, 2016, Art. no.

[13]

P. Bérard, D. Bradley, M. Gross, and T. Beeler, “Lightweight eye capture using a parametric model,”ACM Trans. Graph., vol. 35, no. 4, 2016, Art. no.

[14]

Q. Wen, F. Xu, and J.-H. Yong, “Real-time 3D eye performance reconstruction for RGBD cameras,”IEEE Trans. Vis. Comput. Graphics, vol. 23, no. 12, pp. 2586–2598, Dec.2017.

[15]

A. Bermano, T. Beeler, Y. Kozlov, D. Bradley, B. Bickel, and M. Gross, “Detailed spatio-temporal reconstruction of eyelids,”ACM Trans. Graph., vol. 34, no. 4, 2015, Art. no.

[16]

Q. Wen, F. Xu, M. Lu, and J.-H. Yong, “Real-time 3D eyelids tracking from semantic edges,”ACM Trans. Graph., vol. 36, no. 6, 2017, Art. no.

[17]

C. Wu, et al., “Model-based teeth reconstruction,”ACM Trans. Graph., vol. 35, no. 6, 2016, Art. no.

[18]

P. Garrido, et al., “Corrective 3D reconstruction of lips from monocular video,”ACM Trans. Graph., vol. 35, no. 6, 2016, Art. no.

[19]

S.-H. Lee and D. Terzopoulos, “Heads up!: Biomechanical modeling and neuromuscular control of the neck,”ACM Trans. Graph., vol. 25, no. 3, pp. 1188–1198, 2006.

Digital Library

[20]

L. Zhiping, P. Nicolas, and E. Arjan, “Physically-based human neck simulation,”Workshop on Virtual Reality Interaction and Physical Simulation, J. Bender, J. Dequidt, C. Duriez, and G. Zachmann, Eds. The Eurographics Association, to be published.

[21]

S.-H. Lee, E. Sifakis, and D. Terzopoulos, “Comprehensive biomechanical modeling and simulation of the upper body,”ACM Trans. Graph., vol. 28, no. 4, 2009, Art. no.

[22]

T. Li, T. Bolkart, M. J. Black, H. Li, and J. Romero, “Learning a model of facial shape and expression from 4D scans,”ACM Trans. Graph., vol. 36, no. 6, 2017, Art. no.

Digital Library

[23]

M. Zollhöfer, et al., “State of the art on monocular 3D face reconstruction, tracking, and applications,”Comput. Graph. Forum, vol. 37, no. 2, pp. 523–550, 2018.

[24]

R. Zhang, P.-S. Tsai, J. E. Cryer, and M. Shah, “Shape-from-shading: A survey,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 21, no. 8, pp. 690–706, Aug.1999.

Digital Library

[25]

F. Shi, H.-T. Wu, X. Tong, and J. Chai, “Automatic acquisition of high-fidelity facial performances using monocular videos,”ACM Trans. Graph., vol. 33, no. 6, 2014, Art. no.

[26]

P. Garrido, L. Valgaerts, C. Wu, and C. Theobalt, “Reconstructing detailed dynamic face geometry from monocular video,”ACM Trans. Graph., vol. 32, no. 6, 2013, Art. no.

[27]

V. Blanz and T. Vetter, “A morphable model for the synthesis of 3D faces,” in Proc. 26th Annu. Conf. Comput. Graph. Interactive Techn., 1999, pp. 187–194.

[28]

J. P. Lewis, K. Anjyo, T. Rhee, M. Zhang, F. H. Pighin, and Z. Deng, “Practice and theory of blendshape facial models,”Eurographics, vol. 1, no. 8, pp. 199–218, 2014.

[29]

D. Vlasic, M. Brand, H. Pfister, and J. Popović, “Face transfer with multilinear models,”ACM Trans. Graph., vol. 24, no. 3, pp. 426–433, 2005.

Digital Library

[30]

C. Cao, Y. Weng, S. Zhou, Y. Tong, and K. Zhou, “FaceWarehouse: A 3D facial expression database for visual computing,”IEEE Trans. Vis. Comput. Graphics, vol. 20, no. 3, pp. 413–425, Mar.2014.

Digital Library

[31]

A. E. Ichim, L. Kavan, M. Nimier-David, and M. Pauly, “Building and animating user-specific volumetric face rigs,” in Proc. Symp. Comput. Animation, 2016, pp. 107–117.

[32]

A.-E. Ichim, P. Kadleček, L. Kavan, and M. Pauly, “Phace: Physics-based face modeling and animation,”ACM Trans. Graph., vol. 36, no. 4, 2017, Art. no.

[33]

A. E. Ichim, S. Bouaziz, and M. Pauly, “Dynamic 3D avatar creation from hand-held video input,”ACM Trans. Graph., vol. 34, no. 4, 2015, Art. no.

[34]

P. Garrido, et al., “Reconstruction of personalized 3D face rigs from monocular video,”ACM Trans. Graph., vol. 35, no. 3, 2016, Art. no.

[35]

Y. Liu, F. Xu, J. Chai, X. Tong, L. Wang, and Q. Huo, “Video-audio driven real-time facial animation,”ACM Trans. Graph., vol. 34, no. 6, 2015, Art. no.

[36]

S. Taylor, et al., “A deep learning approach for generalized speech animation,”ACM Trans. Graph., vol. 36, no. 4, 2017, Art. no.

[37]

T. Karras, T. Aila, S. Laine, A. Herva, and J. Lehtinen, “Audio-driven facial animation by joint end-to-end learning of pose and emotion,”ACM Trans. Graph., vol. 36, no. 4, 2017, Art. no.

[38]

S. Suwajanakorn, S. M. Seitz, and I. Kemelmacher-Shlizerman, “Synthesizing obama: Learning lip sync from audio,”ACM Trans. Graph., vol. 36, no. 4, 2017, Art. no.

[39]

N. Dyn, D. Levine, and J. A. Gregory, “A butterfly subdivision scheme for surface interpolation with tension control,”ACM Trans. Graph., vol. 9, no. 2, pp. 160–169, 1990.

Digital Library

[40]

M. Rabinovich, R. Poranne, D. Panozzo, and O. Sorkine-Hornung, “Scalable locally injective mappings,”ACM Trans. Graph., vol. 36, no. 4, Apr.2017. [Online]. Available: http://doi.acm.org/10.1145/3072959.2983621

Digital Library

[41]

I. Baran and J. Popović, “Automatic rigging and animation of 3D characters,”ACM Trans. Graph., vol. 26, no. 3, 2007, Art. no.

Digital Library

[42]

Y. Furukawa and J. Ponce, “Accurate, dense, and robust multiview stereopsis,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 8, pp. 1362–1376, Aug.2010.

Digital Library

[43]

T. Brox, A. Bruhn, N. Papenberg, and J. Weickert, “High accuracy optical flow estimation based on a theory for warping,” in Proc. Eur. Conf. Comput. Vis., 2004, pp. 25–36.

[44]

H. Li, B. Adams, L. J. Guibas, and M. Pauly, “Robust single-view geometry and motion reconstruction,”ACM Trans. Graph., vol. 28, no. 5, 2009, Art. no.

Digital Library

[45]

R. W. Sumner and J. Popović, “Deformation transfer for triangle meshes,”ACM Trans. Graph., vol. 23, no. 3, pp. 399–405, Aug.2004. [Online]. Available: http://doi.acm.org/10.1145/1015706.1015736

Digital Library

Cited By

Huang ZTang FZhang YCao JLi CTang SLi JLee T(2024)Identity-Preserving Face Swapping via Dual Surrogate Generative ModelsACM Transactions on Graphics10.1145/367616543:5(1-19)Online publication date: 9-Aug-2024
https://dl.acm.org/doi/10.1145/3676165
Li SZhao RGuan QChen JZhang Y(2024)A 3D model encryption method supporting adaptive visual effects after decryptionAdvanced Engineering Informatics10.1016/j.aei.2023.10231959:COnline publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.aei.2023.102319
Zhang LZhao ZCong XZhang QGu SGao YZheng RYang WXu LYu J(2023)HACK: Learning a Parametric Head and Neck Model for High-fidelity AnimationACM Transactions on Graphics10.1145/359209342:4(1-20)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592093

Index Terms

Data-Driven 3D Neck Modeling and Animation
1. Computing methodologies

Index terms have been assigned to the content through auto-classification.

Recommendations

Data-Driven 3D Facial Animation
Data driven cloth animation
SIGGRAPH '07: ACM SIGGRAPH 2007 sketches

We present a new method for cloth animation based on data driven synthesis. In contrast to approaches that focus on physical simulation, we animate cloth by manipulating short sequences of existing cloth animation. While our source of data is cloth ...
Data-driven 2D effects animation
ICVGIP '16: Proceedings of the Tenth Indian Conference on Computer Vision, Graphics and Image Processing

Making plausible, high quality visual effects, like water splashes or fire, in traditional 2D animation pipelines require an animator to draw many frames of phenomena that are very difficult to recreate manually. We present a technique that uses a ...

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Visualization and Computer Graphics

IEEE Transactions on Visualization and Computer Graphics Volume 27, Issue 7

July 2021

259 pages

ISSN:1077-2626

Issue’s Table of Contents

1077-2626 © 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Publisher

IEEE Educational Activities Department

United States

Publication History

Published: 01 July 2021

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Huang ZTang FZhang YCao JLi CTang SLi JLee T(2024)Identity-Preserving Face Swapping via Dual Surrogate Generative ModelsACM Transactions on Graphics10.1145/367616543:5(1-19)Online publication date: 9-Aug-2024
https://dl.acm.org/doi/10.1145/3676165
Li SZhao RGuan QChen JZhang Y(2024)A 3D model encryption method supporting adaptive visual effects after decryptionAdvanced Engineering Informatics10.1016/j.aei.2023.10231959:COnline publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.aei.2023.102319
Zhang LZhao ZCong XZhang QGu SGao YZheng RYang WXu LYu J(2023)HACK: Learning a Parametric Head and Neck Model for High-fidelity AnimationACM Transactions on Graphics10.1145/359209342:4(1-20)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592093

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents