research-article

Open access

Fast generation of realistic virtual humans

Authors:

Jascha Achenbach,

Thomas Waltemate,

Marc Erich Latoschik,

Mario BotschAuthors Info & Claims

VRST '17: Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology

Article No.: 12, Pages 1 - 10

https://doi.org/10.1145/3139131.3139154

Published: 08 November 2017 Publication History

Abstract

In this paper we present a complete pipeline to create ready-to-animate virtual humans by fitting a template character to a point set obtained by scanning a real person using multi-view stereo reconstruction. Our virtual humans are built upon a holistic character model and feature a detailed skeleton, fingers, eyes, teeth, and a rich set of facial blendshapes. Furthermore, due to the careful selection of techniques and technology, our reconstructed humans are quite realistic in terms of both geometry and texture. Since we represent our models as single-layer triangle meshes and animate them through standard skeleton-based skinning and facial blendshapes, our characters can be used in standard VR engines out of the box. By optimizing for computation time and minimizing manual intervention, our reconstruction pipeline is capable of processing whole characters in less than ten minutes.

Supplementary Material

MP4 File (a12-achenbach.mp4)

Download
80.43 MB

References

[1]

Jascha Achenbach, Eduard Zell, and Mario Botsch. 2015. Accurate Face Reconstruction through Anisotropic Fitting and Eye Correction. In Proc. of Vision, Modeling & Visualization. 1--8.

[2]

Oleg Alexander, Mike Rogers, William Lambeth, Matt Chiang, and Paul Debevec. 2009. The Digital Emily Project: Photoreal Facial Modeling and Animation. In SIGGRAPH 2009 Courses. ACM, 1--15.

Digital Library

[3]

Brett Allen, Brian Curless, and Zoran Popović. 2003. The Space of Human Body Shapes: Reconstruction and Parameterization from Range Scans. ACM Transactions on Graphics 22, 3 (2003), 587--594.

Digital Library

[4]

Brett Allen, Brian Curless, Zoran Popović, and Aaron Hertzmann. 2006. Learning a Correlated Model of Identity and Pose-Dependent Body Shape Variation for Real-Time Synthesis. In Proc. of Eurographics Symposium on Computer Animation. 147--156.

[5]

Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. 2005. SCAPE: Shape Completion and Animation of People. ACM Transactions on Graphics 24, 3 (2005), 408--416.

Digital Library

[6]

Akshay Asthana, Stefanos Zafeiriou, Shiyang Cheng, and Maja Pantic. 2013. Robust discriminative response map fitting with constrained local models. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition. 3444--3451.

Digital Library

[7]

Autodesk. 2014. Character Generator. https://charactergenerator.autodesk.com/. (2014).

[8]

Domna Banakou and Mel Slater. 2014. Body ownership causes illusory self-attribution of speaking and influences subsequent real speaking. Proceedings of the National Academy of Sciences 111, 49 (2014), 17678--17683.

[9]

Ilya Baran and Jovan Popović. 2007. Automatic Rigging and Animation of 3D Characters. ACM Transactions on Graphics 26, 3, Article 72 (2007), 8 pages.

Digital Library

[10]

Thabo Beeler, Bernd Bickel, Paul Beardsley, Bob Sumner, and Markus Gross. 2010. High-quality Single-shot Capture of Facial Geometry. ACM Transactions on Graphics 29, 4 (2010), 1--9.

Digital Library

[11]

Volker Blanz and Thomas Vetter. 1999. A morphable model for the synthesis of 3D faces. In Proc. of SIGGRAPH. 187--194.

Digital Library

[12]

Federica Bogo, Michael J Black, Matthew Loper, and Javier Romero. 2015. Detailed full-body reconstructions of moving people from monocular RGB-D sequences. In Proc. of IEEE International Conference on Computer Vision. 2300--2308.

Digital Library

[13]

Federica Bogo, Javier Romero, Matthew Loper, and Michael J. Black. 2014. FAUST: Dataset and Evaluation for 3D Mesh Registration. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition. 3794--3801.

Digital Library

[14]

Sofien Bouaziz, Andrea Tagliasacchi, and Mark Pauly. 2014. Dynamic 2D/3D Registration. In Eurographics Tutorials.

[15]

Sofien Bouaziz, Yangang Wang, and Mark Pauly. 2013. Online Modeling for Realtime Facial Animation. ACM Transactions on Graphics 32, 4, Article 40 (2013), 10 pages.

Digital Library

[16]

Samuel R Buss. 2004. Introduction to inverse kinematics with jacobian transpose, pseudoinverse and damped least squares methods. IEEE Journal of Robotics and Automation 17 (2004), 1--19.

[17]

Chen Cao, Qiming Hou, and Kun Zhou. 2014a. Displaced Dynamic Expression Regression for Real-time Facial Tracking and Animation. ACM Transactions on Graphics 33, 4 (2014), 1--10.

Digital Library

[18]

Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. 2014b. FaceWare-house: A 3D facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics 20, 3 (2014), 413--425.

Digital Library

[19]

P. Ekman and W. Friesen. 1978. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press.

[20]

Andrew Feng, Dan Casas, and Ari Shapiro. 2015. Avatar Reshaping and Automatic Rigging Using a Deformable Model. In Proc. of ACM Motion in Games. 57--64.

Digital Library

[21]

Andrew Feng, Evan Suma Rosenberg, and Ari Shapiro. 2017. Just-in-time, viable, 3-D avatars from scans. Computer Animation and Virtual Worlds 28 (2017), 3--4.

[22]

Andrew Feng, Ari Shapiro, Wang Ruizhe, Mark Bolas, Gerard Medioni, and Evan Suma. 2014. Rapid Avatar Capture and Simulation Using Commodity Depth Sensors. In SIGGRAPH 2014 Talks. ACM.

Digital Library

[23]

Pablo Garrido, Michael Zollhöfer, Dan Casas, Levi Valgaerts, Kiran Varanasi, Patrick Pérez, and Christian Theobalt. 2016. Reconstruction of Personalized 3D Face Rigs from Monocular Video. ACM Transactions on Graphics 35, 3, Article 28 (2016), 15 pages.

[24]

Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong, Jay Busch, Xueming Yu, and Paul Debevec. 2011. Multiview Face Capture Using Polarized Spherical Gradient Illumination. ACM Transactions on Graphics 30, 6, Article 129 (2011), 10 pages.

Digital Library

[25]

Mar González-Franco, Daniel Perez-Marcos, Bernhard Spanlang, and Mel Slater. 2010. The contribution of real-time mirror reflections of motor actions on virtual body ownership in an immersive virtual environment. In Proc. of IEEE Virtual Reality Conference. 111--114.

Digital Library

[26]

P. Guan, A. Weiss, A. Balan, and M. J. Black. 2009. Estimating human shape and pose from a single image. In Proc. of International Conference on Computer Vision. 1381--1388.

[27]

Nils Hasler, Carsten Stoll, Martin Sunkel, Bodo Rosenhahn, and H-P Seidel. 2009. A statistical model of human pose and body shape. Computer Graphics Forum 28, 2 (2009), 337--346.

[28]

David A. Hirshberg, Matthew Loper, Eric Rachlin, and Michael J. Black. 2012. Coregistration: Simultaneous Alignment and Modeling of Articulated 3D Shape. In Proc. of European Conference on Computer Vision. 242--255.

Digital Library

[29]

Berthold K. P. Horn. 1987. Closed-form solution of absolute orientation using unit quaternions. Journal of the Optical Society of America A 4, 4 (1987), 629--642.

[30]

Pei-Lun Hsieh, Chongyang Ma, Jihun Yu, and Hao Li. 2015. Unconstrained Realtime Facial Performance Capture. In Proc. of Computer Vision and Pattern Recognition. 1675--1683.

[31]

Alexandru Eugen Ichim, Sofien Bouaziz, and Mark Pauly. 2015. Dynamic 3D Avatar Creation from Hand-held Video Input. ACM Transactions on Graphics 34, 4, Article 45 (2015), 14 pages.

Digital Library

[32]

Tao Ju, Scott Schaefer, and Joe Warren. 2005. Mean Value Coordinates for Closed Triangular Meshes. ACM Transactions on Graphics 24, 3 (2005), 561--566.

Digital Library

[33]

Marc Latoschik, Daniel Roth, Dominik Gall, Jascha Achenbach, Thomas Waltemate, and Mario Botsch. 2017. The Effect of Avatar Realism in Immersive Social Virtual Realities. In Proc. of ACM Symposium on Virtual Reality Software and Technology. to appear.

Digital Library

[34]

Marc Erich Latoschik, Jean-Luc Lugrin, and Daniel Roth. 2016. FakeMi: a fake mirror system for avatar embodiment studies. In Proc. of ACM Virtual Reality Software and Technology. 73--76.

Digital Library

[35]

J. P. Lewis, Ken Anjyo, Taehyun Rhee, Mengjie Zhang, Fred Pighin, and Zhigang Deng. 2014. Practice and Theory of Blendshape Facial Models. In Eurographics 2014 - State of the Art Reports.

[36]

Hao Li, Etienne Vouga, Anton Gudym, Linjie Luo, Jonathan T. Barron, and Gleb Gusev. 2013. 3D Self-portraits. ACM Transactions on Graphics 32, 6, Article 187 (2013), 9 pages.

Digital Library

[37]

Hao Li, Thibaut Weise, and Mark Pauly. 2010. Example-based Facial Rigging. ACM Transactions on Graphics 29, 4, Article 32 (2010), 6 pages.

Digital Library

[38]

Shu Liang, Ira Kemelmacher-Shlizerman, and Linda G. Shapiro. 2014. 3D Face Hallucination from a Single Depth Frame. In Proc. of International Conference on 3D Vision. 31--38.

[39]

Matthew Loper, Naureen Mahmood, and Michael J. Black. 2014. MoSh: Motion and Shape Capture from Sparse Markers. ACM Transactions on Graphics 33, 6, Article 220 (2014), 13 pages.

Digital Library

[40]

Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. 2015. SMPL: A Skinned Multi-person Linear Model. ACM Transactions on Graphics 34, 6, Article 248 (2015), 16 pages.

Digital Library

[41]

Jean-Luc Lugrin, Johanna Latt, and Marc Erich Latoschik. 2015. Anthropomorphism and illusion of virtual body ownership. In Proc. of the 25th International Conference on Artificial Reality and Telexistence and 20th Eurographics Symposium on Virtual Environments. 1--8.

Digital Library

[42]

C. Malleson, M. Kosek, M. Klaudiny, I. Huerta, J. C. Bazin, A. Sorkine-Hornung, M. Mine, and K. Mitchell. 2017. Rapid one-shot acquisition of dynamic VR avatars. In Proc. of IEEE Virtual Reality Conference. 131--140.

[43]

Tabitha C Peck, Sofia Seinfeld, Salvatore M Aglioti, and Mel Slater. 2013. Putting yourself in the skin of a black avatar reduces implicit racial bias. Consciousness and cognition 22, 3 (2013), 779--787.

[44]

Patrick Pérez, Michel Gangnet, and Andrew Blake. 2003. Poisson Image Editing. ACM Transactions on Graphics 22, 3 (2003), 313--318.

Digital Library

[45]

Leonid Pishchulin, Stefanie Wuhrer, Thomas Helten, Christian Theobalt, and Bernt Schiele. 2017. Building Statistical Shape Spaces for 3D Human Modeling. Pattern Recognition (2017), 276--286.

Digital Library

[46]

Gerard Pons-Moll, Sergi Pujades, Sonny Hu, and Michael Black. 2017. ClothCap: Seamless 4D Clothing Capture and Retargeting. ACM Transactions on Graphics 36, 4, Article 73 (2017), 15 pages.

Digital Library

[47]

Daniel Roth, Kristoffer Waldow, Felix Stetter, Gary Bente, Marc Erich Latoschik, and Arnulph Fuhrmann. 2016. SIAMC: a socially immersive avatar mediated communication platform. In Proc. of ACM Virtual Reality Software and Technology. 357--358.

Digital Library

[48]

Fuhao Shi, Hsiang-Tao Wu, Xin Tong, and Jinxiang Chai. 2014. Automatic Acquisition of High-fidelity Facial Performances Using Monocular Videos. ACM Transactions on Graphics 33, 6, Article 222 (2014), 13 pages.

Digital Library

[49]

Leonid Sigal, Alexandru O. Balan, and Michael J. Black. 2007. Combined discriminative and generative articulated pose and non-rigid shape estimation. In Proc. of International Conference on Neural Information Processing Systems. 1337--1344.

[50]

Mel Slater, Bernhard Spanlang, Maria V Sanchez-Vives, and Olaf Blanke. 2010. First person experience of body transfer in virtual reality. PloS one 5, 5 (2010).

[51]

Matthias Straka, Stefan Hauswiesner, Matthias Ruther, and Horst Bischof. 2012. Rapid Skin: Estimating the 3D Human Pose and Shape in Real-Time. In Proc. of International Conference on 3D Imaging, Modeling, Processing, Visualization Transmission. 41--48.

Digital Library

[52]

Jürgen Sturm, Erik Bylow, Fredrik Kahl, and Daniel Cremers. 2013. CopyMe3D: Scanning and Printing Persons in 3D. In Proc. of German Conference on Pattern Recognition. 405--414.

[53]

Robert W. Sumner and Jovan Popović. 2004. Deformation Transfer for Triangle Meshes. ACM Transactions on Graphics 23, 3 (2004), 399--405.

Digital Library

[54]

Justus Thies, Michael Zollhöfer, Matthias Nießner, Levi Valgaerts, Marc Stamminger, and Christian Theobalt. 2015. Real-time Expression Transfer for Facial Reenactment. ACM Transactions on Graphics 34, 6, Article 183 (2015), 14 pages.

Digital Library

[55]

J. Thies, M. Zollhöfer, M. Stamminger, C. Theobalt, and M. Nießner. 2016. Face2Face: Real-time Face Capture and Reenactment of RGB Videos. In Proc. of IEEE Computer Vision and Pattern Recognition. 2387--2395.

Digital Library

[56]

Jing Tong, Jin Zhou, Ligang Liu, Zhigeng Pan, and Hao Yan. 2012. Scanning 3D Full Human Bodies Using Kinects. IEEE Transactions on Visualization and Computer Graphics 18, 4 (2012), 643--650.

Digital Library

[57]

Aggeliki Tsoli, Naureen Mahmood, and Michael J. Black. 2014. Breathing Life into Shape: Capturing, Modeling and Animating 3D Human Breathing. ACM Transactions on Graphics 33, 4, Article 52 (2014), 11 pages.

Digital Library

[58]

Thibaut Weise, Sofien Bouaziz, Hao Li, and Mark Pauly. 2011. Realtime Performance-based Facial Animation. ACM Transactions on Graphics 30, 4, Article 77 (2011), 10 pages.

Digital Library

[59]

Alexander Weiss, David Hirshberg, and Michael J. Black. 2011. Home 3D Body Scans from Noisy Image and Range Data. In Proc. of IEEE International Conference on Computer Vision. 1951--1958.

Digital Library

[60]

Chenglei Wu, Derek Bradley, Markus Gross, and Thabo Beeler. 2016. An Anatomically-Constrained Local Deformation Model for Monocular Face Capture. ACM Transactions on Graphics 35, 4, Article 115 (2016), 12 pages.

Digital Library

[61]

Stefanie Wuhrer, Leonid Pishchulin, Alan Brunton, Chang Shu, and Jochen Lang. 2014. Estimation of Human Body Shape and Posture Under Clothing. Computer Vision and Image Understanding 127 (2014), 31--42.

Digital Library

Cited By

Mal DDöllinger NWolf EWenninger SBotsch MWienrich CLatoschik M(2024)Am I the odd one? Exploring (in)congruencies in the realism of avatars and virtual others in virtual realityFrontiers in Virtual Reality10.3389/frvir.2024.14170665Online publication date: 19-Jul-2024
https://doi.org/10.3389/frvir.2024.1417066
Cao Q(2024)Research on Virtual Character Behavior Simulation and Control Algorithm in Digital Movie ProductionApplied Mathematics and Nonlinear Sciences10.2478/amns-2024-33749:1Online publication date: 18-Nov-2024
https://doi.org/10.2478/amns-2024-3374
Keppel JStrauss MGruenefeld USchneegass S(2024)Magic Mirror: Designing a Weight Change Visualization for Domestic UseProceedings of the ACM on Human-Computer Interaction10.1145/36981498:ISS(569-596)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3698149
Show More Cited By

Index Terms

Fast generation of realistic virtual humans
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision representations
        Appearance and texture representations
  2. Computer graphics
    1. Shape modeling
      1. Mesh geometry models

Recommendations

Realistic Virtual Humans from Smartphone Videos
VRST '20: Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology

This paper introduces an automated 3D-reconstruction method for generating high-quality virtual humans from monocular smartphone cameras. The input of our approach are two video clips, one capturing the whole body and the other providing detailed close-...
Virtual Human Representation and Communication in VLNet

The realism in participant representation in networked virtual environments involves two elements: believable appearance and realistic movements. Using virtual human figures for participant representation fulfills these functionalities with realism, as ...
Investigating Social Distances between Humans, Virtual Humans and Virtual Robots in Mixed Reality
AAMAS '18: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems

Mixed reality environments offer new potentials for the design of compelling social interaction experiences with virtual characters. In this paper, we summarise initial experiments we are conducting in which we measure comfortable social distances ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

VRST '17: Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology

November 2017

437 pages

ISBN:9781450355483

DOI:10.1145/3139131

General Chairs:
Morten Fjeld
Chalmers University of Technology
,
Marco Fratarcangeli
Chalmers University of Technology
,
Daniel Sjölie
University of Gothenburg
,
Program Chairs:
Oliver Staadt
University of Rostock
,
Jonas Unger
Linköping University

Copyright © 2017 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 November 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

VRST '17

Sponsor:

VRST '17: 23rd ACM Symposium on Virtual Reality Software and Technology

November 8 - 10, 2017

Gothenburg, Sweden

Acceptance Rates

Overall Acceptance Rate 66 of 254 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

80
Total Citations
View Citations
2,453
Total Downloads

Downloads (Last 12 months)324
Downloads (Last 6 weeks)32

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Mal DDöllinger NWolf EWenninger SBotsch MWienrich CLatoschik M(2024)Am I the odd one? Exploring (in)congruencies in the realism of avatars and virtual others in virtual realityFrontiers in Virtual Reality10.3389/frvir.2024.14170665Online publication date: 19-Jul-2024
https://doi.org/10.3389/frvir.2024.1417066
Cao Q(2024)Research on Virtual Character Behavior Simulation and Control Algorithm in Digital Movie ProductionApplied Mathematics and Nonlinear Sciences10.2478/amns-2024-33749:1Online publication date: 18-Nov-2024
https://doi.org/10.2478/amns-2024-3374
Keppel JStrauss MGruenefeld USchneegass S(2024)Magic Mirror: Designing a Weight Change Visualization for Domestic UseProceedings of the ACM on Human-Computer Interaction10.1145/36981498:ISS(569-596)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3698149
Herforth JBotev J(2024)A3CplusProceedings of the 16th International Workshop on Immersive Mixed and Virtual Environment Systems10.1145/3652212.3652215(8-14)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3652212.3652215
Merz CTschanter JKern FLugrin JWienrich CLatoschik M(2024)Pipelining Processors for Decomposing Character AnimationProceedings of the 30th ACM Symposium on Virtual Reality Software and Technology10.1145/3641825.3689533(1-2)Online publication date: 9-Oct-2024
https://dl.acm.org/doi/10.1145/3641825.3689533
Unruh FLugrin JLatoschik M(2024)Out-Of-Virtual-Body Experiences: Virtual Disembodiment Effects on Time Perception in VRProceedings of the 30th ACM Symposium on Virtual Reality Software and Technology10.1145/3641825.3687717(1-11)Online publication date: 9-Oct-2024
https://dl.acm.org/doi/10.1145/3641825.3687717
Mal DWolf EDöllinger NBotsch MWienrich CLatoschik M(2024)From 2D-Screens to VR: Exploring the Effect of Immersion on the Plausibility of Virtual HumansExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650773(1-8)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650773
Döllinger NMal DKeppler SWolf EBotsch MIsrael JLatoschik MWienrich C(2024)Virtual Body Swapping: A VR-Based Approach to Embodied Third-Person Self-Processing in Mind-Body TherapyProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642328(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642328
Wenninger SKemper FSchwanecke UBotsch M(2024) T ailor M e : Self‐Supervised Learning of an Anatomically Constrained Volumetric Human Shape Model Computer Graphics Forum10.1111/cgf.1504643:2Online publication date: 30-Apr-2024
https://doi.org/10.1111/cgf.15046
Fielder MWolf EDöllinger NMal DBotsch MLatoschik MWienrich C(2024)From Avatars to Agents: Self-Related Cues Through Embodiment and Personalization Affect Body Perception in Virtual RealityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345621130:11(7386-7396)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3456211
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents