research-article

Video to fully automatic 3D hair model

Authors:

Linda G. Shapiro,

Ira Kemelmacher-ShlizermanAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 37, Issue 6

Article No.: 206, Pages 1 - 14

https://doi.org/10.1145/3272127.3275020

Published: 04 December 2018 Publication History

Abstract

Imagine taking a selfie video with your mobile phone and getting as output a 3D model of your head (face and 3D hair strands) that can be later used in VR, AR, and any other domain. State of the art hair reconstruction methods allow either a single photo (thus compromising 3D quality) or multiple views, but they require manual user interaction (manual hair segmentation and capture of fixed camera views that span full 360°). In this paper, we describe a system that can completely automatically create a reconstruction from any video (even a selfie video), and we don't require specific views, since taking your -90°, 90°, and full back views is not feasible in a selfie capture.

In the core of our system, in addition to the automatization components, hair strands are estimated and deformed in 3D (rather than 2D as in state of the art) thus enabling superior results. We provide qualitative, quantitative, and Mechanical Turk human studies that support the proposed system, and show results on a diverse variety of videos (8 different celebrity videos, 9 selfie mobile videos, spanning age, gender, hair length, type, and styling).

Supplementary Material

MP4 File (a206-liang.mp4)

Download
28.72 MB

References

[1]

Oleg Alexander, Graham Fyffe, Jay Busch, Xueming Yu, Ryosuke Ichikari, Andrew Jones, Paul Debevec, Jorge Jimenez, Etienne Danvoye, Bernardo Antionazzi, et al. 2013. Digital Ira: creating a real-time photoreal digital actor. In ACM SIGGRAPH 2013 Posters. ACM, 1.

Digital Library

[2]

B. Allen, B. Curless, and Z. Popović. 2003. The space of human body shapes: reconstruction and parameterization from range scans. In ACM Transactions on Graphics (TOG), Vol. 22. ACM, 587--594.

Digital Library

[3]

Thabo Beeler, Bernd Bickel, Paul Beardsley, Bob Sumner, and Markus Gross. 2010. High-quality single-shot capture of facial geometry. ACM Transactions on Graphics (TOG) 29, 4 (2010), 40.

Digital Library

[4]

Volker Blanz and Thomas Vetter. 1999. A morphable model for the synthesis of 3D faces. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 187--194.

Digital Library

[5]

Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. 2013. Facewarehouse: a 3d facial expression database for visual computing. (2013).

[6]

Chen Cao, Hongzhi Wu, Yanlin Weng, Tianjia Shao, and Kun Zhou. 2016. Real-time facial animation with image-based dynamic avatars. ACM Transactions on Graphics (TOG) 35, 4 (2016), 126.

Digital Library

[7]

Menglei Chai, Linjie Luo, Kalyan Sunkavalli, Nathan Carr, Sunil Hadap, and Kun Zhou. 2015. High-quality hair modeling from a single portrait photo. ACM Transactions on Graphics (TOG) 34, 6 (2015), 204.

Digital Library

[8]

Menglei Chai, Tianjia Shao, Hongzhi Wu, Yanlin Weng, and Kun Zhou. 2016. Autohair: Fully automatic hair modeling from a single image. ACM Transactions on Graphics (TOG) 35, 4 (2016), 116.

Digital Library

[9]

Menglei Chai, Lvdi Wang, Yanlin Weng, Xiaogang Jin, and Kun Zhou. 2013. Dynamic hair manipulation in images and videos. ACM Transactions on Graphics (TOG) 32, 4 (2013), 75.

Digital Library

[10]

Menglei Chai, Lvdi Wang, Yanlin Weng, Yizhou Yu, Baining Guo, and Kun Zhou. 2012. Single-view hair modeling for portrait manipulation. ACM Transactions on Graphics (TOG) 31, 4 (2012), 116.

Digital Library

[11]

Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. 2016. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. arXiv preprint arXiv:1606.00915 (2016).

[12]

Paul Debevec. 2012. The light stages and their applications to photoreal digital actors. SIGGRAPH Asia (2012).

[13]

Hongbo Fu, Yichen Wei, Chiew-Lan Tai, and Long Quan. 2007. Sketching hairstyles. In Proceedings of the 4th Eurographics workshop on Sketch-based interfaces and modeling. ACM, 31--36.

Digital Library

[14]

Michael Goesele, Noah Snavely, Brian Curless, Hugues Hoppe, and Steven M Seitz. 2007. Multi-view stereo for community photo collections. In Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on. IEEE, 1--8.

[15]

Hairbobo. 2017. Hairbobo. http://www.hairbobo.com/faxingtupian. (Sept. 2017).

[16]

Liwen Hu, Chongyang Ma, Linjie Luo, and Hao Li. 2014a. Robust hair capture using simulated examples. ACM Transactions on Graphics (TOG) 33, 4 (2014), 126.

Digital Library

[17]

Liwen Hu, Chongyang Ma, Linjie Luo, and Hao Li. 2015. Single-view hair modeling using a hairstyle database. ACM Transactions on Graphics (TOG) 34, 4 (2015), 125.

Digital Library

[18]

Liwen Hu, Chongyang Ma, Linjie Luo, Li-Yi Wei, and Hao Li. 2014b. Capturing braided hairstyles. ACM Transactions on Graphics (TOG) 33, 6 (2014), 225.

Digital Library

[19]

Liwen Hu, Shunsuke Saito, Lingyu Wei, Koki Nagano, Jaewoo Seo, Jens Fursund, Iman Sadeghi, Carrie Sun, Yen-Chun Chen, and Hao Li. 2017. Avatar Digitization from a Single Image for Real-time Rendering. ACM Trans. Graph. 36, 6, Article 195 (Nov. 2017), 14 pages.

Digital Library

[20]

Alexandru Eugen Ichim, Sofien Bouaziz, and Mark Pauly. 2015. Dynamic 3D avatar creation from hand-held video input. ACM Transactions on Graphics (TOG) 34, 4 (2015), 45.

Digital Library

[21]

Wenzel Jakob, Jonathan T Moon, and Steve Marschner. 2009. Capturing hair assemblies fiber by fiber. In ACM Transactions on Graphics (TOG), Vol. 28. ACM, 164.

Digital Library

[22]

Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM international conference on Multimedia. ACM, 675--678.

Digital Library

[23]

Michael Kazhdan and Hugues Hoppe. 2013. Screened poisson surface reconstruction. ACM Transactions on Graphics (TOG) 32, 3 (2013), 29.

Digital Library

[24]

Ira Kemelmacher-Shlizerman. 2016. Transfiguring portraits. ACM Transactions on Graphics (TOG) 35, 4 (2016), 94.

Digital Library

[25]

Ira Kemelmacher-Shlizerman and Ronen Basri. 2011. 3d face reconstruction from a single image using a single reference face shape. Pattern Analysis and Machine Intelligence, IEEE Transactions on 33, 2 (2011), 394--405.

Digital Library

[26]

Ira Kemelmacher-Shlizerman and Steven M Seitz. 2011. Face reconstruction in the wild. In Computer Vision (ICCV), 2011 IEEE International Conference on. IEEE, 1746--1753.

Digital Library

[27]

Aldo Laurentini. 1994. The visual hull concept for silhouette-based image understanding. IEEE Transactions on pattern analysis and machine intelligence 16, 2 (1994), 150--162.

Digital Library

[28]

Shu Liang, Ira Kemelmacher-Shlizerman, and Linda G Shapiro. 2014. 3d face hallucination from a single depth frame. In 3D Vision (3DV), 2014 2nd international conference on, Vol. 1. IEEE, 31--38.

Digital Library

[29]

Shu Liang, Linda G Shapiro, and Ira Kemelmacher-Shlizerman. 2016. Head reconstruction from internet photos. In European Conference on Computer Vision. Springer, 360--374.

[30]

Sifei Liu, Jianping Shi, Ji Liang, and Ming-Hsuan Yang. 2017. Face Parsing via Recurrent Propagation. arXiv preprint arXiv:1708.01936 (2017).

[31]

Sifei Liu, Jimei Yang, Chang Huang, and Ming-Hsuan Yang. 2015. Multi-objective convolutional learning for face labeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3451--3459.

[32]

Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3431--3440.

[33]

Linjie Luo, Hao Li, and Szymon Rusinkiewicz. 2013. Structure-aware hair capture. ACM Transactions on Graphics (TOG) 32, 4 (2013), 76.

Digital Library

[34]

Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2012. Hierarchical face parsing via deep learning. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on. IEEE, 2480--2487.

Digital Library

[35]

Fabio Maninchedda, Christian Häne, Bastien Jacquet, Amaël Delaunoy, and Marc Pollefeys. 2016. Semantic 3D Reconstruction of Heads. In European Conference on Computer Vision. Springer, 667--683.

[36]

Richard A Newcombe, Dieter Fox, and Steven M Seitz. 2015. DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 343--352.

[37]

Sylvain Paris, Hector M Briceño, and François X Sillion. 2004. Capture of hair geometry from multiple images. In ACM Transactions on Graphics (TOG), Vol. 23. ACM, 712--719.

Digital Library

[38]

Sylvain Paris, Will Chang, Oleg I Kozhushnyan, Wojciech Jarosz, Wojciech Matusik, Matthias Zwicker, and Frédo Durand. 2008. Hair photobooth: geometric and photometric acquisition of real hairstyles. In ACM Transactions on Graphics (TOG), Vol. 27. ACM, 30.

Digital Library

[39]

Elad Richardson, Matan Sela, and Ron Kimmel. 2016. 3D face reconstruction by learning from synthetic data. In 3D Vision (3DV), 2016 Fourth International Conference on. IEEE, 460--469.

[40]

Elad Richardson, Matan Sela, Roy Or-El, and Ron Kimmel. 2017. Learning detailed face reconstruction from a single image. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 5553--5562.

[41]

Shunsuke Saito, Lingyu Wei, Liwen Hu, Koki Nagano, and Hao Li. 2016. Photorealistic Facial Texture Inference Using Deep Neural Networks. arXiv preprint arXiv:1612.00523 (2016).

[42]

Olga Sorkine, Daniel Cohen-Or, Yaron Lipman, Marc Alexa, Christian Rössl, and H-P Seidel. 2004. Laplacian surface editing. In Proceedings of the 2004 Eurographics/ACM SIGGRAPH symposium on Geometry processing. ACM, 175--184.

Digital Library

[43]

Supasorn Suwajanakorn, Ira Kemelmacher-Shlizerman, and Steven M Seitz. 2014. Total moving face reconstruction. In Computer Vision-ECCV 2014.

[44]

Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman. 2015. What Makes Tom Hanks Look Like Tom Hanks. In Proceedings of the IEEE International Conference on Computer Vision. 3952--3960.

Digital Library

[45]

Justus Thies, Michael Zollhoefer, Matthias Niessner, Levi Valgaerts, Marc Stamminger, and Christian Theobalt. 2015. Real-time Expression Transfer for Facial Reenactment. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) (2015).

Digital Library

[46]

Anh Tuan Tran, Tal Hassner, Iacopo Masi, and Gérard Medioni. 2017. Regressing robust and discriminative 3D morphable models with a very deep neural network. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 1493--1502.

[47]

Nuttapon Vanakittistien, Attawith Sudsang, and Nuttapong Chentanez. 2016. 3D hair model from small set of images. In Proceedings of the 9th International Conference on Motion in Games. ACM, 85--90.

Digital Library

[48]

Dan Wang, Xiujuan Chai, Hongming Zhang, Hong Chang, Wei Zeng, and Shiguang Shan. 2011. A novel coarse-to-fine hair segmentation method. In Automatic Face & Gesture Recognition and Workshops (FG 2011), 2011 IEEE International Conference on. IEEE, 233--238.

[49]

Kelly Ward, Florence Bertails, Tae-Yong Kim, Stephen R Marschner, Marie-Paule Cani, and Ming C Lin. 2007. A survey on hair modeling: Styling, simulation, and rendering. IEEE Transactions on Visualization and Computer Graphics 13, 2 (2007).

Digital Library

[50]

Yanlin Weng, Lvdi Wang, Xiao Li, Menglei Chai, and Kun Zhou. 2013. Hair interpolation for portrait morphing. In Computer Graphics Forum, Vol. 32. Wiley Online Library, 79--84.

[51]

Changchang Wu. 2011. VisualSFM: A visual structure from motion system. (2011).

[52]

Xuehan Xiong and Fernando De la Torre. 2013. Supervised Descent Method and its Applications to Face Alignment. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Digital Library

[53]

Yaser Yacoob and Larry S Davis. 2006. Detection and analysis of hair. IEEE transactions on pattern analysis and machine intelligence 28, 7 (2006), 1164--1169.

Digital Library

[54]

Meng Zhang, Menglei Chai, Hongzhi Wu, Hao Yang, and Kun Zhou. 2017. Adata-driven approach to four-view image-based hair modeling. ACM Transactions on Graphics (TOG) 36, 4 (2017), 156.

Digital Library

[55]

Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, and Philip HS Torr. 2015. Conditional random fields as recurrent neural networks. In Proceedings of the IEEE International Conference on Computer Vision. 1529--1537.

Digital Library

[56]

Michael Zollhöfer, Matthias Nießner, Shahram Izadi, Christoph Rehmann, Christopher Zach, Matthew Fisher, Chenglei Wu, Andrew Fitzgibbon, Charles Loop, Christian Theobalt, et al. 2014. Real-time non-rigid reconstruction using an rgb-d camera. ACM Transactions on Graphics, TOG (2014).

Digital Library

Cited By

Wang ZNam GBozic ACao CSaragih JZollhöfer MHodgins J(2024)A Local Appearance Model for Volumetric Capture of Diverse Hairstyles2024 International Conference on 3D Vision (3DV)10.1109/3DV62453.2024.00013(190-200)Online publication date: 18-Mar-2024
https://doi.org/10.1109/3DV62453.2024.00013
Li CJin LZheng YYu YHan X(2023)EMS: 3D Eyebrow Modeling from Single-View ImagesACM Transactions on Graphics10.1145/361832342:6(1-19)Online publication date: 5-Dec-2023
https://dl.acm.org/doi/10.1145/3618323
Zhou YChai MPepe AGross MBeeler T(2023)GroomGen: A High-Quality Generative Hair Model Using Hierarchical Latent RepresentationsACM Transactions on Graphics10.1145/361830942:6(1-16)Online publication date: 5-Dec-2023
https://dl.acm.org/doi/10.1145/3618309
Show More Cited By

Index Terms

Video to fully automatic 3D hair model
1. Computing methodologies
  1. Computer graphics
    1. Shape modeling

Recommendations

Structure-aware hair capture

Existing hair capture systems fail to produce strands that reflect the structures of real-world hairstyles. We introduce a system that reconstructs coherent and plausible wisps aware of the underlying hair structures from a set of still images without ...
Wide-Baseline Hair Capture Using Strand-Based Refinement
CVPR '13: Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition

We propose a novel algorithm to reconstruct the 3D geometry of human hairs in wide-baseline setups using strand-based refinement. The hair strands are first extracted in each 2D view, and projected onto the 3D visual hull for initialization. The 3D ...
Editable 3D Model Based on Generative Adversarial Network
CONF-CDS 2021: The 2nd International Conference on Computing and Data Science

In recent years, 3D modeling technology has become more and more popular in emerging VR and AR applications. Due to the complexity and diversity of hair styles in the real world, hair modeling is still challenging, so it is very important to study hair ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 37, Issue 6

December 2018

1401 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3272127

Editor:
Takeo Igarashi
The University of Tokyo, Japan

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 December 2018

Published in TOG Volume 37, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
505
Total Downloads

Downloads (Last 12 months)22
Downloads (Last 6 weeks)1

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang ZNam GBozic ACao CSaragih JZollhöfer MHodgins J(2024)A Local Appearance Model for Volumetric Capture of Diverse Hairstyles2024 International Conference on 3D Vision (3DV)10.1109/3DV62453.2024.00013(190-200)Online publication date: 18-Mar-2024
https://doi.org/10.1109/3DV62453.2024.00013
Li CJin LZheng YYu YHan X(2023)EMS: 3D Eyebrow Modeling from Single-View ImagesACM Transactions on Graphics10.1145/361832342:6(1-19)Online publication date: 5-Dec-2023
https://dl.acm.org/doi/10.1145/3618323
Zhou YChai MPepe AGross MBeeler T(2023)GroomGen: A High-Quality Generative Hair Model Using Hierarchical Latent RepresentationsACM Transactions on Graphics10.1145/361830942:6(1-16)Online publication date: 5-Dec-2023
https://dl.acm.org/doi/10.1145/3618309
Shen YSaito SWang ZMaury OWu CHodgins JZheng YNam G(2023)CT2Hair: High-Fidelity 3D Hair Modeling using Computed TomographyACM Transactions on Graphics10.1145/359210642:4(1-13)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592106
Tretschk EKairanda NB R MDabral RKortylewski AEgger BHabermann MFua PTheobalt CGolyanik V(2023)State of the Art in Dense Monocular Non‐Rigid 3D ReconstructionComputer Graphics Forum10.1111/cgf.1477442:2(485-520)Online publication date: 23-May-2023
https://doi.org/10.1111/cgf.14774
Kuang ZChen YFu HZhou KZheng Y(2022)DeepMVSHair: Deep Hair Modeling from Sparse ViewsSIGGRAPH Asia 2022 Conference Papers10.1145/3550469.3555385(1-8)Online publication date: 29-Nov-2022
https://dl.acm.org/doi/10.1145/3550469.3555385
Wang XGuo YYang ZZhang J(2022)Prior-Guided Multi-View 3D Head ReconstructionIEEE Transactions on Multimedia10.1109/TMM.2021.311148524(4028-4040)Online publication date: 2022
https://doi.org/10.1109/TMM.2021.3111485
Chen HChen MLu L(2022)3D Printed hair modeling from strand-level hairstylesGraphical Models10.1016/j.gmod.2022.101135121:COnline publication date: 15-Jun-2022
https://dl.acm.org/doi/10.1016/j.gmod.2022.101135
Mane TBayramova ADaniilidis KMordohai PBernardis E(2022)Single-camera 3D head fitting for mixed reality clinical applicationsComputer Vision and Image Understanding10.1016/j.cviu.2022.103384218(103384)Online publication date: Apr-2022
https://doi.org/10.1016/j.cviu.2022.103384
Qiu SLi CWang CQin H(2021)A Rapid, End‐to‐end, Generative Model for Gaseous Phenomena from Limited ViewsComputer Graphics Forum10.1111/cgf.1427040:6(242-257)Online publication date: 27-Apr-2021
https://doi.org/10.1111/cgf.14270
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents