research-article

Lightweight eye capture using a parametric model

Authors:

Pascal Bérard,

Thabo BeelerAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 35, Issue 4

Article No.: 117, Pages 1 - 12

https://doi.org/10.1145/2897824.2925962

Published: 11 July 2016 Publication History

Abstract

Facial scanning has become ubiquitous in digital media, but so far most efforts have focused on reconstructing the skin. Eye reconstruction, on the other hand, has received only little attention, and the current state-of-the-art method is cumbersome for the actor, time-consuming, and requires carefully setup and calibrated hardware. These constraints currently make eye capture impractical for general use. We present the first approach for high-quality lightweight eye capture, which leverages a database of pre-captured eyes to guide the reconstruction of new eyes from much less constrained inputs, such as traditional single-shot face scanners or even a single photo from the internet. This is accomplished with a new parametric model of the eye built from the database, and a novel image-based model fitting algorithm. Our method provides both automatic reconstructions of real eyes, as well as artistic control over the parameters to generate user-specific eyes.

Supplementary Material

ZIP File (a117-berard-supp.zip)

Supplemental files.

Download
165.02 MB

References

[1]

Allen, B., Curless, B., and Popović, Z. 2003. The space of human body shapes: reconstruction and parameterization from range scans. In ACM Transactions on Graphics (TOG), vol. 22, ACM, 587--594.

Digital Library

[2]

Amberg, B., Romdhani, S., and Vetter, T. 2007. Optimal step nonrigid icp algorithms for surface registration. In Computer Vision and Pattern Recognition, 2007. CVPR'07. IEEE Conference on, IEEE, 1-8.

[3]

Anguelov, D., Srinivasan, P., Koller, D., Thrun, S., Rodgers, J., and Davis, J. 2005. Scape: shape completion and animation of people. In ACM Transactions on Graphics (TOG), vol. 24, ACM, 408-416.

Digital Library

[4]

Beeler, T., Bickel, B., Sumner, R., Beardsley, P., and Gross, M. 2010. High-quality single-shot capture of facial geometry. ACM Trans. Graphics (Proc. SIGGRAPH).

Digital Library

[5]

Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C, Sumner, R. W., and Gross, M. 2011. High-quality passive facial performance capture using anchor frames. ACM Trans. Graphics (Proc. SIGGRAPH) 30, 4, 75.

Digital Library

[6]

Bérard, P., Bradley, D., Nitti, M., Beeler, T., and Gross, M. 2014. High-quality capture of eyes. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 33, 6, 223:1--223:12.

Digital Library

[7]

Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In Proc. of the 26th annual conference on Computer graphics and interactive techniques, 187--194.

Digital Library

[8]

Boykov, Y., and Kolmogorov, V. 2004. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. Pattern Analysis and Machine Intelligence, IEEE Transactions on 26, 9, 1124--1137.

Digital Library

[9]

Bradley, D., Heidrich, W., Popa, T., and Sheffer, A. 2010. High resolution passive facial performance capture. ACM Trans. Graphics (Proc. SIGGRAPH) 29, 4, 41.

Digital Library

[10]

Brown, B. J., and Rusinkiewicz, S. 2004. Non-rigid range-scan alignment using thin-plate splines. In 3D Data Processing, Visualization and Transmission, 2004. 3DPVT 2004. Proceedings. 2nd International Symposium on, IEEE, 759--765.

Digital Library

[11]

Cao, C., Hou, Q., and Zhou, K. 2014. Displaced dynamic expression regression for real-time facial tracking and animation. ACM Transactions on Graphics (TOG) 33, 4, 43.

Digital Library

[12]

Chen, K., Johan, H., and Mueller-Wittig, W. 2013. Simple and efficient example-based texture synthesis using tiling and deformation. In Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, ACM, 145--152.

Digital Library

[13]

Efros, A. A., and Freeman, W. T. 2001. Image quilting for texture synthesis and transfer. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, ACM, 341--346.

Digital Library

[14]

Flickr. 2006. Mohammed Alnaser - Mr. Falcon. Creative Commons - Attribution 2.0. https://www.flickr.com/photos/69er/324313066.

[15]

Francois G. Gautron P. Breton G. and Bouatouch K. 2009. Image-based modeling of the human eye. IEEE TVCG 15, 5, 815--827.

Digital Library

[16]

Funkhouser T. Kazhdan M. Shilane P. Min P. Kiefer W. Tal A. Rusinkiewicz S. and Dobkin D. 2004. Modeling by example. In ACM Transactions on Graphics (TOG), vol. 23, ACM, 652--663.

Digital Library

[17]

Fyffe G. Hawkins T. Watts C. Ma W.-C. and Debevec P. 2011. Comprehensive facial performance capture. In Eurographics.

[18]

Fyffe, G., Jones, A., Alexander, O., Ichikari, R., and Debevec, P. 2014. Driving high-resolution facial scans with video performance capture. ACM Trans. Graphics 34, 1, 8:1--8:14.

Digital Library

[19]

Garrido, P., Valgaerts, L., Wu, C., and Theobalt, C. 2013. Reconstructing detailed dynamic face geometry from monocular video. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 32, 6, 158.

Digital Library

[20]

Ghosh, A., Fyffe, G., Tunwattanapong, B., Busch, J., Yu, X., and Debevec, P. 2011. Multiview face capture using polarized spherical gradient illumination. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 30, 6, 129.

Digital Library

[21]

Guo, B., Liang, L., Liu, C., Shum, H.-Y., and Xu, Y. 2001. Real-time texture synthesis by patch-based sampling.

[22]

Haehnel, D., Thrun, S., and Burgard, W. 2003. An extension of the icp algorithm for modeling nonrigid objects with mobile robots. In IJCAI, vol. 3, 915--920.

Digital Library

[23]

Ikemoto, L., Gelfand, N., and Levoy, M. 2003. A hierarchical method for aligning warped meshes. In 3-D Digital Imaging and Modeling, 2003. 3DIM 2003. Proceedings. Fourth International Conference on, IEEE, 434--441.

[24]

Jacob, M., and Unser, M. 2004. Design of steerable filters for feature detection using canny-like criteria. Pattern Analysis and Machine Intelligence, IEEE Transactions on 26, 8, 1007--1019.

Digital Library

[25]

Kazhdan, M., Funkhouser, T., and Rusinkiewicz, S. 2004. Shape matching and anisotropy. In ACM Transactions on Graphics (TOG), vol. 23, ACM, 623--629.

Digital Library

[26]

Kwatra, V., Schödl, A., Essa, I., Turk, G., and Bobick, A. 2003. Graphcut textures: image and video synthesis using graph cuts. In ACM Transactions on Graphics (ToG), vol. 22, ACM, 277--286.

Digital Library

[27]

Lefohn, A., Budge, B., Shirley, P., Caruso, R., and Reinhard, E. 2003. An ocularist's approach to human iris synthesis. IEEE CG&A 23, 6, 70--75.

Digital Library

[28]

Li, H., Sumner, R. W., and Pauly, M. 2008. Global correspondence optimization for non-rigid registration of depth scans. In Computer graphics forum, vol. 27, Wiley Online Library, 1421--1430.

Digital Library

[29]

Li, J., Xu, W., Cheng, Z., Xu, K., and Klein, R. 2015. Lightweight wrinkle synthesis for 3d facial modeling and animation. Computer-Aided Design 58, 117--122.

Digital Library

[30]

Ma, W.-C., Hawkins, T., Peers, P., Chabert, C.-F., Weiss, M., and Debevec, P. 2007. Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. In Proc. Rendering Techniques, 183--194.

Digital Library

[31]

Mohammed, U., Prince, S. J., and Kautz, J. 2009. Visiolization: generating novel facial images. ACM Transactions on Graphics (TOG) 28, 3, 57.

Digital Library

[32]

Palubicki, W., Horel, K., Longay, S., Runions, A., Lane, B., Měch, R., and Prusinkiewicz, P. 2009. Self-organizing tree models for image synthesis. In ACM Transactions on Graphics (TOG), vol. 28, ACM, 58.

Digital Library

[33]

Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. In ACM Transactions on Graphics (TOG), vol. 22, ACM, 313--318.

Digital Library

[34]

Praun, E., Finkelstein, A., and Hoppe, H. 2000. Lapped textures. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, ACM Press/Addison-Wesley Publishing Co., 465--470.

Digital Library

[35]

Ramanarayanan, G., and Bala, K. 2007. Constrained texture synthesis via energy minimization. IEEE TVCG 13, 1.

Digital Library

[36]

Rozenberg, G., and Salomaa, A. 1976. The mathematical theory of L systems. Springer.

Digital Library

[37]

Ruhland, K., Andrist, S., Badler, J., Peters, C., Badler, N., Gleicher, M., Mutlu, B., and McDonnell, R. 2014. Look me in the eyes: A survey of eye and gaze animation for virtual agents and artificial systems. In Eurographics State of the Art Reports, 69--91.

[38]

Sagar, M. A., Bullivant, D., Mallinson, G. D., and Hunter, P. J. 1994. A virtual environment and model of the eye for surgical simulation. In Proceedings of Computer Graphics and Interactive Techniques, 205--212.

Digital Library

[39]

Shen, H.-L., and Cai, Q.-Y. 2009. Simple and efficient method for specularity removal in an image. Applied optics 48, 14, 2711--2719.

[40]

Sorkine, O., Cohen-Or, D., Lipman, Y., Alexa, M., Rössl, C., and Seidel, H.-P. 2004. Laplacian surface editing. In Proceedings of the 2004 Eurographics/ACM SIGGRAPH symposium on Geometry processing, ACM, 175--184.

Digital Library

[41]

Suwajanakorn, S., Kemelmacher-Shlizerman, I., and Seitz, S. M. 2014. Total moving face reconstruction. In Computer Vision--ECCV 2014. Springer, 796--812.

[42]

Tai, Y.-W., Liu, S., Brown, M. S., and Lin, S. 2010. Super resolution using edge prior and single image detail synthesis. In CVPR.

[43]

Valgaerts, L., Wu, C., Bruhn, A., Seidel, H.-P., and Theobalt, C. 2012. Lightweight binocular facial performance capture under uncontrolled lighting. ACM Trans. Graph. 31, 6, 187.

Digital Library

[44]

Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. In ACM Transactions on Graphics (TOG), vol. 24, ACM, 426--433.

Digital Library

[45]

Wei, L.-Y., Lefebvre, S., Kwatra, V., and Turk, G. 2009. State of the art in example-based texture synthesis. In Eurographics 2009, State of the Art Report, EG-STAR, Eurographics Association, 93--117.

[46]

Wikimedia Commons. 1485. Sandro Botticelli - The Birth of the Venus. Public Domain. https://commons.wikimedia.org/wiki/File:Venus_botticelli_detail.jpg.

[47]

Wikimedia Commons. 1887. Vincent Van Gogh - Self-Portrait. Public Domain. https://commons.wikimedia.org/wiki/File:VanGogh_1887_Selbstbildnis.jpg.

[48]

Wikimedia Commons. 2006. Blue Eye Image. GNU Free Documentation License Version 1.2. https://commons.wikimedia.org/wiki/File:Blueye.JPG.

Cited By

Liang SWang HLu F(2024)EyeIR: Single Eye Image Inverse Rendering In the WildACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657506(1-11)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657506
Li GSarkar KMeka ABuehler MMueller FGotardo PHilliges OBeeler T(2024)ShellNeRF: Learning a Controllable High‐resolution Model of the Eye and Periocular RegionComputer Graphics Forum10.1111/cgf.1504143:2Online publication date: 24-Apr-2024
https://doi.org/10.1111/cgf.15041
Saito SSchwartz GSimon TLi JNam G(2024)Relightable Gaussian Codec Avatars2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00021(130-141)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00021
Show More Cited By

Index Terms

Lightweight eye capture using a parametric model
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Reconstruction
  2. Computer graphics
    1. Shape modeling

Recommendations

High-quality capture of eyes

Even though the human eye is one of the central features of individual appearance, its shape has so far been mostly approximated in our community with gross simplifications. In this paper we demonstrate that there is a lot of individuality to every eye, ...
Dual-State Parametric Eye Tracking
FG '00: Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000

Most eye trackers work well for open eyes. However, blinking is a physiological necessity for humans. Moreover, for applications such as facial expression analysis and driver awareness systems, we need to do more than tracking the locations of the ...
Eye&Head: Synergetic Eye and Head Movement for Gaze Pointing and Selection
UIST '19: Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology

Eye gaze involves the coordination of eye and head movement to acquire gaze targets, but existing approaches to gaze pointing are based on eye-tracking in abstraction from head motion. We propose to leverage the synergetic movement of eye and head, and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 35, Issue 4

July 2016

1396 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2897824

Issue’s Table of Contents

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 July 2016

Published in TOG Volume 35, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

45
Total Citations
View Citations
634
Total Downloads

Downloads (Last 12 months)25
Downloads (Last 6 weeks)4

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liang SWang HLu F(2024)EyeIR: Single Eye Image Inverse Rendering In the WildACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657506(1-11)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657506
Li GSarkar KMeka ABuehler MMueller FGotardo PHilliges OBeeler T(2024)ShellNeRF: Learning a Controllable High‐resolution Model of the Eye and Periocular RegionComputer Graphics Forum10.1111/cgf.1504143:2Online publication date: 24-Apr-2024
https://doi.org/10.1111/cgf.15041
Saito SSchwartz GSimon TLi JNam G(2024)Relightable Gaussian Codec Avatars2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00021(130-141)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00021
Reinhard JUrban PBell SCarpenter DSagoo M(2024)Automatic data-driven design and 3D printing of custom ocular prosthesesNature Communications10.1038/s41467-024-45345-515:1Online publication date: 27-Feb-2024
https://doi.org/10.1038/s41467-024-45345-5
Li CJin LZheng YYu YHan X(2023)EMS: 3D Eyebrow Modeling from Single-View ImagesACM Transactions on Graphics10.1145/361832342:6(1-19)Online publication date: 5-Dec-2023
https://dl.acm.org/doi/10.1145/3618323
Zhang LZhao ZCong XZhang QGu SGao YZheng RYang WXu LYu J(2023)HACK: Learning a Parametric Head and Neck Model for High-fidelity AnimationACM Transactions on Graphics10.1145/359209342:4(1-20)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592093
Tretschk EKairanda NB R MDabral RKortylewski AEgger BHabermann MFua PTheobalt CGolyanik V(2023)State of the Art in Dense Monocular Non‐Rigid 3D ReconstructionComputer Graphics Forum10.1111/cgf.1477442:2(485-520)Online publication date: 23-May-2023
https://doi.org/10.1111/cgf.14774
Kerbiriou GAvril QDanieau FMarchal M(2023)Detailed Eye Region Capture and AnimationComputer Graphics Forum10.1111/cgf.1464241:8(279-282)Online publication date: 20-Mar-2023
https://doi.org/10.1111/cgf.14642
Qiu ZLi YHe DZhang QZhang LZhang YWang JXu LWang XZhang YYu J(2022)SCULPTORACM Transactions on Graphics10.1145/3550454.355546241:6(1-17)Online publication date: 30-Nov-2022
https://dl.acm.org/doi/10.1145/3550454.3555462
Li GMeka AMueller FBuehler MHilliges OBeeler T(2022)EyeNeRFACM Transactions on Graphics10.1145/3528223.353013041:4(1-16)Online publication date: 22-Jul-2022
https://dl.acm.org/doi/10.1145/3528223.3530130
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents