research-article

Public Access

EyeOpener: Editing Eyes in the Wild

Authors:

Dimitris Samaras,

Sunil HadapAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 36, Issue 1

Article No.: 1, Pages 1 - 13

https://doi.org/10.1145/2926713

Published: 09 September 2016 Publication History

Abstract

Closed eyes and look-aways can ruin precious moments captured in photographs. In this article, we present a new framework for automatically editing eyes in photographs. We leverage a user’s personal photo collection to find a “good” set of reference eyes and transfer them onto a target image. Our example-based editing approach is robust and effective for realistic image editing. A fully automatic pipeline for realistic eye editing is challenging due to the unconstrained conditions under which the face appears in a typical photo collection. We use crowd-sourced human evaluations to understand the aspects of the target-reference image pair that will produce the most realistic results. We subsequently train a model that automatically selects the top-ranked reference candidate(s) by narrowing the gap in terms of pose, local contrast, lighting conditions, and even expressions. Finally, we develop a comprehensive pipeline of three-dimensional face estimation, image warping, relighting, image harmonization, automatic segmentation, and image compositing in order to achieve highly believable results. We evaluate the performance of our method via quantitative and crowd-sourced experiments.

References

[1]

A. Agarwala. 2007. Efficient gradient-domain compositing using quadtrees. ACM Trans. Graph. 26, 3, 94.

Digital Library

[2]

A. Agarwala, M. Dontcheva, M. Agrawala, S. Drucker, A. Colburn, B. Curless, D. Salesin, and M. Cohen. 2004. Interactive digital photomontage. ACM Trans. Graph. 23, 3, 294--302.

Digital Library

[3]

S. Bakhshi, D. A. Shamma, and E. Gilbert. 2014. Faces engage us: Photos with faces attract more likes and comments on instagram. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 965--974.

Digital Library

[4]

J. C. Bazin, D. Q. Pham, I. Kweon, and K. J. Yoon. 2009. Automatic closed eye correction. In 2009 16th IEEE International Conference on Image Processing (ICIP). IEEE, 2433--2436.

Digital Library

[5]

P. Bhat, C. L. Zitnick, M. Cohen, and B. Curless. 2010. Gradientshop: A gradient-domain optimization framework for image and video filtering. ACM Trans. Graph. 29, 2, 10.

Digital Library

[6]

D. Bitouk, N. Kumar, S. Dhillon, P. Belhumeur, and S. K. Nayar. 2008. Face swapping: Automatically replacing faces in photographs. ACM Trans. Graph. 27, 3, 39.

Digital Library

[7]

V. Blanz and T. Vetter. 1999. A morphable model for the synthesis of 3d faces. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniquesk. ACM Press/Addison-Wesley Publishing Co., 187--194.

Digital Library

[8]

L. Breiman. 2001. Random forests. Mach. Learn. 45, 1, 5--32.

Digital Library

[9]

T. Brox and J. Malik. 2011. Large displacement optical flow: Descriptor matching in variational motion estimation. IEEE Trans. Pattern Anal. Mach. Intell. 33, 3, 500--513.

Digital Library

[10]

X. Chen, M. Chen, X. Jin, and Q. Zhao. 2011. Face illumination transfer through edge-preserving filters. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on. IEEE, 281--287.

Digital Library

[11]

N. Dalal and B. Triggs. 2005. Histograms of oriented gradients for human detection. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on. Vol. 1. IEEE, 886--893.

Digital Library

[12]

K. Dale, K. Sunkavalli, M. K. Johnson, D. Vlasic, W. Matusik, and H. Pfister. 2011. Video face replacement. ACM Trans. Graph. 30, 6, 130.

Digital Library

[13]

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and Fei-L. Fei. 2009. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on. IEEE, 248--255.

[14]

Z. Farbman, G. Hoffer, Y. Lipman, Cohen-D. Or, and D. Lischinski. 2009. Coordinates for instant image cloning. In ACM Transaction on Graphics (TOG). 28, 67.

Digital Library

[15]

P. Garrido, L. Valgaerts, O. Rehmsen, T. Thormaehlen, P. Perez, and C. Theobalt. 2014. Automatic face reenactment. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on. IEEE, 4217--4224.

Digital Library

[16]

P. Garrido, L. Valgaerts, H. Sarmadi, I. Steiner, K. Varanasi, P. Perez, and C. Theobalt. 2015. Vdub: Modifying face video of actors for plausible visual alignment to a dubbed audio track. In Eurographics 2015.

Digital Library

[17]

P. Garrido, L. Valgaerts, C. Wu, and C. Theobalt. 2013. Reconstructing detailed dynamic face geometry from monocular video. In ACM Trans. Graph. (Proceedings of SIGGRAPH Asia 2013). Vol. 32. 158:1--158:10.

Digital Library

[18]

D. Guo and T. Sim. 2009. Digital face makeup by example. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on. IEEE, 73--79.

[19]

J. Hays and A. A. Efros, 2007. Scene completion using millions of photographs. ACM Transactions on Graphics (SIGGRAPH 2007) 26, 3.

Digital Library

[20]

K. He, J. Sun, and X. Tang. 2013. Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 35, 6, 1397--1409.

Digital Library

[21]

N. Joshi, W. Matusik, E. H. Adelson, and D. J. Kriegman. 2010. Personal photo enhancement using example images. ACM Trans. Graph. 29, 2, 12.

Digital Library

[22]

I. Kemelmacher-Shlizerman, S. Suwajanakorn, and S. M. Seitz. 2014. Illumination-aware age progression. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on. IEEE, 3334--3341.

Digital Library

[23]

M. H. Kiapour, K. Yamaguchi, A. C. Berg, and T. L. Berg. 2014. Hipster wars: Discovering elements of fashion styles. In Computer Vision--ECCV 2014. Springer, 472--488.

[24]

V. Kolmogorov and R. Zabin. 2004. What energy functions can be minimized via graph cuts? IEEE Trans. Pattern Anal. Mach. Intell. 26, 2, 147--159.

Digital Library

[25]

D. Kononenko and V. Lempitsky. 2015. Learning to look up: Realtime monocular gaze correction using machine learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4667--4675.

[26]

C. Kuster, T. Popa, J.-C. Bazin, C. Gotsman, and M. Gross. 2012. Gaze correction for home video conferencing. ACM Trans. Graph. 31, 6, 174.

Digital Library

[27]

P.-Y. Laffont, Z. Ren, X. Tao, C. Qian, and J. Hays. 2014. Transient attributes for high-level understanding and editing of outdoor scenes. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 33, 4.

Digital Library

[28]

T. Leyvand, D. Cohen-Or, G. Dror, and D. Lischinski. 2008. Data-driven enhancement of facial attractiveness. ACM Trans. Graph. 27, 3, 38.

Digital Library

[29]

C. Liu, J. Yuen, and A. Torralba. 2011. Sift flow: Dense correspondence across scenes and its applications. IEEE Trans. Pattern Anal. Mach. Intell. 33, 5, 978--994.

Digital Library

[30]

Z. Liu, Y. Shan, and Z. Zhang. 2001. Expressive expression mapping with ratio images. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. ACM, 271--276.

Digital Library

[31]

M. H. Nguyen, J.-F. Lalonde, A. Efros, and F. De la Torre. 2008. Image-based shaving. Comput. Graph. Forum 27, 2, 627--635.

[32]

P. O’Donovan, J. Lībeks, A. Agarwala, and A. Hertzmann. 2014. Exploratory font selection using crowdsourced attributes. ACM Trans. Graph. 33, 4, 92.

Digital Library

[33]

D. Parikh and K. Grauman. 2011a. Interactively building a discriminative vocabulary of nameable attributes. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on. IEEE, 1681--1688.

Digital Library

[34]

D. Parikh and K. Grauman. 2011b. Relative attributes. In Computer Vision (ICCV), 2011 IEEE International Conference on. IEEE, 503--510.

Digital Library

[35]

P. Pérez, M. Gangnet, and A. Blake. 2003. Poisson image editing. In ACM Transactions on Graphics (TOG). Vol. 22. ACM, 313--318.

Digital Library

[36]

F. Pitié, A. Kokaram, and R. Dahyot. 2005. N-dimensional probability density function transfer and its application to color transfer. In ICCV 2005. Vol. 2. 1434--1439 Vol. 2.

Digital Library

[37]

J. Saragih. 2011. Principal regression analysis. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on. IEEE, 2881--2888.

Digital Library

[38]

F. Shi, H.-T. Wu, X. Tong, and J. Chai. 2014. Automatic acquisition of high-fidelity facial performances using monocular videos. ACM Trans. Graph. 33, 6, 222.

Digital Library

[39]

J. Shi and J. Malik. 2000. Normalized cuts and image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 22, 8, 888--905.

Digital Library

[40]

Y. Shih, S. Paris, C. Barnes, W. T. Freeman, and F. Durand. 2014. Style transfer for headshot portraits. ACM Trans. Graph. 33, 4, 148.

Digital Library

[41]

A. J. Smola and B. Schölkopf. 2004. A tutorial on support vector regression. Stat. Comput. 14, 3, 199--222.

Digital Library

[42]

K. Sunkavalli, M. K. Johnson, W. Matusik, and H. Pfister. 2010. Multi-scale image harmonization. In ACM Trans. Graph. 29, 125.

Digital Library

[43]

S. Suwajanakorn, I. Kemelmacher-Shlizerman, and S. M. Seitz. 2014. Total moving face reconstruction. In Computer Vision--ECCV 2014. Springer, 796--812.

[44]

S. Suwajanakorn, S. M. Seitz, and I. Kemelmacher-Shlizerman. 2015. What makes tom hanks look like tom hanks. In Proceedings of the IEEE International Conference on Computer Vision. 3952--3960.

Digital Library

[45]

M. W. Tao, M. K. Johnson, and S. Paris. 2013. Error-tolerant image compositing. Int. J. Comput. Vis. 103, 2, 178--189.

Digital Library

[46]

A. Torralba, R. Fergus, and W. T. Freeman. 2008. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. 30, 11, 1958--1970.

Digital Library

[47]

Y. Wang, Z. Liu, G. Hua, Z. Wen, Z. Zhang, and D. Samaras. 2007. Face re-lighting from a single image under harsh lighting conditions. IEEE Conference on Computer Vision and Pattern Recognition, 2007. CVPR’07, 1--8.

[48]

Y. Wang, L. Zhang, Z. Liu, G. Hua, Z. Wen, Z. Zhang, and D. Samaras. 2009. Face relighting from a single image under arbitrary unknown lighting conditions. IEEE Trans. Pattern Anal. Mach. Intell. 31, 11 (Nov.), 1968--1984.

Digital Library

[49]

P. Welinder, S. Branson, P. Perona, and S. J. Belongie. 2010. The multidimensional wisdom of crowds. In Advances in Neural Information Processing Systems. 2424--2432.

Digital Library

[50]

Z. Wen, Z. Liu, and T. S. Huang. 2003. Face relighting with radiance environment maps. In Computer Vision and Pattern Recognition, 2003. Proceedings. 2003 IEEE Computer Society Conference on. Vol. 2. IEEE, II--158.

[51]

L. Wolf, Z. Freund, and S. Avidan. 2010. An eye for an eye: A single camera gaze-replacement method. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on. IEEE, 817--824.

[52]

F. Yang, J. Wang, E. Shechtman, L. Bourdev, and D. Metaxas. 2011. Expression flow for 3d-aware face component transfer. ACM Trans. Graph. 30, 4, 60.

Digital Library

[53]

R. Yang and Z. Zhang. 2002. Eye gaze correction with stereovision for video-teleconferencing. In Computer VisionECCV 2002. Springer, 479--494.

Digital Library

[54]

L. Zhang and D. Samaras. 2006. Face recognition from a single training image under arbitrary unknown lighting using spherical harmonics. IEEE Trans. Pattern Anal. Mach. Intell. 28, 3, 351--363.

Digital Library

[55]

J.-Y. Zhu, A. Agarwala, A. A. Efros, E. Shechtman, and J. Wang. 2014. Mirror mirror: Crowdsourcing better portraits. ACM Trans. Graph. 33, 6, 234.

Digital Library

Cited By

Wang YSong LWu WQian CHe RLoy C(2022)Talking Faces: Audio-to-Video Face GenerationHandbook of Digital Face Manipulation and Detection10.1007/978-3-030-87664-7_8(163-188)Online publication date: 31-Jan-2022
https://doi.org/10.1007/978-3-030-87664-7_8
MIYATA MAIZAWA K(2021)Estimation of Semantic Impressions from PortraitsIEICE Transactions on Information and Systems10.1587/transinf.2020EDP7140E104.D:6(863-872)Online publication date: 1-Jun-2021
https://doi.org/10.1587/transinf.2020EDP7140
Xiao QZhang HZhang ZWu YWang LJin XJiang XYang YShao TZhou K(2021)EyelashNetACM Transactions on Graphics10.1145/3478513.348054040:6(1-17)Online publication date: 10-Dec-2021
https://dl.acm.org/doi/10.1145/3478513.3480540
Show More Cited By

Index Terms

EyeOpener: Editing Eyes in the Wild
1. Applied computing
  1. Document management and text processing
    1. Document capture
      1. Graphics recognition and interpretation
2. Computing methodologies
  1. Computer graphics

Recommendations

Coarse-to-fine: facial structure editing of portrait images via latent space classifications

Facial structure editing of portrait images is challenging given the facial variety, the lack of ground-truth, the necessity of jointly adjusting color and shape, and the requirement of no visual artifacts. In this paper, we investigate how to perform ...
DeepFaceEditing: deep face generation and editing with disentangled geometry and appearance control

Recent facial image synthesis methods have been mainly based on conditional generative models. Sketch-based conditions can effectively describe the geometry of faces, including the contours of facial components, hair structures, as well as salient edges ...
SketchFaceNeRF: Sketch-based Facial Generation and Editing in Neural Radiance Fields

Realistic 3D facial generation based on Neural Radiance Fields (NeRFs) from 2D sketches benefits various applications. Despite the high realism of free-view rendering results of NeRFs, it is tedious and difficult for artists to achieve detailed 3D ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 36, Issue 1

February 2017

165 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2996392

Editor:
Kavita Bala
Cornell University

Issue’s Table of Contents

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 September 2016

Accepted: 01 April 2016

Revised: 01 April 2016

Received: 01 October 2015

Published in TOG Volume 36, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Adobe Systems
NSF
DIGITEO Institute France

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
948
Total Downloads

Downloads (Last 12 months)56
Downloads (Last 6 weeks)9

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang YSong LWu WQian CHe RLoy C(2022)Talking Faces: Audio-to-Video Face GenerationHandbook of Digital Face Manipulation and Detection10.1007/978-3-030-87664-7_8(163-188)Online publication date: 31-Jan-2022
https://doi.org/10.1007/978-3-030-87664-7_8
MIYATA MAIZAWA K(2021)Estimation of Semantic Impressions from PortraitsIEICE Transactions on Information and Systems10.1587/transinf.2020EDP7140E104.D:6(863-872)Online publication date: 1-Jun-2021
https://doi.org/10.1587/transinf.2020EDP7140
Xiao QZhang HZhang ZWu YWang LJin XJiang XYang YShao TZhou K(2021)EyelashNetACM Transactions on Graphics10.1145/3478513.348054040:6(1-17)Online publication date: 10-Dec-2021
https://dl.acm.org/doi/10.1145/3478513.3480540
Wang ZYu XLu MWang QQian CXu F(2020)Single image portrait relighting via explicit multiple reflectance channel modelingACM Transactions on Graphics10.1145/3414685.341782439:6(1-13)Online publication date: 27-Nov-2020
https://dl.acm.org/doi/10.1145/3414685.3417824
Fried OJacobs JFinkelstein AAgrawala M(2020)Editing self-imageCommunications of the ACM10.1145/332660163:3(70-79)Online publication date: 24-Feb-2020
https://dl.acm.org/doi/10.1145/3326601
Isikdogan LGerasimow TMichael G(2020)Eye Contact Correction using Deep Neural Networks2020 IEEE Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV45572.2020.9093554(3307-3315)Online publication date: Mar-2020
https://doi.org/10.1109/WACV45572.2020.9093554
Han XHou KDu DQiu YCui SZhou KYu Y(2020)CaricatureShop: Personalized and Photorealistic Caricature SketchingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2018.288600726:7(2349-2361)Online publication date: 1-Jul-2020
https://doi.org/10.1109/TVCG.2018.2886007
Hsu CWang YLei CChen K(2019)Look at Me! Correcting Eye Gaze in Live Video CommunicationACM Transactions on Multimedia Computing, Communications, and Applications10.1145/331178415:2(1-21)Online publication date: 5-Jun-2019
https://dl.acm.org/doi/10.1145/3311784
Gu SBao JYang HChen DWen FYuan L(2019)Mask-Guided Portrait Editing With Conditional GANs2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR.2019.00355(3431-3440)Online publication date: Jun-2019
https://doi.org/10.1109/CVPR.2019.00355
Wood EBaltrušaitis TMorency LRobinson PBulling A(2018)GazeDirector: Fully Articulated Eye Gaze Redirection in VideoComputer Graphics Forum10.1111/cgf.1335537:2(217-225)Online publication date: 22-May-2018
https://doi.org/10.1111/cgf.13355
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents