research-article

Personal photo enhancement using example images

Authors:

Wojciech Matusik,

Edward H. Adelson,

David J. KriegmanAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 29, Issue 2

Article No.: 12, Pages 1 - 15

https://doi.org/10.1145/1731047.1731050

Published: 21 April 2010 Publication History

Abstract

We describe a framework for improving the quality of personal photos by using a person's favorite photographs as examples. We observe that the majority of a person's photographs include the faces of a photographer's family and friends and often the errors in these photographs are the most disconcerting. We focus on correcting these types of images and use common faces across images to automatically perform both global and face-specific corrections. Our system achieves this by using face detection to align faces between “good” and “bad” photos such that properties of the good examples can be used to correct a bad photo. These “personal” photos provide strong guidance for a number of operations and, as a result, enable a number of high-quality image processing operations. We illustrate the power and generality of our approach by presenting a novel deblurring algorithm, and we show corrections that perform sharpening, superresolution, in-painting of over- and underexposured regions, and white-balancing.

Supplementary Material

JPG File (joshi.jpg)

Download
12.97 KB

joshi (joshi.avi)

1st supplemental movie file for Personal photo enhancement using example images

Download
14.75 MB

References

[1]

Agarwala, A., Dontcheva, M., Agarwala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. ACM Trans. Graph. 23, 3, 294--302.

Digital Library

[2]

Agrawal, A., Raskar, R., Nayar, S. K., and Li, Y. 2005. Removing photography artifacts using gradient projection and flash-exposure sampling. ACM Trans. Graph. 24, 3, 828--835.

Digital Library

[3]

Apostoloff, N. and Fitzgibbon, A. 2004. Bayesian video matting using learnt image priors. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR'04). Vol. 01. 407--414.

[4]

Bae, S., Paris, S., and Durand, F. 2006. Two-Scale tone management for photographic look. ACM Trans. Graph. 25, 3, 637--645.

Digital Library

[5]

Baker, S. and Kanade, T. 2000. Hallucinating faces. In Proceedings of the 4th International Conference on Automatic Face and Gesture Recognition.

Digital Library

[6]

Barrow, H. and Tenenbaum, J. 1978. Recovering intrinsic scene characteristics from images. Comput. Vision Syst., 3--26.

[7]

Bascle, B., Blake, A., and Zisserman, A. 1996. Motion deblurring and super-resolution from an image sequence. In Proceedings of the 4th European Conference on Computer Vision-Volume II (ECCV'96). Springer, 573--582.

Digital Library

[8]

Blanz, V. and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In Proceedings of SIGGRAPH 99. 187--194.

Digital Library

[9]

Eisemann, E. and Durand, F. 2004. Flash photography enhancement via intrinsic relighting. ACM Trans. Graph. 23, 3, 673--678.

Digital Library

[10]

Elad, M. and Aharon, M. 2006. Image denoising via learned dictionaries and sparse representation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06). IEEE Computer Society, 895--900.

Digital Library

[11]

Fergus, R., Singh, B., Hertzmann, A., Roweis, S. T., and Freeman, W. T. 2006. Removing camera shake from a single photograph. ACM Trans. Graph. 25, 787--794.

Digital Library

[12]

Finlayson, G. D., Drew, M. S., and Lu, C. 2004. Intrinsic images by entropy minimization. In Proceedings of the IEEE International Conference on Computer Vision (ECCV). 582--595.

[13]

Fitzgibbon, A., Wexler, Y., and Zisserman, A. 2005. Image-based rendering using image-based priors. Int. J. Comput. Vision 63, 2, 141--151.

Digital Library

[14]

Freeman, W. T., Jones, T. R., and Pasztor, E. C. 2002. Example-based super-resolution. IEEE Comput. Graph. Appl. 22, 2, 56--65.

Digital Library

[15]

Gross, R., Matthews, I., and Baker, S. 2005. Generic vs. person specific active appearance models. Image Vision Comput. 23, 11, 1080--1093.

Digital Library

[16]

Hays, J. and Efros, A. A. 2007. Scene completion using millions of photographs. In ACM SIGGRAPH Papers (SIGGRAPH'07). ACM, New York, 4.

Digital Library

[17]

Hertzmann, A., Jacobs, C. E., Oliver, N., Curless, B., and Salesin, D. H. 2001. Image analogies. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'01). ACM, New York, 327--340.

Digital Library

[18]

Land, E. H. and McCann, J. J. 1971. Lightness and retinex theory. J. Optical Soc. Amer. 61, 1--11.

[19]

Levin, A., Fergus, R., Durand, F., and Freeman, W. T. 2007. Image and depth from a conventional camera with a coded aperture. In ACM SIGGRAPH Papers (SIGGRAPH'07). ACM Press, New York, 70.

Digital Library

[20]

Levin, A., Lischinski, D., and Weiss, Y. 2006. A closed form solution to natural image matting. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06). IEEE Computer Society, 61--68.

Digital Library

[21]

Levin, A., Zomat, A., Peleg, S., and Weiss, Y. 2004. Seamless image stitching in the gradient domain. In IEEE International Conference on Computer Vision (ECCV'04).

[22]

Levin, A., Zomet, A., and Weiss, Y. 2003. Learning how to inpaint from global image statistics. In Proceedings of the 9th IEEE International Conference on Computer Vision (ICCV'03). 305.

Digital Library

[23]

Leyvand, T., Cohen-Or, D., Dror, G., and Lischinski, D. 2006. Digital face beautification. In ACM SIGGRAPH Sketches (SIGGRAPH'06). 169.

Digital Library

[24]

Liu, C., Freeman, W. T., Szeliski, R., and Kang, S. B. 2006. Noise estimation from a single image. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06). IEEE Computer Society, 901--908.

Digital Library

[25]

Liu, C., Shum, H.-Y., and Freeman, W. T. 2007. Face hallucination: Theory and practice. Int. J. Comput. Vision 75, 1, 115--134.

Digital Library

[26]

Liu, C., Shum, H.-Y., and Zhang, C. 2001. A two-step approach to hallucinating faces: Global parametric model and local nonparametric model. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR'01). 192--198.

[27]

Lucy, L. 1974. Bayesian-Based iterative method of image restoration. J. Astro. 79, 745--754.

[28]

Penev, P. S. and Sirovich, L. 2000. The global dimensionality of face space. In Proceedings of the 4th IEEE International Conference on Automatic Face and Gesture Recognition 2000 (FG'00). IEEE Computer Society, 264.

Digital Library

[29]

Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Trans. Graph. 22, 3, 313--318.

Digital Library

[30]

Petschnigg, G., Szeliski, R., Agarwala, M., Cohen, M., Hoppe, H., and Toyama, K. 2004. Digital photography with flash and no-flash image pairs. ACM Trans. Graph. 23, 3, 664--672.

Digital Library

[31]

Rav-Acha, A. and Peleg, S. 2005. Two motion-blurred images are better than one. Pattern Recogn. Lett. 26, 3, 311--317.

Digital Library

[32]

Reinhard, E., Ashikhmin, M., Gooch, B., and Shirley, P. 2001. Color transfer between images. IEEE Comput. Graph. Appl. 21, 5, 34--41.

Digital Library

[33]

Richardson, W. 1972. Bayesian-Based iterative method of image restoration. J. Optical Soc. Amer. A 62, 55--59.

[34]

Roth, S. and Black, M. J. 2005. Fields of experts: A framework for learning image priors. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR'05). 860--867.

Digital Library

[35]

Rother, C., Bordeaux, L., Hamadi, Y., and Blake, A. 2006. Autocollage. In ACM SIGGRAPH Papers (SIGGRAPH'06). ACM Press, New York, 847--852.

Digital Library

[36]

Tappen, M. F., Adelson, E. H., and Freeman, W. T. 2006. Estimating intrinsic component images using non-linear regression. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR'06). 1992--1999.

Digital Library

[37]

Turk, M. A. and Pentland, A. P. 1991. Face recognition using eigenfaces. In Proceedings of the Computer Vision and Pattern Recognition (CVPR'91). 586--591.

[38]

van de Weijer, J., Gevers, T., and Gijsenij, A. 2007. Edge-Based color constancy. In Proceedings of the IEEE Conference on IEEE Trans. Image Process. 16, 9, 2207--2214.

Digital Library

[39]

Viola, P. and Jones, M. 2001. Rapid object detection using a boosted cascade of simple features. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR'61). 609--615.

[40]

Weiss, Y. 2001. Deriving intrinsic images from image sequences. http://www.ai.mit.edu/courses/6.899/papers/13-02.PDF

[41]

Yuan, L., Sun, J., Quan, L., and Shum, H.-Y. 2007. Image deblurring with blurred/noisy image pairs. In ACM SIGGRAPH Papers (SIGGRAPH'07). ACM, New York, 1.

Digital Library

Cited By

Kim HLee K(2024)Learning Controllable ISP for Image EnhancementIEEE Transactions on Image Processing10.1109/TIP.2023.330581633(867-880)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIP.2023.3305816
Treccani C(2023)Noise-Canceling Technologies for Smartphone Cameras: Self-Image in the Age of AnxietyLeonardo10.1162/leon_a_0240656:4(383-387)Online publication date: 1-Aug-2023
https://doi.org/10.1162/leon_a_02406
Tedla SYang BBrown M(2023)Examining Autoexposure for Challenging Scenes2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.01202(13030-13039)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.01202
Show More Cited By

Index Terms

Personal photo enhancement using example images
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Computer graphics
    1. Image manipulation

Recommendations

Image dehazing via enhancement, restoration, and fusion: A survey
Abstract
Haze usually causes severe interference to image visibility. Such degradation on images troubles both human observers and computer vision systems. To seek high-quality images from degraded ones, a large number of image dehazing ...
Highlights
- Image dehazing methods via enhancement, restoration and fusion are surveyed.
- ...
Restoration of motion blurred images
SCCG '07: Proceedings of the 23rd Spring Conference on Computer Graphics

Degradation of digital images is often caused by camera movement when a long exposition time is set. The blurred images can be enhanced by either iterative or non-iterative methods. From the non-iterative methods, Wiener reconstruction is often used. ...
Blind and semi-blind deblurring of natural images

A method for blind image deblurring is presented. The method only makes weak assumptions about the blurring filter and is able to undo a wide variety of blurring degradations. To overcome the ill-posedness of the blind image deblurring problem, the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 29, Issue 2

March 2010

145 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/1731047

Issue’s Table of Contents

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 April 2010

Accepted: 01 November 2009

Received: 01 September 2008

Published in TOG Volume 29, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

77
Total Citations
View Citations
2,490
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)1

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kim HLee K(2024)Learning Controllable ISP for Image EnhancementIEEE Transactions on Image Processing10.1109/TIP.2023.330581633(867-880)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIP.2023.3305816
Treccani C(2023)Noise-Canceling Technologies for Smartphone Cameras: Self-Image in the Age of AnxietyLeonardo10.1162/leon_a_0240656:4(383-387)Online publication date: 1-Aug-2023
https://doi.org/10.1162/leon_a_02406
Tedla SYang BBrown M(2023)Examining Autoexposure for Challenging Scenes2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.01202(13030-13039)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.01202
Onzon EMannan FHeide F(2021)Neural Auto-Exposure for High-Dynamic Range Object Detection2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR46437.2021.00762(7706-7716)Online publication date: Jun-2021
https://doi.org/10.1109/CVPR46437.2021.00762
Anwar SHayder ZPorikli F(2021)Deblur and deep depth from single defocus imageMachine Vision and Applications10.1007/s00138-020-01162-632:1Online publication date: 7-Jan-2021
https://doi.org/10.1007/s00138-020-01162-6
Joshi N(2021)Defocus BlurComputer Vision10.1007/978-3-030-63416-2_511(277-279)Online publication date: 13-Oct-2021
https://doi.org/10.1007/978-3-030-63416-2_511
Tan ZChai MChen DLiao JChu QYuan LTulyakov SYu N(2020)MichiGANACM Transactions on Graphics10.1145/3386569.339248839:4(95:1-95:13)Online publication date: 12-Aug-2020
https://dl.acm.org/doi/10.1145/3386569.3392488
Yasarla RPerazzi FPatel V(2020)Deblurring Face Images Using Uncertainty Guided Multi-Stream Semantic NetworksIEEE Transactions on Image Processing10.1109/TIP.2020.299035429(6251-6263)Online publication date: 2020
https://doi.org/10.1109/TIP.2020.2990354
Zhang CWang JYen GZhao CSun QTang YQian FKurths J(2020)When Autonomous Systems Meet Accuracy and Transferability through AI: A SurveyPatterns10.1016/j.patter.2020.1000501:4(100050)Online publication date: Jul-2020
https://doi.org/10.1016/j.patter.2020.100050
Hu SH. Shum HLiang XB. Li FAslam N(2020)Facial Reshaping Operator for Controllable Face BeautificationExpert Systems with Applications10.1016/j.eswa.2020.114067(114067)Online publication date: Oct-2020
https://doi.org/10.1016/j.eswa.2020.114067
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents