article

Open access

Scene completion using millions of photographs

Authors:

Alexei A. EfrosAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 26, Issue 3

Pages 4 - es

https://doi.org/10.1145/1276377.1276382

Published: 29 July 2007 Publication History

Abstract

What can you do with a million images? In this paper we present a new image completion algorithm powered by a huge database of photographs gathered from the Web. The algorithm patches up holes in images by finding similar image regions in the database that are not only seamless but also semantically valid. Our chief insight is that while the space of images is effectively infinite, the space of semantically differentiable scenes is actually not that large. For many image completion tasks we are able to find similar scenes which contain image fragments that will convincingly complete the image. Our algorithm is entirely data-driven, requiring no annotations or labelling by the user. Unlike existing image completion methods, our algorithm can generate a diverse set of results for each input image and we allow users to select among them. We demonstrate the superiority of our algorithm over existing image completion approaches.

Supplementary Material

MP4 File (pps004.mp4)

Download
67.39 MB

References

[1]

Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. ACM Trans. Graph. 23, 3, 294--302.

Digital Library

[2]

Agrawal, A., Raskar, R., and Chellappa, R. 2006. What is the range of surface reconstructions from a gradient field? In ECCV.

Digital Library

[3]

Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23, 11, 1222--1239.

Digital Library

[4]

Criminisi, A., Perez, P., and Toyama, K. 2003. Object removal by exemplar-based inpainting. CVPR 02, 721.

[5]

Diakopoulos, N., Essa, I., and Jain, R. 2004. Content based image synthesis. In Conference on Image and Video Retrieval.

[6]

Drori, I., Cohen-Or, D., and Yeshurun, H. 2003. Fragment-based image completion. ACM Trans. Graph. 22, 3, 303--312.

Digital Library

[7]

Efros, A. A., and Freeman, W. T. 2001. Image quilting for texture synthesis and transfer. Proceedings of SIGGRAPH 2001 (August), 341--346.

Digital Library

[8]

Efros, A. A., and Leung, T. K. 1999. Texture synthesis by non-parametric sampling. In ICCV, 1033--1038.

Digital Library

[9]

Irani, M., Anandan, P., and Hsu, S. 1995. Mosaic based representations of video sequences and their applications.

[10]

Jia, J., Sun, J., Tang, C.-K., and Shum, H.-Y. 2006. Drag-and-drop pasting. ACM Trans. Graph..

Digital Library

[11]

Johnson, M., Brostow, G. J., Shotton, J., Arandjelović, O., Kwatra, V., and Cipolla, R. 2006. Semantic photo synthesis. Computer Graphics Forum (Proc. Eurographics) 25, 3 (September), 407--413.

[12]

King, D. 1997. The Commissar Vanishes. Henry Holt and Co.

[13]

Komodakis, N. 2006. Image completion using global optimization. In CVPR, 442--452.

Digital Library

[14]

Kwatra, V., Schodl, A., Essa, I., Turk, G., and Bobick, A. 2003. Graphcut textures: Image and video synthesis using graph cuts. ACM Trans. Graph. 22, 3 (July), 277--286.

Digital Library

[15]

Kwatra, V., Essa, I., Bobick, A., and Kwatra, N. 2005. Texture optimization for example-based synthesis. In ACM Trans. Graph., 795--802.

Digital Library

[16]

Oliva, A., and Torralba, A. 2006. Building the gist of a scene: The role of global image features in recognition. In Visual Perception, Progress in Brain Research, vol. 155.

[17]

Perez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Trans. Graph. 22, 3, 313--318.

Digital Library

[18]

Russell, B. C., Torralba, A., Murphy, K. P., and Freeman, W. T. 2005. LabelMe: a database and web-based tool for image annotation. Tech. rep., MIT, 2005.

[19]

Snavely, N., Seitz, S. M., and Szeliski, R. 2006. Photo tourism: exploring photo collections in 3d. ACM Trans. Graph. 25, 3, 835--846.

Digital Library

[20]

Sun, J., Yuan, L., Jia, J., and Shum, H.-Y. 2005. Image completion with structure propagation. ACM Trans. Graph. 24, 3, 861--868.

Digital Library

[21]

Torralba, A., Murphy, K. P., Freeman, W. T., and Rubin, M. A. 2003. Context-based vision system for place and object recognition. In ICCV.

Digital Library

[22]

Torralba, A., Fergus, R., and Freeman, W. T. 2007. Tiny images. Tech. Rep. MIT-CSAIL-TR-2007-024.

[23]

Wertheimer, M. 1938. Laws of organization in perceptual forms (partial translation). In A sourcebook of Gestalt Psychology, W. Ellis, Ed. Harcourt Brace and Company, 71--88.

[24]

Wexler, Y., Shechtman, E., and Irani, M. 2004. Space-time video completion. CVPR 01, 120--127.

[25]

Wilczkowiak, M., Brostow, G. J., Tordoff, B., and Cipolla, R. 2005. Hole filling through photomontage. In BMVC, 492--501.

Cited By

Öztürk ÖBalaban Y(2024)ART IN PARTNERSHIP WITH HUMAN AND ARTIFICIAL INTELLIGENCE: CREATIVITY AND ALGORITHMSanat ve Tasarım Dergisi10.20488/sanattasarim.150608614:1(335-352)Online publication date: 28-Jun-2024
https://doi.org/10.20488/sanattasarim.1506086
Wan TFu CPeng JLu JLi PZhuo J(2024)Repairing the in situ hybridization missing data in the hippocampus region by using a 3D residual U-Net modelBiomedical Optics Express10.1364/BOE.52207815:6(3541)Online publication date: 1-May-2024
https://doi.org/10.1364/BOE.522078
Tang LRuiz NChu QLi YHolynski AJacobs DHariharan BPritch YWadhwa NAberman KRubinstein M(2024)RealFill: Reference-Driven Generation for Authentic Image CompletionACM Transactions on Graphics10.1145/365823743:4(1-12)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658237
Show More Cited By

Index Terms

Scene completion using millions of photographs
1. Applied computing
  1. Document management and text processing
    1. Document capture
      1. Graphics recognition and interpretation
2. Computing methodologies

Recommendations

Scene completion using millions of photographs

What can you do with a million images? In this paper, we present a new image completion algorithm powered by a huge database of photographs gathered from the Web. The algorithm patches up holes in images by finding similar image regions in the database ...
Scene Completion Using Millions of Photographs
Seminal Graphics Papers: Pushing the Boundaries, Volume 2

What can you do with a million images? In this paper we present a new image completion algorithm powered by a huge database of photographs gathered from the Web. The algorithm patches up holes in images by finding similar image regions in the database ...
Scene completion using millions of photographs
SIGGRAPH '07: ACM SIGGRAPH 2007 papers

What can you do with a million images? In this paper we present a new image completion algorithm powered by a huge database of photographs gathered from the Web. The algorithm patches up holes in images by finding similar image regions in the database ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 26, Issue 3

July 2007

976 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/1276377

Issue’s Table of Contents

Seminal Graphics Papers: Pushing the Boundaries, Volume 2
August 2023
893 pages
ISBN:9798400708978
DOI:10.1145/3596711
Editor:
Mary C. Whitton
Department of Computer Science, UNC Chapel Hill, USA

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 July 2007

Published in TOG Volume 26, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Seminal Paper

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

467
Total Citations
View Citations
4,799
Total Downloads

Downloads (Last 12 months)417
Downloads (Last 6 weeks)53

Reflects downloads up to 03 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Öztürk ÖBalaban Y(2024)ART IN PARTNERSHIP WITH HUMAN AND ARTIFICIAL INTELLIGENCE: CREATIVITY AND ALGORITHMSanat ve Tasarım Dergisi10.20488/sanattasarim.150608614:1(335-352)Online publication date: 28-Jun-2024
https://doi.org/10.20488/sanattasarim.1506086
Wan TFu CPeng JLu JLi PZhuo J(2024)Repairing the in situ hybridization missing data in the hippocampus region by using a 3D residual U-Net modelBiomedical Optics Express10.1364/BOE.52207815:6(3541)Online publication date: 1-May-2024
https://doi.org/10.1364/BOE.522078
Tang LRuiz NChu QLi YHolynski AJacobs DHariharan BPritch YWadhwa NAberman KRubinstein M(2024)RealFill: Reference-Driven Generation for Authentic Image CompletionACM Transactions on Graphics10.1145/365823743:4(1-12)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658237
Yu HYang RYu J(2024)Mutual encoder-decoder with bi-gated convolution for image inpaintingJournal of Electronic Imaging10.1117/1.JEI.33.1.01303633:01Online publication date: 1-Jan-2024
https://doi.org/10.1117/1.JEI.33.1.013036
Liu QJiang YTan ZChen DFu YChu QHua GYu N(2024)Transformer Based Pluralistic Image Completion With Reduced Information LossIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.338440646:10(6652-6668)Online publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1109/TPAMI.2024.3384406
Xiao JBi X(2024)Model-Guided Generative Adversarial Networks for Unsupervised Fine-Grained Image GenerationIEEE Transactions on Multimedia10.1109/TMM.2023.327775826(1188-1199)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2023.3277758
Sun ZLiu HLiang J(2024)Eyes Attribute Editing Assisted by Dual-Coordinate System and Multi-Probability Fusion Prediction2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10651383(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/IJCNN60899.2024.10651383
Shi Y(2024)Automatic Face Image Restoration Based on an Improved Generative Adversarial Network2024 IEEE 4th International Conference on Electronic Technology, Communication and Information (ICETCI)10.1109/ICETCI61221.2024.10594061(358-365)Online publication date: 24-May-2024
https://doi.org/10.1109/ICETCI61221.2024.10594061
Lee DHan D(2024)GAN-Based Two Stage Network for De-occlusion Face Image2024 IEEE International Conference on Consumer Electronics (ICCE)10.1109/ICCE59016.2024.10444472(1-4)Online publication date: 6-Jan-2024
https://doi.org/10.1109/ICCE59016.2024.10444472
Slavcheva MGausebeck DChen KBuchhofer DSabik AMa CDhillon SBrandt ODolhasz A(2024)An Empty Room is All We Want: Automatic Defurnishing of Indoor Panoramas2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00734(7384-7394)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00734
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents