research-article

Image completion using planar structure guidance

Authors:

Sing Bing Kang,

Narendra Ahuja,

Johannes KopfAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 33, Issue 4

Article No.: 129, Pages 1 - 10

https://doi.org/10.1145/2601097.2601205

Published: 27 July 2014 Publication History

Abstract

We propose a method for automatically guiding patch-based image completion using mid-level structural cues. Our method first estimates planar projection parameters, softly segments the known region into planes, and discovers translational regularity within these planes. This information is then converted into soft constraints for the low-level completion algorithm by defining prior probabilities for patch offsets and transformations. Our method handles multiple planes, and in the absence of any detected planes falls back to a baseline fronto-parallel image completion algorithm. We validate our technique through extensive comparisons with state-of-the-art algorithms on a variety of scenes.

Supplementary Material

MP4 File (a129-sidebyside.mp4)

Download
20.44 MB

References

[1]

Aiger, D., Cohen-Or, D., and Mitra, N. J. 2012. Repetition maximization based texture rectification. Computer Graphics Forum (EUROGRAPHICS) 31, 2pt2, 439--448.

Digital Library

[2]

Ballester, C., Bertalmio, M., Caselles, V., Sapiro, G., and Verdera, J. 2001. Filling-in by joint interpolation of vector fields and gray levels. IEEE TIP 10, 8, 1200--1211.

Digital Library

[3]

Barinova, O., Konushin, V., Yakubenko, A., Lee, K., Lim, H., and Konushin, A. 2008. Fast automatic single-view 3-d reconstruction of urban scenes. In ECCV.

Digital Library

[4]

Barnes, C., Shechtman, E., Finkelstein, A., and Goldman, D. 2009. PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Trans. on Graphics (Proc. of Siggraph) 28, 3, 24.

Digital Library

[5]

Bertalmio, M., Sapiro, G., Caselles, V., and Ballester, C. 2000. Image inpainting. ACM Trans. on Graphics (Proc. of Siggraph) 19, 3, 417--424.

Digital Library

[6]

Bertalmio, M., Vese, L., Sapiro, G., and Osher, S. 2003. Simultaneous structure and texture image inpainting. IEEE TIP 12, 8, 882--889.

Digital Library

[7]

Chum, O., and Matas, J. 2010. Planar affine rectification from change of scale. In ACCV.

Digital Library

[8]

Comaniciu, D., and Meer, P. 2002. Mean Shift: A robust approach toward feature space analysis. IEEE TPAMI 24, 5, 603--619.

Digital Library

[9]

Criminisi, A., Pérez, P., and Toyama, K. 2004. Region filling and object removal by exemplar-based image inpainting. IEEE TIP 13, 9, 1200--1212.

Digital Library

[10]

Darabi, S., Shechtman, E., Barnes, C., Goldman, D. B., and Sen, P. 2012. Image Melding: Combining Inconsistent Images using Patch-based Synthesis. ACM Trans. on Graphics (Proc. of Siggraph) 31, 4.

Digital Library

[11]

Efros, A. A., and Freeman, W. T. 2001. Image quilting for texture synthesis and transfer. ACM Trans. on Graphics (Proc. of Siggraph) 20, 3, 341--346.

Digital Library

[12]

Efros, A. A., and Leung, T. K. 1999. Texture synthesis by non-parametric sampling. In ICCV.

Digital Library

[13]

Hartley, R. I., and Zisserman, A. 2004. Multiple View Geometry in Computer Vision, second ed. Cambridge University Press.

Digital Library

[14]

Hays, J., and Efros, A. A. 2007. Scene completion using millions of photographs. ACM Trans. on Graphics (Proc. of Siggraph) 26, 3, 4.

Digital Library

[15]

He, K., and Sun, J. 2012. Statistics of patch offsets for image completion. In ECCV.

Digital Library

[16]

Hertzmann, A., Jacobs, C. E., Oliver, N., Curless, B., and Salesin, D. H. 2001. Image analogies. ACM Trans. on Graphics (Proc. of Siggraph) 20, 3, 327--340.

Digital Library

[17]

Huang, H., K. Yin, Gong, M., Lischinski, D., Cohen-Or, D., Ascher, U., and Chen, B. 2013. Mind the gap: Tele-registration for structure-driven image completion. ACM Trans. on Graphics (Proc. of Siggraph Asia) 32, 174:1--174:10.

Digital Library

[18]

Huang, J.-B., Kopf, J., Ahuja, N., and Kang, S. B. 2013. Transformation guided image completion. In ICCP.

[19]

Jia, J., and Tang, C. 2003. Image repairing: Robust image synthesis by adaptive nd tensor voting. In CVPR.

Digital Library

[20]

Komodakis, N., and Tziritas, G. 2007. Image completion using efficient belief propagation via priority scheduling and dynamic pruning. IEEE TIP 16, 11, 2649--2661.

Digital Library

[21]

Kopf, J., Kienzle, W., Drucker, S., and Kang, S. B. 2012. Quality prediction for image completion. ACM Trans. on Graphics (Proc. of Siggraph Asia) 31, 6.

Digital Library

[22]

Kwatra, V., Essa, I., Bobick, A., and Kwatra, N. 2005. Texture optimization for example-based synthesis. ACM Trans. on Graphics (Proc. of Siggraph) 24, 3, 795--802.

Digital Library

[23]

Liu, Y., Lin, W.-C., and Hays, J. 2004. Near-regular texture analysis and manipulation. ACM Trans. on Graphics (Proc. of Siggraph) 23, 3, 368--376.

Digital Library

[24]

Liu, Y., Hel-Or, H., and Kaplan, C. 2010. Computational symmetry in computer vision and computer graphics. Now Publishers.

[25]

Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60, 2, 91--110.

Digital Library

[26]

Mansfield, A., Prasad, M., Rother, C., Sharp, T., Kohli, P., and Van Gool, L. 2011. Transforming image completion. In BMVC.

[27]

Pavić, D., Schönefeld, V., and Kobbelt, L. 2006. Interactive image completion with perspective correction. The Visual Computer 22, 9, 671--681.

Digital Library

[28]

Pritch, Y., Kav-Venaki, E., and Peleg, S. 2009. Shift-map image editing. In ICCV.

[29]

Sun, J., Yuan, L., Jia, J., and Shum, H. 2005. Image completion with structure propagation. ACM Trans. on Graphics (Proc. of Siggraph) 24, 3, 861--868.

Digital Library

[30]

Wexler, Y., Shechtman, E., and Irani, M. 2007. Space-time completion of video. IEEE TPAMI 29, 3, 463--476.

Digital Library

[31]

Whyte, O., Sivic, J., and Zisserman, A. 2009. Get out of my picture! internet-based inpainting. In BMVC.

[32]

Zhang, Z., Ganesh, A., Liang, X., and Ma, Y. 2012. TILT: transform invariant low-rank textures. International Journal of Computer Vision 99, 1, 1--24.

Digital Library

[33]

Zhang, Y., Xiao, J., Hays, J., and Tan, P. 2013. FrameBreak: Dramatic image extrapolation by guided shift-maps. In CVPR.

Digital Library

Cited By

Liu SWang HYan DLi QLuo FTeng ZLiu X(2025)Spectral Descriptors for 3D Deformable Shape Matching: A Comparative SurveyIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.336808331:3(1677-1697)Online publication date: 1-Mar-2025
https://dl.acm.org/doi/10.1109/TVCG.2024.3368083
Huang LHuang YGuan Q(2025)Improving Image Inpainting via Adversarial Collaborative TrainingIEEE Transactions on Multimedia10.1109/TMM.2024.352180027(356-370)Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1109/TMM.2024.3521800
Yao YHan TJia SLyu S(2025)Dense Feature Interaction Network for Image Inpainting LocalizationIEEE Transactions on Information Forensics and Security10.1109/TIFS.2025.353110320(1636-1648)Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1109/TIFS.2025.3531103
Show More Cited By

Index Terms

Image completion using planar structure guidance
1. Applied computing
  1. Document management and text processing
    1. Document capture
      1. Graphics recognition and interpretation
2. Computing methodologies
  1. Computer graphics

Recommendations

Image completion with variable scope patch sampling
ICAIR-CACRE '16: Proceedings of the International Conference on Artificial Intelligence and Robotics and the International Conference on Automation, Control and Robotics Engineering

This paper presents a simple but effective way of patch sampling for image completion which changes the assignment scope in different pixels when sampling patches during completion. In each pixel of missing regions, the scope expands from narrow to wide ...
Sample-based image completion using structure synthesis

Image completion technique is widely used in image processing applications such as textural recovery, object removal, image edit, etc. When filling in the missing areas of an image, it is often a challenge to keep local consistency of image structures ...
Image completion by structure reconstruction and texture synthesis

We present a new image completion method that can deal with large holes surrounded by different types of structure and texture. Our approach is based upon creating image structure in the hole while preserving global image structure, and then creating ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 33, Issue 4

July 2014

1366 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2601097

Issue’s Table of Contents

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 July 2014

Published in TOG Volume 33, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Office of Naval Research

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

245
Total Citations
View Citations
2,570
Total Downloads

Downloads (Last 12 months)61
Downloads (Last 6 weeks)4

Reflects downloads up to 19 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Liu SWang HYan DLi QLuo FTeng ZLiu X(2025)Spectral Descriptors for 3D Deformable Shape Matching: A Comparative SurveyIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.336808331:3(1677-1697)Online publication date: 1-Mar-2025
https://dl.acm.org/doi/10.1109/TVCG.2024.3368083
Huang LHuang YGuan Q(2025)Improving Image Inpainting via Adversarial Collaborative TrainingIEEE Transactions on Multimedia10.1109/TMM.2024.352180027(356-370)Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1109/TMM.2024.3521800
Yao YHan TJia SLyu S(2025)Dense Feature Interaction Network for Image Inpainting LocalizationIEEE Transactions on Information Forensics and Security10.1109/TIFS.2025.353110320(1636-1648)Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1109/TIFS.2025.3531103
Li RDai JLi XHan LWang C(2025)Deep Image Inpainting Incorporating Texture Prior Based on Gabor FilterData Information in Online Environments10.1007/978-3-031-80713-8_3(37-48)Online publication date: 9-Feb-2025
https://doi.org/10.1007/978-3-031-80713-8_3
Wei LTianqiang HLiqing HAokun ZChao X(2024)Large-scale datasets for facial tampering detection with inpainting techniquesJournal of Image and Graphics10.11834/jig.23042229:7(1834-1848)Online publication date: 2024
https://doi.org/10.11834/jig.230422
Le QBu JQu YZhu BDu T(2024)Computational Biomimetics of Winged SeedsACM Transactions on Graphics10.1145/368789943:6(1-13)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687899
Chen DLiao XWu XChen SCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)SafePaint: Anti-forensic Image Inpainting with Domain AdaptationProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681279(7774-7782)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681279
Lu YMa JYin YCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)View-consistent Object Removal in Radiance FieldsProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681090(3597-3606)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681090
Sattler FCarrillo Perez BStephan MBarnes S(2024)Automatic occlusion removal from 3D maps for maritime situational awarenessArtificial Intelligence and Image and Signal Processing for Remote Sensing XXX10.1117/12.3030924(29)Online publication date: 20-Nov-2024
https://doi.org/10.1117/12.3030924
Liu WLi ZShi CJia XLiu J(2024)Appearance flow based structure prior guided image inpaintingJournal of Electronic Imaging10.1117/1.JEI.33.5.05301133:05Online publication date: 1-Sep-2024
https://doi.org/10.1117/1.JEI.33.5.053011
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents