research-article

Image completion using planar structure guidance

Authors:

Sing Bing Kang,

Narendra Ahuja,

Johannes KopfAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 33, Issue 4

Article No.: 129, Pages 1 - 10

https://doi.org/10.1145/2601097.2601205

Published: 27 July 2014 Publication History

Abstract

We propose a method for automatically guiding patch-based image completion using mid-level structural cues. Our method first estimates planar projection parameters, softly segments the known region into planes, and discovers translational regularity within these planes. This information is then converted into soft constraints for the low-level completion algorithm by defining prior probabilities for patch offsets and transformations. Our method handles multiple planes, and in the absence of any detected planes falls back to a baseline fronto-parallel image completion algorithm. We validate our technique through extensive comparisons with state-of-the-art algorithms on a variety of scenes.

Supplementary Material

MP4 File (a129-sidebyside.mp4)

Download
20.44 MB

References

[1]

Aiger, D., Cohen-Or, D., and Mitra, N. J. 2012. Repetition maximization based texture rectification. Computer Graphics Forum (EUROGRAPHICS) 31, 2pt2, 439--448.

Digital Library

[2]

Ballester, C., Bertalmio, M., Caselles, V., Sapiro, G., and Verdera, J. 2001. Filling-in by joint interpolation of vector fields and gray levels. IEEE TIP 10, 8, 1200--1211.

Digital Library

[3]

Barinova, O., Konushin, V., Yakubenko, A., Lee, K., Lim, H., and Konushin, A. 2008. Fast automatic single-view 3-d reconstruction of urban scenes. In ECCV.

Digital Library

[4]

Barnes, C., Shechtman, E., Finkelstein, A., and Goldman, D. 2009. PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Trans. on Graphics (Proc. of Siggraph) 28, 3, 24.

Digital Library

[5]

Bertalmio, M., Sapiro, G., Caselles, V., and Ballester, C. 2000. Image inpainting. ACM Trans. on Graphics (Proc. of Siggraph) 19, 3, 417--424.

Digital Library

[6]

Bertalmio, M., Vese, L., Sapiro, G., and Osher, S. 2003. Simultaneous structure and texture image inpainting. IEEE TIP 12, 8, 882--889.

Digital Library

[7]

Chum, O., and Matas, J. 2010. Planar affine rectification from change of scale. In ACCV.

Digital Library

[8]

Comaniciu, D., and Meer, P. 2002. Mean Shift: A robust approach toward feature space analysis. IEEE TPAMI 24, 5, 603--619.

Digital Library

[9]

Criminisi, A., Pérez, P., and Toyama, K. 2004. Region filling and object removal by exemplar-based image inpainting. IEEE TIP 13, 9, 1200--1212.

Digital Library

[10]

Darabi, S., Shechtman, E., Barnes, C., Goldman, D. B., and Sen, P. 2012. Image Melding: Combining Inconsistent Images using Patch-based Synthesis. ACM Trans. on Graphics (Proc. of Siggraph) 31, 4.

Digital Library

[11]

Efros, A. A., and Freeman, W. T. 2001. Image quilting for texture synthesis and transfer. ACM Trans. on Graphics (Proc. of Siggraph) 20, 3, 341--346.

Digital Library

[12]

Efros, A. A., and Leung, T. K. 1999. Texture synthesis by non-parametric sampling. In ICCV.

Digital Library

[13]

Hartley, R. I., and Zisserman, A. 2004. Multiple View Geometry in Computer Vision, second ed. Cambridge University Press.

Digital Library

[14]

Hays, J., and Efros, A. A. 2007. Scene completion using millions of photographs. ACM Trans. on Graphics (Proc. of Siggraph) 26, 3, 4.

Digital Library

[15]

He, K., and Sun, J. 2012. Statistics of patch offsets for image completion. In ECCV.

Digital Library

[16]

Hertzmann, A., Jacobs, C. E., Oliver, N., Curless, B., and Salesin, D. H. 2001. Image analogies. ACM Trans. on Graphics (Proc. of Siggraph) 20, 3, 327--340.

Digital Library

[17]

Huang, H., K. Yin, Gong, M., Lischinski, D., Cohen-Or, D., Ascher, U., and Chen, B. 2013. Mind the gap: Tele-registration for structure-driven image completion. ACM Trans. on Graphics (Proc. of Siggraph Asia) 32, 174:1--174:10.

Digital Library

[18]

Huang, J.-B., Kopf, J., Ahuja, N., and Kang, S. B. 2013. Transformation guided image completion. In ICCP.

[19]

Jia, J., and Tang, C. 2003. Image repairing: Robust image synthesis by adaptive nd tensor voting. In CVPR.

Digital Library

[20]

Komodakis, N., and Tziritas, G. 2007. Image completion using efficient belief propagation via priority scheduling and dynamic pruning. IEEE TIP 16, 11, 2649--2661.

Digital Library

[21]

Kopf, J., Kienzle, W., Drucker, S., and Kang, S. B. 2012. Quality prediction for image completion. ACM Trans. on Graphics (Proc. of Siggraph Asia) 31, 6.

Digital Library

[22]

Kwatra, V., Essa, I., Bobick, A., and Kwatra, N. 2005. Texture optimization for example-based synthesis. ACM Trans. on Graphics (Proc. of Siggraph) 24, 3, 795--802.

Digital Library

[23]

Liu, Y., Lin, W.-C., and Hays, J. 2004. Near-regular texture analysis and manipulation. ACM Trans. on Graphics (Proc. of Siggraph) 23, 3, 368--376.

Digital Library

[24]

Liu, Y., Hel-Or, H., and Kaplan, C. 2010. Computational symmetry in computer vision and computer graphics. Now Publishers.

[25]

Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60, 2, 91--110.

Digital Library

[26]

Mansfield, A., Prasad, M., Rother, C., Sharp, T., Kohli, P., and Van Gool, L. 2011. Transforming image completion. In BMVC.

[27]

Pavić, D., Schönefeld, V., and Kobbelt, L. 2006. Interactive image completion with perspective correction. The Visual Computer 22, 9, 671--681.

Digital Library

[28]

Pritch, Y., Kav-Venaki, E., and Peleg, S. 2009. Shift-map image editing. In ICCV.

[29]

Sun, J., Yuan, L., Jia, J., and Shum, H. 2005. Image completion with structure propagation. ACM Trans. on Graphics (Proc. of Siggraph) 24, 3, 861--868.

Digital Library

[30]

Wexler, Y., Shechtman, E., and Irani, M. 2007. Space-time completion of video. IEEE TPAMI 29, 3, 463--476.

Digital Library

[31]

Whyte, O., Sivic, J., and Zisserman, A. 2009. Get out of my picture! internet-based inpainting. In BMVC.

[32]

Zhang, Z., Ganesh, A., Liang, X., and Ma, Y. 2012. TILT: transform invariant low-rank textures. International Journal of Computer Vision 99, 1, 1--24.

Digital Library

[33]

Zhang, Y., Xiao, J., Hays, J., and Tan, P. 2013. FrameBreak: Dramatic image extrapolation by guided shift-maps. In CVPR.

Digital Library

Cited By

Wei LTianqiang HLiqing HAokun ZChao X(2024)Large-scale datasets for facial tampering detection with inpainting techniquesJournal of Image and Graphics10.11834/jig.23042229:7(1834-1848)Online publication date: 2024
https://doi.org/10.11834/jig.230422
Chen DLiao XWu XChen SCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)SafePaint: Anti-forensic Image Inpainting with Domain AdaptationProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681279(7774-7782)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681279
Lu YMa JYin YCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)View-consistent Object Removal in Radiance FieldsProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681090(3597-3606)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681090
Show More Cited By

Index Terms

Image completion using planar structure guidance
1. Applied computing
  1. Document management and text processing
    1. Document capture
      1. Graphics recognition and interpretation
2. Computing methodologies
  1. Computer graphics

Recommendations

Image completion with variable scope patch sampling
ICAIR-CACRE '16: Proceedings of the International Conference on Artificial Intelligence and Robotics and the International Conference on Automation, Control and Robotics Engineering

This paper presents a simple but effective way of patch sampling for image completion which changes the assignment scope in different pixels when sampling patches during completion. In each pixel of missing regions, the scope expands from narrow to wide ...
Sample-based image completion using structure synthesis

Image completion technique is widely used in image processing applications such as textural recovery, object removal, image edit, etc. When filling in the missing areas of an image, it is often a challenge to keep local consistency of image structures ...
Image completion by structure reconstruction and texture synthesis

We present a new image completion method that can deal with large holes surrounded by different types of structure and texture. Our approach is based upon creating image structure in the hole while preserving global image structure, and then creating ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 33, Issue 4

July 2014

1366 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2601097

Issue’s Table of Contents

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 July 2014

Published in TOG Volume 33, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Office of Naval Research

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

237
Total Citations
View Citations
2,557
Total Downloads

Downloads (Last 12 months)68
Downloads (Last 6 weeks)15

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wei LTianqiang HLiqing HAokun ZChao X(2024)Large-scale datasets for facial tampering detection with inpainting techniquesJournal of Image and Graphics10.11834/jig.23042229:7(1834-1848)Online publication date: 2024
https://doi.org/10.11834/jig.230422
Chen DLiao XWu XChen SCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)SafePaint: Anti-forensic Image Inpainting with Domain AdaptationProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681279(7774-7782)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681279
Lu YMa JYin YCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)View-consistent Object Removal in Radiance FieldsProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681090(3597-3606)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681090
Liu WLi ZShi CJia XLiu J(2024)Appearance flow based structure prior guided image inpaintingJournal of Electronic Imaging10.1117/1.JEI.33.5.05301133:05Online publication date: 1-Sep-2024
https://doi.org/10.1117/1.JEI.33.5.053011
Yu HYang RYu J(2024)Mutual encoder-decoder with bi-gated convolution for image inpaintingJournal of Electronic Imaging10.1117/1.JEI.33.1.01303633:01Online publication date: 1-Jan-2024
https://doi.org/10.1117/1.JEI.33.1.013036
Shang QHu HYu HXu BWang LZhu Q(2024)Semantic Image Translation for Repairing the Texture Defects of Building ModelsIEEE Transactions on Geoscience and Remote Sensing10.1109/TGRS.2023.333896262(1-20)Online publication date: 2024
https://doi.org/10.1109/TGRS.2023.3338962
Zhan DWu JLuo XJin Z(2024)Learning From Text: A Multimodal Face Inpainting Network for Irregular HolesIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.337057834:8(7484-7497)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3370578
Li YHu LDong LWu HTian JZhou JLi X(2024)Transformer-Based Image Inpainting Detection via Label Decoupling and Constrained Adversarial TrainingIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2023.329927834:3(1857-1872)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1109/TCSVT.2023.3299278
Park JHong CBaik SLee K(2024)CoLaNet: Adaptive Context and Latent Information Blending for Face Image InpaintingIEEE Signal Processing Letters10.1109/LSP.2023.334099831(91-95)Online publication date: 2024
https://doi.org/10.1109/LSP.2023.3340998
Li CZhang ZLi PHe Z(2024)I3FDM: IRIS Inpainting Via Inverse Fusion of Diffusion ModelsICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10447757(1636-1640)Online publication date: 14-Apr-2024
https://doi.org/10.1109/ICASSP48485.2024.10447757
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents