research-article

GCM-Net: Towards Effective Global Context Modeling for Image Inpainting

Authors:

Meng WangAuthors Info & Claims

MM '21: Proceedings of the 29th ACM International Conference on Multimedia

Pages 2586 - 2594

https://doi.org/10.1145/3474085.3475433

Published: 17 October 2021 Publication History

Abstract

Deep learning based inpainting methods have obtained promising performance for image restoration, however current image inpainting methods still tend to produce unreasonable structures and blurry textures when processing the damaged images with heavy corruptions. In this paper, we propose a new image inpainting method termed Global Context Modeling Network (GCM-Net). By capturing the global contextual information, GCM-Net can potentially improve the performance of recovering the missing region in the damaged images with irregular masks. To be specific, we first use four convolution layers to extract the shadow features. Then, we design a progressive multi-scale fusion block termed PMSFB to extract and fuse the multi-scale features for obtaining local features. Besides, a dense context extraction (DCE) module is also designed to aggregate the local features extracted by PMSFBs. To improve the information flow, a channel attention guided residual learning module is deployed in both the DCE and PMSFB, which can reweight the learned residual features and refine the extracted information. To capture more global contextual information and enhance the representation ability, a coordinate context attention (CCA) based module is also presented. Finally, the extracted features with rich information are decoded as the image inpainting result. Extensive results on the Paris Street View, Places2 and CelebA-HQ datasets demonstrate that our method can better recover the structures and textures, and deliver significant improvements, compared with some related inpainting methods.

References

[1]

Naoufal Amrani, Joan Serra-Sagristà, Pascal Peter, and Joachim Weickert. 2017. Diffusion-based inpainting for coding remote-sensing data. IEEE Geoscience and Remote Sensing Letters 14, 8 (2017), 1203--1207.

[2]

Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28, 3 (2009), 24.

Digital Library

[3]

Ugo V Boscain, Roman Chertovskih, Jean-Paul Gauthier, Dario Prandi, and Alexey Remizov. 2018. Highly corrupted image inpainting through hypoelliptic diffusion. Journal of Mathematical Imaging and Vision 60, 8 (2018), 1231--1245.

Digital Library

[4]

Pierre Buyssens, Olivier Le Meur, Maxime Daisy, David Tschumperlé, and Olivier Lézoray. 2016. Depth-guided disocclusion inpainting of synthesized RGB-D images. IEEE Transactions on Image Processing 26, 2 (2016), 525--538.

Digital Library

[5]

Tony F Chan and Jianhong Shen. 2001. Nontexture inpainting by curvaturedriven diffusions. Journal of visual communication and image representation 12, 4 (2001), 436--449.

Digital Library

[6]

Antonio Criminisi, Patrick Pérez, and Kentaro Toyama. 2004. Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on image processing 13, 9 (2004), 1200--1212.

Digital Library

[7]

Carl Doersch, Saurabh Singh, Abhinav Gupta, Josef Sivic, and Alexei Efros. 2012. What makes paris look like paris? ACM Transactions on Graphics 31, 4 (2012).

Digital Library

[8]

Xinjian Gao, Zhao Zhang, Tingting Mu, Xudong Zhang, Chaoran Cui, and Meng Wang. 2020. Self-attention driven adversarial similarity learning network. Pattern Recognition 105 (2020), 107331.

[9]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020. Generative adversarial networks. Commun. ACM 63, 11 (2020), 139--144.

Digital Library

[10]

Zongyu Guo, Zhibo Chen, Tao Yu, Jiale Chen, and Sen Liu. 2019. Progressive image inpainting with full-resolution residual network. In Proceedings of the 27th ACM International Conference on Multimedia. 2496--2504.

Digital Library

[11]

Kaiming He and Jian Sun. 2012. Statistics of patch offsets for image completion. In European conference on computer vision. Springer, 16--29.

[12]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.

[13]

Xin Hong, Pengfei Xiong, Renhe Ji, and Haoqiang Fan. 2019. Deep fusion network for image completion. In Proceedings of the 27th ACM International Conference on Multimedia. 2033--2042.

Digital Library

[14]

Yibing Song Wei Huang Hongyu Liu, Bin Jiang and Chao Yang. 2020. Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations. In Proceedings of the European Conference on Computer Vision.

[15]

Alain Hore and Djemel Ziou. 2010. Image quality metrics: PSNR vs. SSIM. In 2010 20th international conference on pattern recognition. IEEE, 2366--2369.

Digital Library

[16]

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4700--4708.

[17]

Xun Huang and Serge Belongie. 2017. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision. 1501--1510.

[18]

Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2017. Globally and locally consistent image completion. ACM Transactions on Graphics (ToG) 36, 4 (2017), 1--14.

Digital Library

[19]

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).

[20]

Jingyuan Li, Fengxiang He, Lefei Zhang, Bo Du, and Dacheng Tao. 2019. Progressive reconstruction of visual structure for image inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 5962--5971.

[21]

Jingyuan Li, Ning Wang, Lefei Zhang, Bo Du, and Dacheng Tao. 2020. Recurrent feature reasoning for image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7760--7768.

[22]

Guilin Liu, FitsumAReda, Kevin J Shih, Ting-ChunWang, AndrewTao, and Bryan Catanzaro. 2018. Image inpainting for irregular holes using partial convolutions. In Proceedings of the European Conference on Computer Vision (ECCV). 85--100.

Digital Library

[23]

Hongyu Liu, Bin Jiang, Yi Xiao, and Chao Yang. 2019. Coherent semantic attention for image inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 4170--4179.

[24]

Or Lotan and Michal Irani. 2016. Needle-match: Reliable patch matching under high uncertainty. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 439--448.

[25]

Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Z Qureshi, and Mehran Ebrahimi. 2019. Edgeconnect: Generative image inpainting with adversarial edge learning. arXiv preprint arXiv:1901.00212 (2019).

[26]

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. (2017).

[27]

Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. 2016. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2536--2544.

[28]

Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H Li, Shan Liu, and Ge Li. 2019. Structureflow: Image inpainting via structure-aware appearance flow. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 181--190.

[29]

Jianhong Shen and Tony F Chan. 2002. Mathematical models for local nontexture inpaintings. SIAM J. Appl. Math. 62, 3 (2002), 1019--1043.

Digital Library

[30]

Ling Shen, Richang Hong, Haoran Zhang, Hanwang Zhang, and Meng Wang. 2019. Single-shot semantic image inpainting with densely connected generative networks. In Proceedings of the 27th ACM International Conference on Multimedia. 1861--1869.

Digital Library

[31]

Yong-Goo Shin, Min-Cheol Sagong, Yoon-Jae Yeo, Seung-Wook Kim, and Sung- Jea Ko. 2020. Pepsi++: Fast and lightweight network for image inpainting. IEEE transactions on neural networks and learning systems 32, 1 (2020), 252--265.

[32]

Linsen Song, Jie Cao, Lingxiao Song, Yibo Hu, and Ran He. 2019. Geometry-aware face completion and editing. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 2506--2513.

Digital Library

[33]

Ning Wang, Jingyuan Li, Lefei Zhang, and Bo Du. 2019. MUSICAL: Multi-Scale Image Contextual Attention Learning for Inpainting. In IJCAI. 3748--3754.

Digital Library

[34]

Yanyan Wei, Zhao Zhang, Yang Wang, Mingliang Xu, Yi Yang, Shuicheng Yan, and Meng Wang. 2021. DerainCycleGAN: Rain attentive CycleGAN for single image deraining and rainmaking. IEEE Transactions on Image Processing 30 (2021), 4788--4801.

[35]

Yanyan Wei, Zhao Zhang, Haijun Zhang, Richang Hong, and Meng Wang. 2019. A Coarse-to-Fine Multi-stream Hybrid Deraining Network for Single Image Deraining. In 2019 IEEE International Conference on Data Mining (ICDM). IEEE, 628--637.

[36]

Chaohao Xie, Shaohui Liu, Chao Li, Ming-Ming Cheng, Wangmeng Zuo, Xiao Liu, Shilei Wen, and Errui Ding. 2019. Image inpainting with learnable bidirectional attention maps. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 8858--8867.

[37]

Wei Xiong, Jiahui Yu, Zhe Lin, Jimei Yang, Xin Lu, Connelly Barnes, and Jiebo Luo. 2019. Foreground-aware image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5840--5848.

[38]

Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. 2015. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853 (2015).

[39]

Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li. 2017. High-resolution image inpainting using multi-scale neural patch synthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition. 6721--6729.

[40]

Raymond A Yeh, Chen Chen, Teck Yian Lim, Alexander G Schwing, Mark Hasegawa-Johnson, and Minh N Do. 2017. Semantic image inpainting with deep generative models. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5485--5493.

[41]

Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. 2018. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5505--5514.

[42]

Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. 2019. Free-form image inpainting with gated convolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 4471--4480.

[43]

Tao Yu, Zongyu Guo, Xin Jin, ShilinWu, Zhibo Chen,Weiping Li, Zhizheng Zhang, and Sen Liu. 2020. Region normalization for image inpainting. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 12733--12740.

[44]

Yanhong Zeng, Jianlong Fu, Hongyang Chao, and Baining Guo. 2019. Learning pyramid-context encoder network for high-quality image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1486--1494.

[45]

Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. 2019. Selfattention generative adversarial networks. In International conference on machine learning. PMLR, 7354--7363.

[46]

Yulun Zhang, Kunpeng Li, Kai Li, LichenWang, Bineng Zhong, and Yun Fu. 2018. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European conference on computer vision (ECCV). 286--301.

Digital Library

[47]

Zhao Zhang, Zemin Tang, Yang Wang, Zheng Zhang, Choujun Zhan, Zhengjun Zha, and Meng Wang. 2021. Dense Residual Network: Enhancing global dense feature flow for character recognition. Neural Networks 139 (2021), 77--85.

[48]

Zhao Zhang, Yanyan Wei, Haijun Zhang, Yi Yang, Shuicheng Yan, and Meng Wang. 2021. Data-Driven Single Image Deraining: A Comprehensive Review and New Perspectives. TechRxiv preprint (2021).

[49]

Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. 2019. Pluralistic image completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1438--1447.

[50]

Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence 40, 6 (2017), 1452--1464.

[51]

Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision. 2223--2232.

Cited By

Zhang ZZheng HHong RFan JYang YYan S(2024)FRC-Net: A Simple Yet Effective Architecture for Low-Light Image EnhancementIEEE Transactions on Consumer Electronics10.1109/TCE.2023.328046770:1(3332-3340)Online publication date: Feb-2024
https://doi.org/10.1109/TCE.2023.3280467
Ma XZhou XHuang HJia GWang YChen XChen C(2024)Uncertainty-aware image inpainting with adaptive feedback network▪Expert Systems with Applications: An International Journal10.1016/j.eswa.2023.121148235:COnline publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.eswa.2023.121148
Quan WChen JLiu YYan DWonka P(2024)Deep Learning-Based Image and Video Inpainting: A SurveyInternational Journal of Computer Vision10.1007/s11263-023-01977-6132:7(2367-2400)Online publication date: 19-Jan-2024
https://dl.acm.org/doi/10.1007/s11263-023-01977-6
Show More Cited By

Index Terms

GCM-Net: Towards Effective Global Context Modeling for Image Inpainting
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision representations
        Hierarchical representations
        Image representations

Recommendations

Image Retrieval Using Digital Image Inpainting Techniques

Image retrieval is an inverse problem in digital image processing. In this paper, the authors deal with restoration of image using digitally image inpainting methods. In this inpainting technique, one can extract a missing an important part or can ...
An efficient framework for image/video inpainting

Image inpainting has been widely applied to many applications, such as restoring corrupted old photos, erasing video logos, concealing errors in a digital video processing system, and so on. However, traditional geometric inpainting methods suffer low ...
A new structure tensor based image inpainting algorithm

A new structure tensor based image inpainting algorithm STIA is proposed for solving the deficiencies of the classical Criminisi method, such as the error repair accumulation, high time complexity caused by the unreasonable design of the patch priority, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '21: Proceedings of the 29th ACM International Conference on Multimedia

October 2021

5796 pages

ISBN:9781450386517

DOI:10.1145/3474085

General Chairs:
Heng Tao Shen
University of Electronic Science&Technology of China, China
,
Yueting Zhuang
Zhejiang University, China
,
John R. Smith
IBM, USA
,
Program Chairs:
Yang Yang
University of Electronic Science and Technology of China, China
,
Pablo Cesar
CWI&TU Delft, The Netherlands
,
Florian Metze
FACEBOOK, Inc., USA
,
Balakrishnan Prabhakaran
University of Texas at Dallas, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China

Conference

MM '21

Sponsor:

SIGMM

MM '21: ACM Multimedia Conference

October 20 - 24, 2021

Virtual Event, China

Acceptance Rates

Overall Acceptance Rate 995 of 4,171 submissions, 24%

Upcoming Conference

MM '24

Sponsor:
sigmm

The 32nd ACM International Conference on Multimedia

October 28 - November 1, 2024

Melbourne , VIC , Australia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
305
Total Downloads

Downloads (Last 12 months)37
Downloads (Last 6 weeks)4

Reflects downloads up to 01 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang ZZheng HHong RFan JYang YYan S(2024)FRC-Net: A Simple Yet Effective Architecture for Low-Light Image EnhancementIEEE Transactions on Consumer Electronics10.1109/TCE.2023.328046770:1(3332-3340)Online publication date: Feb-2024
https://doi.org/10.1109/TCE.2023.3280467
Ma XZhou XHuang HJia GWang YChen XChen C(2024)Uncertainty-aware image inpainting with adaptive feedback network▪Expert Systems with Applications: An International Journal10.1016/j.eswa.2023.121148235:COnline publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.eswa.2023.121148
Quan WChen JLiu YYan DWonka P(2024)Deep Learning-Based Image and Video Inpainting: A SurveyInternational Journal of Computer Vision10.1007/s11263-023-01977-6132:7(2367-2400)Online publication date: 19-Jan-2024
https://dl.acm.org/doi/10.1007/s11263-023-01977-6
Shen HZhao ZZhang YZhang ZEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Mutual Information-driven Triple Interaction Network for Efficient Image DehazingProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612299(7-16)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612299
Bao XZheng HZhang ZZha ZWang M(2023)Hierarchical Context Modeling Network for Landmark Recognition2023 IEEE International Conference on Data Mining (ICDM)10.1109/ICDM58522.2023.00103(930-937)Online publication date: 1-Dec-2023
https://doi.org/10.1109/ICDM58522.2023.00103
Li GZhang KSu YWang J(2023)Feature pre-inpainting enhanced transformer for video inpaintingEngineering Applications of Artificial Intelligence10.1016/j.engappai.2023.106323123:PBOnline publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1016/j.engappai.2023.106323
Tian XXie KZhang H(2022)A Low-Rank Tensor Decomposition Model With Factors Prior and Total Variation for Impulsive Noise RemovalIEEE Transactions on Image Processing10.1109/TIP.2022.316969431(4776-4789)Online publication date: 2022
https://doi.org/10.1109/TIP.2022.3169694

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents