Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article

Image inpainting based on GAN-driven structure- and texture-aware learning with application to object removal

Published: 01 August 2024 Publication History

Abstract

In this paper, a novel deep learning-based image inpainting framework consisting of restoring image structure and reconstructing image details from corrupted images is proposed. Most image inpainting methods in the literature aim at restoring image details, outlines, and colors, simultaneously, which may suffer from blurring, deformation, and unreasonable content recovery due to interference among various information. To solve these problems, a two-stage image inpainting deep neural network based on GAN (generative adversarial network) architecture is proposed. The proposed inpainting framework consists of two modules: (1) the first stage, called the structure-aware learning stage, aims at learning a GAN-based structure restoration network, focusing on recovering the low-frequency image component, including colors and outlines of the missing regions of the input corrupted image; and (2) the second stage, called the texture-aware learning stage, aims at learning a GAN-based detail refinement network, focusing on rebuilding the high-frequency image details and texture information. In particular, we also propose to remove details from the training images to better train the structure restoration network to avoid inadequate image structure recovery induced by richer image textures, where the detail reconstruction task is left to the second stage. This strategy achieves to balance the workload between the two stages and the image quality can be progressively enhanced through the two stages. Experimental results have shown that the proposed deep inpainting framework quantitatively and qualitatively achieves state-of-the-art performance on the well-known datasets, including the CelebA, Places2, and ImageNet datasets, compared with existing deep learning-based image inpainting approaches. More specifically, in terms of the two well-known image quality assessment metrics, PSNR (peak signal-to-noise ratio) and SSIM (structural similarity), the improvement percentage of the proposed method, compared with the baseline approach, respectively, ranges from 3.23 % to 11.12 %, and 1.95 % to 13.39 %. The improvements have been shown to stably and significantly outperform the compared state-of-the-art methods in most types of inpainting mask. We also show that the proposed method is applicable to image editing in object removal from a single image.

Highlights

A two-stage deep image inpainting architecture is proposed for learning image structure and texture.
A blurring strategy is proposed for creating training image pairs to avoid implausible image reconstruction, induced by richer image textures.
Our two-stage deep image inpainting framework outperforms or is competitive to state-of-the-art methods on the three benchmarks.

References

[1]
Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, Coloma Ballester, Image inpainting, in: Proceedings of the ACM Conference on Computer Graphics and Interactive Techniques, 2000, pp. 417–24.
[2]
Christine Guillemot, Olivier Le Meur, Image inpainting: overview and recent advances, IEEE Signal Process. Mag. 31 (1) (2014) 127–144.
[3]
Omar Elharrouss, Noor Almaadeed, Somaya Al-Maadeed, Younes Akbari, Image inpainting: a review, Neural Process. Lett. 51 (2020) 2007–2028.
[4]
Jireh Jam, Connah Kendrick, Kevin Walker, Vincent Drouard, Gee-Sern Jison Hsu, Moi Hoon Yap, A comprehensive review of past and present image inpainting methods, Comput. Vis. Image Underst. 203 (2021).
[5]
A. Criminisi, P. P´erez, K. Toyama, Region filling and object removal by exemplar-based image inpainting, IEEE Trans. Image Process. 13 (9) (2004) 1200–1212.
[6]
Connelly Barnes, Eli Shechtman, Adam Finkelstein, Dan B. Goldman, PatchMatch: a randomized correspondence algorithm for structural image editing, ACM Trans. Graph. 28 (3) (2009) 24–33.
[7]
Zongben Xu, Jian Sun, Image inpainting by patch propagation using patch sparsity, IEEE Trans. Image Process. 19 (5) (2010) 1153–1165.
[8]
Ding Ding, Sundaresh Ram, Jeffrey J. Rodriguez, Perceptually aware image inpainting, Pattern Recognit. 83 (2018) 174–184.
[9]
Pengzhi Li, Yan Pei, Jianqiang Li, A comprehensive survey on design and application of autoencoder in deep learning, Appl. Soft Comput. 138 (2023).
[10]
Ahmad Ali Rafiee, Mahmoud Farhang, A deep convolutional neural network for salt-and-pepper noise removal using selective convolutional blocks, Appl. Soft Comput. (2023).
[11]
Chia-Hung Yeh, Chih-Hsiang Huang, Li-Wei Kang, Multi-scale deep residual learning-based single image haze removal via image decomposition, IEEE Trans. Image Process 29 (2020) 3153–3167.
[12]
Chih-Yang Lin, Zhuang Tao, Ai-Sheng Xu, Li-Wei Kang, Fityanul Akhyar, Sequential dual attention network for rain streak removal in a single image, IEEE Trans. Image Process. 29 (2020) 9250–9265.
[13]
Chia-Hung Yeh, Chu-Han Lin, Min-Hui Lin, Li-Wei Kang, Chih-Hsiang Huang, Mei-Juan Chen, Deep learning-based compressed image artifacts reduction based on multi-scale image fusion, Inf. Fusion 67 (2021) 195–207.
[14]
Elizabeth Irenne Yuwono, Dian Tjondonegoro, Golam Sorwar, Alireza Alaei, Scalability of knowledge distillation in incremental deep learning for fast object detection, Appl. Soft Comput. 129 (2022).
[15]
Chia-Hung Yeh, Chu-Han Lin, Li-Wei Kang, Chih-Hsiang Huang, Min-Hui Lin, Chuan-Yu Chang, Chua-Chin Wang, Lightweight deep neural network for joint learning of underwater object detection and color conversion, IEEE Trans. Neural Netw. Learn. Syst. 33 (11) (2022) 6129–6143.
[16]
Zhen Qin, Qingliang Zeng, Yixin Zong, Fan Xu, Image inpainting based on deep learning: a review, Displays 69 (2021).
[17]
Hanyu Xiang, Qin Zou, Muhammad Ali Nawaz, Xianfeng Huang, Fan Zhang, Hongkai Yu, Deep learning for image inpainting: a survey, Pattern Recognit. 134 (2023).
[18]
Xiaobo Zhang, Donghai Zhai, Tianrui Li, Yuxin Zhou, Yang Lin, Image inpainting based on deep learning: a review, Inf. Fusion 90 (2023) 74–94.
[19]
Yann LeCun, Yoshua Bengio, Geoffrey Hinton, Deep learning, Nature 521 (2015) 436–444.
[20]
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, Yoshua Bengio, Generative adversarial nets, in: Advances in Neural Information Processing Systems, 2014, pp. 2672–80.
[21]
Evan Shelhamer, Jonathan Long, Trevor Darrell, Fully convolutional networks for semantic segmentation, IEEE Trans. Pattern Anal. Mach. Intell. 39 (4) (2017) 640–651.
[22]
Olaf Ronneberger, Philipp Fischer, Thomas Brox, U-net: convolutional networks for biomedical image segmentation, in: Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, 2015, pp. 234–41.
[23]
Nian Cai, Zhenghang Su, Zhineng Lin, Han Wang, Zhijing Yang, Bingo Wing-Kuen Ling, Blind inpainting using the fully convolutional neural network, Vis. Comput. 33 (2) (2017) 249–261.
[24]
Yu-Zhe Su, Tsung-Jung Liu, Kuan-Hsien Liu, Hsin-Hua Liu, Soo-Chang Pei, Image inpainting for random areas using dense context features, in: Proceedings of the IEEE International Conference Image Processing, 2019, pp. 4679–83.
[25]
Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, Alexei A. EfrosContext encoders: feature learning by inpainting, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2536–44.
[26]
Satoshi Iizuka, Edgar Simo-Serra, Hiroshi Ishikawa, Globally and locally consistent image completion, ACM Trans. Graph. 36 (4) (2017).
[27]
Y. Li, S. Liu, J. Yang, M.-H. YangGenerative face completion, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 3911–9.
[28]
J. Zhang, R. Zhan, D. Sun, G. Pan, Symmetry-aware face completion with generative adversarial networks, in: Proceedings of the Asian Conference on Computer Vision, 2018, pp. 289–304.
[29]
H. Li, G. Li, L. Lin, Y. Yu, Context-aware semantic inpainting, IEEE Trans. Cybern. 49 (12) (2019) 4398–4411.
[30]
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, Thomas S. Huang, Generative image inpainting with contextual attention, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 5505–14.
[31]
Yuhang Song, Chao Yang, Yeji Shen, Peng Wang, Qin Huang, C.-C. Jay Kuo, SPG-Net: segmentation prediction and guidance network for image inpainting, in: Proceedings of the British Machine Vision Conference, 2018.
[32]
iahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, Thomas Huang, Free-form image inpainting with gated convolution, in: Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 4470–9.
[33]
Yanhong Zeng, Jianlong Fu, Hongyang Chao, Baining Guo, Learning pyramid-context encoder network for high-quality image inpainting, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 1486–94.
[34]
K. Nazeri, E. Ng, Tony Joseph, Faisal Qureshi, Mehran Ebrahimi, Edgeconnect: Structure guided image inpainting using edge prediction, in: Proceedings of the International Conference on Computer Vision Workshop, 2019, pp. 3265–74.
[35]
Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H. Li, Shan Liu, Ge Li, Structureflow: Image inpainting via structure-aware appearance flow, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019, pp. 181–90.
[36]
Ang Li, Jianzhong Qi, Rui Zhang, Ramamohanarao Kotagiri, Boosted GAN with semantically interpretable information for image inpainting, in: Proceedings of the International Joint Conference on Neural Networks (IJCNN), 2019.
[37]
Wei Xiong, Jiahui Yu, Zhe Lin, Jimei Yang, Xin Lu, Connelly Barnes, Jiebo Luo, Foreground-aware image inpainting, IEEE Conf. Comput. Vis. Pattern Recognit. (2019).
[38]
Mingwen Shao, Wentao Zhang, Wangmeng Zuo, Deyu Meng, Multi-scale generative adversarial inpainting network based on cross-layer attention transfer mechanism, Knowl.-Based Syst. 196 (2020).
[39]
Xian Zhang, Canghong Shi, Xin Wang, Xi Wu, Xiaojie Li, Jiancheng Lv, Imran Mumtaz, Face inpainting based on GAN by facial prediction and fusion as guidance information, Appl. Soft Comput. 111 (2021).
[40]
Ziwei Liu, Ping Luo, Xiaogang Wang, Xiaoou Tang, Deep learning face attributes in the wild, in: Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 3730–8.
[41]
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, Antonio Torralba, Places: a 10 million image database for scene recognition, IEEE Trans. Pattern Anal. Mach. Intell. 40 (6) (2017) 1452–1464.
[42]
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, Li Fei-Fei, ImageNet large scale visual recognition challenge, Int. J. Comput. Vis. 115 (3) (2015) 211–252.
[43]
Martin Arjovsky, Soumith Chintala, Léon Bottou, Wasserstein generative adversarial networks, in: Proceedings of the International Conference on Machine Learning, 2017, pp. 214–23.
[44]
Jingyuan Li, Ning Wang, Lefei Zhang, Bo Du, Dacheng Tao, Recurrent feature reasoning for image inpainting, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 7760–8.
[45]
Hongyu Liu, Bin Jiang, Yibing Song, Wei Huang, Chao Yang, Rethinking image inpainting via a mutual encoder-decoder with feature equalizations, in: Proceedings of the European Conference on Computer Vision, 2020, pp. 725–41.
[46]
Tao Yu, Zongyu Guo, Xin Jin, Shilin Wu, Zhibo Chen, Weiping Li, Zhizheng Zhang, Sen Liu, Region normalization for image inpainting, in: Proceedings of the AAAI Conference on Artificial Intelligence, 2020, pp. 12733–740.
[47]
Haipeng Liu, Yang Wang, Meng Wang, Yong Rui, Delving globally into texture and structure for image inpainting, in: ACM Multimedia, 2022, pp. 1270–8.
[48]
Li Huang, Yaping Huang, DRGAN: a dual resolution guided low-resolution image inpainting, Knowl. Based Syst. 264 (2023).
[49]
Aäron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, Koray Kavukcuoglu, Conditional image generation with PixelCNN decoders, in: Proceedings of the International Conference on Neural Information Processing Systems, 2016, pp. 4797–805.
[50]
Jie Hu, Li Shen, Gang Sun, Squeeze-and-excitation networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 7132–41.
[51]
Yann N. Dauphin, Angela Fan, Michael Auli, David Grangier, Language modeling with gated convolutional networks, in: Proceedings of the International Conference on Machine Learning, 2017, pp. 933–41.
[52]
Djork-Arné Clevert, Thomas Unterthiner, Sepp Hochreiter, Fast and accurate deep network learning by exponential linear units (ELUs), in: Proceedings of the International Conference on Learning Representations, 2016.
[53]
V. Nair, G.E. Hinton, Rectified linear units improve restricted boltzmann machines, in: Proceedings of the International Conference on Machine Learning, Haifa, Israel, 2010, pp. 807–14.
[54]
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, Yuichi Yoshida, Spectral normalization for generative adversarial networks, in: Proceedings of the International Conference on Learning Representations, 2018.
[55]
J. Johnson, A. Alahi, F.-F. Li, Perceptual losses for real-time style transfer and super-resolution, in: Proceedings of the European Conference on Computer Vision, 2016, pp. 694–711.
[56]
K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, in: Proceedings of the International Conference on Learning Representations, 2015.
[57]
Leon A. Gatys, Alexander S. Ecker, Matthias Bethge, Image style transfer using convolutional neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2414–23.
[58]
Guilin Liu, Fitsum A. Reda, Kevin J. Shih, Ting-Chun Wang, Andrew Tao, Bryan Catanzaro, Image inpainting for irregular holes using partial convolutions, in: Proceedings of the European Conference Computer Vision, 2018, pp. 89–105.
[59]
D.P. Kingma, J.L. Ba, Adam: a method for stochastic optimization, in: Proceedings of the Int. Conf. Learning Representations, 2015.
[60]
Q. Huynh-Thu, M. Ghanbari, Scope of validity of PSNR in image/video quality assessment, IET Electron. Lett. 44 (13) (2008) 800–801.
[61]
Z. Wang, A.C. Bovik, H.R. Sheikh, E.P. Simoncelli, Image quality assessment: from error visibility to structural similarity, IEEE Trans. Image Process. 13 (4) (2004) 600–612.
[62]
M.H. Amiri, N.M. Hashjin, M. Montazeri, S. Mirjalili, N. Khodadadi, Hippopotamus optimization algorithm: a novel nature-inspired optimization algorithm, Sci. Rep. 14 (2024) 5032.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Applied Soft Computing
Applied Soft Computing  Volume 161, Issue C
Aug 2024
1077 pages

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 01 August 2024

Author Tags

  1. Image inpainting
  2. Image restoration
  3. Deep learning
  4. Convolutional neural networks
  5. Generative adversarial networks

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 0
    Total Downloads
  • Downloads (Last 12 months)0
  • Downloads (Last 6 weeks)0
Reflects downloads up to 24 Jan 2025

Other Metrics

Citations

View Options

View options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media