Reference-Based Multi-Stage Progressive Restoration for Multi-Degraded Images
Pages 4982 - 4997
Abstract
Image restoration (IR) via deep learning has been vigorously studied in recent years. However, due to the ill-posed nature of the problem, it is challenging to recover the high-quality image details from a single distorted input especially when images are corrupted by multiple distortions. In this paper, we propose a multi-stage IR approach for progressive restoration of multi-degraded images via transferring similar edges/textures from the reference image. Our method, called a Reference-based Image Restoration Transformer (Ref-IRT), operates via three main stages. In the first stage, a cascaded U-Transformer network is employed to perform the preliminary recovery of the image. The proposed network consists of two U-Transformer architectures connected by feature fusion of the encoders and decoders, and the residual image is estimated by each U-Transformer in an easy-to-hard and coarse-to-fine fashion to gradually recover the high-quality image. The second and third stages perform texture transfer from a reference image to the preliminarily-recovered target image to further enhance the restoration performance. To this end, a quality-degradation-restoration method is proposed for more accurate content/texture matching between the reference and target images, and a texture transfer/reconstruction network is employed to map the transferred features to the high-quality image. Experimental results tested on three benchmark datasets demonstrate the effectiveness of our model as compared with other state-of-the-art multi-degraded IR methods. Our code and dataset are available at <uri>https://vinelab.jp/refmdir/</uri>.
References
[1]
X.-L. Zhao, W. Wang, T.-Y. Zeng, T.-Z. Huang, and M. K. Ng, “Total variation structured total least squares method for image restoration,” SIAM J. Sci. Comput., vol. 35, no. 6, pp. B1304–B1320, Jan. 2013.
[2]
M. Elad and M. Aharon, “Image denoising via sparse and redundant representations over learned dictionaries,” IEEE Trans. Image Process., vol. 15, no. 12, pp. 3736–3745, Dec. 2006.
[3]
J. Zhang, D. Zhao, and W. Gao, “Group-based sparse representation for image restoration,” IEEE Trans. Image Process., vol. 23, no. 8, pp. 3336–3351, Aug. 2014.
[4]
J.-F. Cai, E. J. Candès, and Z. Shen, “A singular value thresholding algorithm for matrix completion,” SIAM J. Optim., vol. 20, no. 4, pp. 1956–1982, Jan. 2010.
[5]
S. Gu, L. Zhang, W. Zuo, and X. Feng, “Weighted nuclear norm minimization with application to image denoising,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2014, pp. 2862–2869.
[6]
A. Buades, B. Coll, and J.-M. Morel, “A non-local algorithm for image denoising,” in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (CVPR), vol. 2, Jun. 2005, pp. 60–65.
[7]
W. Dong, L. Zhang, G. Shi, and X. Li, “Nonlocally centralized sparse representation for image restoration,” IEEE Trans. Image Process., vol. 22, no. 4, pp. 1620–1630, Apr. 2013.
[8]
W. Dong, G. Shi, and X. Li, “Image deblurring with low-rank approximation structured sparse representation,” in Proc. Asia–Pacific Signal Inf. Process. Assoc. Annu. Summit Conf., Dec. 2012, pp. 1–5.
[9]
X. Liu, X. Wu, J. Zhou, and D. Zhao, “Data-driven soft decoding of compressed images in dual transform-pixel domain,” IEEE Trans. Image Process., vol. 25, no. 4, pp. 1649–1659, Apr. 2016.
[10]
H. Wang, Y. Cen, Z. He, Z. He, R. Zhao, and F. Zhang, “Reweighted low-rank matrix analysis with structural smoothness for image denoising,” IEEE Trans. Image Process., vol. 27, no. 4, pp. 1777–1792, Apr. 2018.
[11]
C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 2, pp. 295–307, Feb. 2016.
[12]
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 770–778.
[13]
G. Lin, Q. Wu, L. Qiu, and X. Huang, “Image super-resolution using a dilated convolutional neural network,” Neurocomputing, vol. 275, pp. 1219–1230, Jan. 2018.
[14]
J. Dai et al., “Deformable convolutional networks,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Oct. 2017, pp. 764–773.
[15]
G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jul. 2017, pp. 4700–4708.
[16]
O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” in Proc. 18th Int. Conf. Med. Image Comput. Comput.-Assist. Intervent., vol. 9351, 2015, pp. 234–241.
[17]
I. Goodfellow et al., “Generative adversarial networks,” Commun. ACM, vol. 63, no. 11, pp. 139–144, 2020.
[18]
L. R. Medsker and L. Jain, “Recurrent neural networks,” Design Appl., vol. 5, pp. 64–67, May 2001.
[19]
X. Zhang, R. Jiang, T. Wang, and J. Wang, “Recursive neural network for video deblurring,” IEEE Trans. Circuits Syst. Video Technol., vol. 31, no. 8, pp. 3025–3036, Aug. 2021.
[20]
X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2018, pp. 7794–7803.
[21]
A. Vaswani et al., “Attention is all you need,” in Proc. Adv. Neural Inf. Process. Syst., vol. 30, 2017, pp. 1–22.
[22]
Z. Liu et al., “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2021, pp. 10012–10022.
[23]
S. Ayas and E. Tunc-Gormus, “SpectralSWIN: A spectral-Swin transformer network for hyperspectral image classification,” Int. J. Remote Sens., vol. 43, no. 11, pp. 4025–4044, Jun. 2022.
[24]
X. Huang, M. Dong, J. Li, and X. Guo, “A 3-D-Swin transformer-based hierarchical contrastive learning method for hyperspectral image classification,” IEEE Trans. Geosci. Remote Sens., vol. 60, 2022, Art. no.
[25]
H. Gong et al., “Swin-transformer-enabled YOLOv5 with attention mechanism for small object detection on satellite images,” Remote Sens., vol. 14, no. 12, p. 2861, Jun. 2022.
[26]
Z. Liu, Y. Tan, Q. He, and Y. Xiao, “SwinNet: Swin transformer drives edge-aware RGB-D and RGB-T salient object detection,” IEEE Trans. Circuits Syst. Video Technol., vol. 32, no. 7, pp. 4486–4497, Jul. 2022.
[27]
J. Ma, L. Tang, F. Fan, J. Huang, X. Mei, and Y. Ma, “SwinFusion: Cross-domain long-range learning for general image fusion via Swin transformer,” IEEE/CAA J. Autom. Sinica, vol. 9, no. 7, pp. 1200–1217, Jul. 2022.
[28]
Z. Wang, Y. Chen, W. Shao, H. Li, and L. Zhang, “SwinFuse: A residual Swin transformer fusion network for infrared and visible images,” 2022, arXiv:2204.11436.
[29]
A. Lin, B. Chen, J. Xu, Z. Zhang, G. Lu, and D. Zhang, “DS-TransUNet: Dual Swin transformer U-Net for medical image segmentation,” IEEE Trans. Instrum. Meas., vol. 71, pp. 1–15, 2022.
[30]
A. Hatamizadeh, V. Nath, Y. Tang, D. Yang, H. R. Roth, and D. Xu, “Swin UNETR: Swin transformers for semantic segmentation of brain tumors in MRI images,” in Proc. Int. MICCAI Brainlesion Workshop, 2022, pp. 272–284.
[31]
J. Liang, J. Cao, G. Sun, K. Zhang, L. Van G., and R. Timofte, “SwinIR: Image restoration using Swin transformer,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV) Workshops, Oct. 2021, pp. 1833–1844.
[32]
J. Fu et al., “Dual attention network for scene segmentation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2019, pp. 3146–3154.
[33]
S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang, “Restormer: Efficient transformer for high-resolution image restoration,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2022, pp. 5728–5739.
[34]
Z. Zhang, Z. Wang, Z. Lin, and H. Qi, “Image super-resolution by neural texture transfer,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2019, pp. 7982–7991.
[35]
X. Liu, M. Suganuma, X. Luo, and T. Okatani, “Restoring images with unknown degradation factors by recurrent use of a multi-branch network,” 2019, arXiv:1907.04508.
[36]
M. Suganuma, X. Liu, and T. Okatani, “Attention-based adaptive selection of operations for image restoration in the presence of unknown combined distortions,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Apr. 2019, pp. 9039–9048.
[37]
S. Kim, N. Ahn, and K.-A. Sohn, “Restoring spatially-heterogeneous distortions using mixture of experts network,” in Proc. Asian Conf. Comput. Vis., 2020, pp. 1–226.
[38]
Z. Huang, C. Li, F. Duan, and Q. Zhao, “Multi-distorted image restoration with tensor 1×1 convolutional layer,” in Proc. Int. Joint Conf. Neural Netw. (IJCNN), Jul. 2021, pp. 1–8.
[39]
W. Shin, N. Ahn, J.-H. Moon, and K.-A. Sohn, “Exploiting distortion information for multi-degraded image restoration,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), Jun. 2022, pp. 536–545.
[40]
F. Yang, H. Yang, J. Fu, H. Lu, and B. Guo, “Learning texture transformer network for image super-resolution,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2020, pp. 5791–5800.
[41]
L. Lu, W. Li, X. Tao, J. Lu, and J. Jia, “MASA-SR: Matching acceleration and spatial adaptation for reference-based image super-resolution,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2021, pp. 6368–6377.
[42]
Y. Jiang, K. C. K. Chan, X. Wang, C. C. Loy, and Z. Liu, “Robust reference-based super-resolution via C2-matching,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2021, pp. 2103–2112.
[43]
J. Z. Cao et al., “Reference-based image super-resolution with deformable attention transformer,” in Proc. Eur. Conf. Comput. Vis., 2022, pp. 325–342.
[44]
K. Yu, C. Dong, L. Lin, and C. C. Loy, “Crafting a toolchain for image restoration by deep reinforcement learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2018, pp. 2443–2452.
[45]
X. Li et al., “Learning disentangled feature representation for hybrid-distorted image restoration,” in Proc. Eur. Conf. Comput. Vis., 2020, pp. 313–329.
[46]
K. Zhang, J. Liang, L. Van Gool, and R. Timofte, “Designing a practical degradation model for deep blind image super-resolution,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2021, pp. 4791–4800.
[47]
H. Zheng, M. Ji, H. Wang, Y. Liu, and L. Fang, “CrossNet: An end-to-end reference-based super resolution network using cross-scale warping,” in Proc. Eur. Conf. Comput. Vis. (ECCV), 2018, pp. 88–104.
[48]
A. Dosovitskiy et al., “FlowNet: Learning optical flow with convolutional networks,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Dec. 2015, pp. 2758–2766.
[49]
Z.-S. Liu, W.-C. Siu, and L.-W. Wang, “Variational autoencoder for reference based image super-resolution,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), Jun. 2021, pp. 516–525.
[50]
Y. Xie, J. Xiao, M. Sun, C. Yao, and K. Huang, “Feature representation matters: End-to-end learning for reference-based image super-resolution,” in Proc. Eur. Conf. Comput. Vis., 2020, pp. 230–245.
[51]
L. Zhang, X. Li, D. He, F. Li, E. Ding, and Z. Zhang, “LMR: A large-scale multi-reference dataset for reference-based super-resolution,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2023, pp. 13118–13127.
[52]
Y. Zhou, C. Barnes, E. Shechtman, and S. Amirghodsi, “TransFill: Reference-guided image inpainting by merging multiple color and spatial transformations,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2021, pp. 2266–2276.
[53]
T. Liu, L. Liao, Z. Wang, and S. Satoh, “Reference-guided texture and structure inference for image inpainting,” in Proc. IEEE Int. Conf. Image Process. (ICIP), Oct. 2022, pp. 1996–2000.
[54]
Y. Zhao, C. Barnes, Y. Zhou, E. Shechtman, S. Amirghodsi, and C. Fowlkes, “GeoFill: Reference-based image inpainting with better geometric understanding,” in Proc. IEEE/CVF Winter Conf. Appl. Comput. Vis. (WACV), Jan. 2023, pp. 1776–1786.
[55]
D. Yoon, J. Kwak, Y. Li, D. Han, Y. Jin, and H. Ko, “Reference guided image inpainting using facial attributes,” 2023, arXiv:2301.08044.
[56]
R. Yasarla, H. R. V. Joze, and V. M. Patel, “Network architecture search for face enhancement,” 2021, arXiv:2105.06528.
[57]
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” 2014, arXiv:1409.1556.
[58]
Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Trans. Image Process., vol. 13, no. 4, pp. 600–612, Apr. 2004.
[59]
E. Agustsson and R. Timofte, “NTIRE 2017 challenge on single image super-resolution: Dataset and study,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), Jul. 2017, pp. 126–135.
[60]
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 2014, arXiv:1412.6980.
[61]
K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising,” IEEE Trans. Image Process., vol. 26, no. 7, pp. 3142–3155, Jul. 2017.
[62]
X. Liu, M. Suganuma, Z. Sun, and T. Okatani, “Dual residual networks leveraging the potential of paired operations for image restoration,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2019, pp. 7000–7009.
[63]
S. W. Zamir et al., “Learning enriched features for real image restoration and enhancement,” in Proc. Eur. Conf. Comput. Vis., vol. 12370, 2020, pp. 492–511.
[64]
C. Mou, J. Zhang, X. Fan, H. Liu, and R. Wang, “COLA-Net: Collaborative attention network for image restoration,” IEEE Trans. Multimedia, vol. 24, pp. 1366–1377, 2022.
[65]
D. Jha, M. A. Riegler, D. Johansen, P. Halvorsen, and H. D. Johansen, “DoubleU-Net: A deep convolutional neural network for medical image segmentation,” in Proc. IEEE 33rd Int. Symp. Comput.-Based Med. Syst. (CBMS), Jul. 2020, pp. 558–564.
[66]
X. Xia and B. Kulis, “W-Net: A deep model for fully unsupervised image segmentation,” 2017, arXiv:1711.08506.
[67]
A. Sevastopolsky, S. Drapak, K. Kiselev, B. M. Snyder, J. D. Keenan, and A. Georgievskaya, “Stack-U-Net: Refinement network for improved optic disc and cup image segmentation,” in Medical Imaging 2019: Image Processing. Bellingham, WA, USA: SPIE, 2019, pp. 576–584.
[68]
R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2018, pp. 586–595.
[69]
K. Ding, K. Ma, S. Wang, and E. P. Simoncelli, “Image quality assessment: Unifying structure and texture similarity,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 5, pp. 2567–2581, May 2022.
[70]
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” Commun. ACM, vol. 60, no. 6, pp. 84–90, May 2017.
[71]
H. R. Sheikh, Z. Wang, A. C. Bovik, and L. K. Cormack. Image and Video Quality Assessment Research at Live. [Online]. Available: http://live.ece.utexas.edu/research/quality/
[72]
D. M. Chandler, “Most apparent distortion: Full-reference image quality assessment and the role of strategy,” J. Electron. Imag., vol. 19, no. 1, Jan. 2010, Art. no.
[73]
D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” in Proc. 8th IEEE Int. Conf. Comput. Vis., Jul. 2001, pp. 416–423.
[74]
D. Ghadiyaram and A. C. Bovik, “Massive online crowdsourced study of subjective and objective picture quality,” IEEE Trans. Image Process., vol. 25, no. 1, pp. 372–387, Jan. 2016.
Index Terms
- Reference-Based Multi-Stage Progressive Restoration for Multi-Degraded Images
Index terms have been assigned to the content through auto-classification.
Recommendations
Unified Restoration Method for Different Degraded Images
ICOIP '10: Proceedings of the 2010 International Conference on Optoelectronics and Image Processing - Volume 02Blurred images are caused by many factors such as defocus, motion, and atmospheric turbulence. Due to the unknown various factors that cannot be distinguished in the blurred image, it is necessary to propose a unified method for image restoration. In ...
Multi-DIP: A General Framework for Unsupervised Multi-degraded Image Restoration
Neural Information ProcessingAbstractMost existing image restoration algorithms only perform a single task. But in the real world, the degradation pattern could be much more complex, such as blurred images that have been smudged or images with haze that have been blurred, and we call ...
Blind restoration of atmospherically degraded images by automatic best step-edge detection
Image restoration algorithms often require previous knowledge about the point spread function (PSF) of the disturbance. Deriving the PSF manually from a degraded ideal step-edge in the image is a well known procedure intended mainly for isotropic ...
Comments
Information & Contributors
Information
Published In
1941-0042 © 2024 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.
Publisher
IEEE Press
Publication History
Published: 01 January 2024
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 18 Feb 2025