research-article

Progressive network based on detail scaling and texture extraction: : A more general framework for image deraining

Authors:

Defeng Zhou, and

Calvin Yu-Chian ChenAuthors Info & Claims

Volume 568, Issue C

https://doi.org/10.1016/j.neucom.2023.127066

Published: 14 March 2024 Publication History

Abstract

Many feature extraction components have been proposed for image deraining tasks, aiming to improve feature learning. However, few models have addressed the integration of multi-scale features from derain images. The fusion of multiple features at different scales in one model has the potential to significantly enhance the authenticity and detail of rainy images restoration. This study introduces a migratable multi-scale feature blending model, which is a progressive learning model based on detail dilation and texture extraction. First, the degraded image is sent to the detail dilation module, which is designed to increase the detailed outline and obtain the coarse image features. Second, the extracted feature maps are sent to the multi-scale feature extraction (MFE) module and the multi-scale hybrid strategy (MHS) module for improved texture restoration. Third, the simple convolution modules are replaced by an optimized transformer model to more efficiently extract contextual features and multi-scale information in images. Finally, a progressive learning strategy is employed to incrementally restore the degraded images. Empirical results show that our proposed module for progressive restoration achieves near state-of-the-art performance in several rain removal tasks. In particular, our model exhibits better rain removal realism compared to state-of-the-art models. The source code is available at https://github.com/JackAILab/DTPNet.

Highlights

•

Introduce detail scaling module to extract generalized features from rainfall image.

•

Improved Transform block was introduced to enhance the model’s generalized ability.

•

Scale-mixing strategy is proposed for capturing more multi-scale features in images.

References

[1]

Cai L., Fu Y., Huo W., Xiang Y., Zhu T., Zhang Y., Zeng H., Zeng D., Multiscale attentive image de-raining networks via neural architecture search, IEEE Trans. Circuits Syst. Video Technol. 33 (2) (2022) 618–633.

[2]

Agrawal S.C., Jalal A.S., A comprehensive review on analysis and implementation of recent image dehazing methods, Arch. Comput. Methods Eng. 29 (7) (2022) 4799–4850.

[3]

Guo F., Yang J., Liu Z., Tang J., Haze removal for single image: A comprehensive review, Neurocomputing (2023).

[4]

Jiang N., Luo J., Lin J., Chen W., Zhao T., Lightweight semi-supervised network for single image rain removal, Pattern Recognit. 137 (2023).

[5]

Wang H., Xie Q., Zhao Q., Li Y., Liang Y., Zheng Y., Meng D., RCDNet: An interpretable rain convolutional dictionary network for single image deraining, IEEE Trans. Neural Netw. Learn. Syst. (2023).

[6]

Pan J., Xu B., Bai H., Tang J., Yang M.-H., Cascaded deep video deblurring using temporal sharpness prior and non-local spatial-temporal similarity, IEEE Trans. Pattern Anal. Mach. Intell. (2023).

[7]

Jiang K., Wang Z., Yi P., Chen C., Wang Z., Wang X., Jiang J., Lin C.-W., Rain-free and residue hand-in-hand: A progressive coupled network for real-time image deraining, IEEE Trans. Image Process. 30 (2021) 7404–7418.

[8]

Fu X., Huang J., Ding X., Liao Y., Paisley J., Clearing the skies: A deep network architecture for single-image rain removal, IEEE Trans. Image Process. 26 (6) (2017) 2944–2956.

Digital Library

[9]

W. Wei, D. Meng, Q. Zhao, Z. Xu, Y. Wu, Semi-supervised transfer learning for image rain removal, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 3877–3886.

[10]

H. Zhang, V.M. Patel, Density-aware single image de-raining using a multi-stream dense network, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 695–704.

[11]

R. Yasarla, V.M. Patel, Uncertainty guided multi-scale residual learning-using a cycle spinning cnn for single image de-raining, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 8405–8414.

[12]

X. Li, J. Wu, Z. Lin, H. Liu, H. Zha, Recurrent squeeze-and-excitation context aggregation net for single image deraining, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 254–269.

[13]

D. Ren, W. Zuo, Q. Hu, P. Zhu, D. Meng, Progressive image deraining networks: A better and simpler baseline, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 3937–3946.

[14]

K. Jiang, Z. Wang, P. Yi, C. Chen, B. Huang, Y. Luo, J. Ma, J. Jiang, Multi-scale progressive fusion network for single image deraining, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 8346–8355.

[15]

S.W. Zamir, A. Arora, S. Khan, M. Hayat, F.S. Khan, M.-H. Yang, L. Shao, Multi-stage progressive image restoration, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 14821–14831.

[16]

K. Purohit, M. Suin, A. Rajagopalan, V.N. Boddeti, Spatially-adaptive image restoration using distortion-guided networks, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 2309–2319.

[17]

S.W. Zamir, A. Arora, S. Khan, M. Hayat, F.S. Khan, M.-H. Yang, Restormer: Efficient transformer for high-resolution image restoration, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5728–5739.

[18]

Jiang T.-X., Huang T.-Z., Zhao X.-L., Deng L.-J., Wang Y., Fastderain: A novel video rain streak removal method using directional gradient priors, IEEE Trans. Image Process. 28 (4) (2018) 2089–2102.

[19]

Su Z., Zhang Y., Zhang X.-P., Qi F., Non-local channel aggregation network for single image rain removal, Neurocomputing 469 (2022) 261–272.

[20]

Rudin L.I., Osher S., Fatemi E., Nonlinear total variation based noise removal algorithms, Physica D 60 (1–4) (1992) 259–268.

Digital Library

[21]

Z. Wang, X. Cun, J. Bao, W. Zhou, J. Liu, H. Li, Uformer: A general u-shaped transformer for image restoration, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 17683–17693.

[22]

Cai L., Fu Y., Zhu T., Xiang Y., Zhang Y., Zeng H., Joint depth and density guided single image de-raining, IEEE Trans. Circuits Syst. Video Technol. 32 (7) (2021) 4108–4121.

[23]

Zhang Z., Wei Y., Zhang H., Yang Y., Yan S., Wang M., Data-driven single image deraining: A comprehensive review and new perspectives, Pattern Recognit. (2023).

[24]

Bai Y., Liu M., Yao C., Lin C., Zhao Y., MSPNet: Multi-stage progressive network for image denoising, Neurocomputing 517 (2023) 71–80.

[25]

Chen J., Yang Z., Chan T.N., Li H., Hou J., Chau L.-P., Attention-guided progressive neural texture fusion for high dynamic range image restoration, IEEE Trans. Image Process. 31 (2022) 2661–2672.

[26]

J. Yoo, S.-h. Lee, N. Kwak, Image restoration by estimating frequency distribution of local patches, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 6684–6692.

[27]

Tian C., Zheng M., Zuo W., Zhang B., Zhang Y., Zhang D., Multi-stage image denoising with the wavelet transform, Pattern Recognit. 134 (2023).

[28]

Du S., Fan H., Zhao M., Zong H., Hu J., Li P., A two-stage method for single image de-raining based on attention smoothed dilated network, IET Image Process. 16 (10) (2022) 2557–2567.

[29]

Chang Y., Yan L., Fang H., Zhong S., Liao W., HSI-DeNet: Hyperspectral image restoration via convolutional neural network, IEEE Trans. Geosci. Remote Sens. 57 (2) (2018) 667–682.

[30]

X. Liu, M. Suganuma, Z. Sun, T. Okatani, Dual residual networks leveraging the potential of paired operations for image restoration, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 7007–7016.

[31]

Zhang Y., Tian Y., Kong Y., Zhong B., Fu Y., Residual dense network for image restoration, IEEE Trans. Pattern Anal. Mach. Intell. 43 (7) (2020) 2480–2495.

[32]

Shao M., Qiao Y., Meng D., Zuo W., Uncertainty-guided hierarchical frequency domain transformer for image restoration, Knowl.-Based Syst. 263 (2023).

[33]

G. Yu, J. Zhang, Z. Ma, H. Wang, Efficient progressive high dynamic range image restoration via attention and alignment network, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 1124–1131.

[34]

Yang W., Liu J., Yang S., Guo Z., Scale-free single image deraining via visibility-enhanced recurrent wavelet learning, IEEE Trans. Image Process. 28 (6) (2019) 2948–2961.

[35]

W. Yang, R.T. Tan, J. Feng, J. Liu, Z. Guo, S. Yan, Deep joint rain detection and removal from a single image, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1357–1366.

[36]

Shi W., Jiang F., Zhao D., Single image super-resolution with dilated convolution based multi-scale information learning inception module, in: 2017 IEEE International Conference on Image Processing, ICIP, IEEE, 2017, pp. 977–981.

[37]

K. Zhang, W. Zuo, S. Gu, L. Zhang, Learning deep CNN denoiser prior for image restoration, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 3929–3938.

[38]

P. Liu, H. Zhang, K. Zhang, L. Lin, W. Zuo, Multi-level wavelet-CNN for image restoration, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018, pp. 773–782.

[39]

X. Chen, J. Pan, K. Jiang, Y. Li, Y. Huang, C. Kong, L. Dai, Z. Fan, Unpaired deep image deraining using dual contrastive learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 2017–2026.

[40]

Wei Y., Zhang Z., Wang Y., Xu M., Yang Y., Yan S., Wang M., Deraincyclegan: Rain attentive cyclegan for single image deraining and rainmaking, IEEE Trans. Image Process. 30 (2021) 4788–4801.

[41]

Falahkheirkhah K., Yeh K., Confer M.P., Bhargava R., DRB-net: Dilated residual block network for infrared image restoration, in: International Symposium on Visual Computing, Springer, 2022, pp. 104–115.

[42]

Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A.N., Kaiser Ł., Polosukhin I., Attention is all you need, Adv. Neural Inf. Process. Syst. 30 (2017).

Digital Library

[43]

D. Yu, Q. Li, X. Wang, Z. Zhang, Y. Qian, C. Xu, DSTrans: Dual-Stream Transformer for Hyperspectral Image Restoration, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023, pp. 3739–3749.

[44]

Jiang K., Liu W., Wang Z., Zhong X., Jiang J., Lin C.-W., DAWN: Direction-aware attention wavelet network for image deraining, 2023.

[45]

R. Xu, Z. Xiao, J. Huang, Y. Zhang, Z. Xiong, EDPN: Enhanced deep pyramid network for blurry image restoration, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 414–423.

[46]

S. Zheng, C. Lu, Y. Wu, G. Gupta, SAPNet: Segmentation-aware progressive network for perceptual contrastive deraining, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp. 52–62.

[47]

Yang Y., Guan J., Huang S., Wan W., Xu Y., Liu J., End-to-end rain removal network based on progressive residual detail supplement, IEEE Trans. Multimed. 24 (2021) 1622–1636.

[48]

Lahiri A., Bairagya S., Bera S., Haldar S., Biswas P.K., Lightweight modules for efficient deep learning based image restoration, IEEE Trans. Circuits Syst. Video Technol. 31 (4) (2020) 1395–1410.

[49]

Wang Z., Bovik A.C., Sheikh H.R., Simoncelli E.P., Image quality assessment: from error visibility to structural similarity, IEEE Trans. Image Process. 13 (4) (2004) 600–612.

Digital Library

[50]

X. Fu, J. Huang, D. Zeng, Y. Huang, X. Ding, J. Paisley, Removing rain from single images via a deep detail network, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 3855–3863.

[51]

X. Hu, C.-W. Fu, L. Zhu, P.-A. Heng, Depth-attentional features for single-image rain removal, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 8022–8031.

[52]

T. Wang, X. Yang, K. Xu, S. Chen, Q. Zhang, R.W. Lau, Spatial attentive single-image deraining with a high quality real rain dataset, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 12270–12279.

[53]

H. Wang, Q. Xie, Q. Zhao, D. Meng, A model-driven deep neural network for single image rain removal, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 3103–3112.

[54]

C. Chen, H. Li, Robust representation learning with feedback for single image deraining, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 7742–7751.

[55]

Y. Luo, Y. Xu, H. Ji, Removing rain from a single image via discriminative sparse coding, in: Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 3397–3405.

[56]

Y. Li, R.T. Tan, X. Guo, J. Lu, M.S. Brown, Rain streak removal using layer priors, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2736–2744.

[57]

R. Qian, R.T. Tan, W. Yang, J. Su, J. Liu, Attentive generative adversarial network for raindrop removal from a single image, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 2482–2491.

[58]

S. Li, I.B. Araujo, W. Ren, Z. Wang, E.K. Tokuda, R.H. Junior, R. Cesar-Junior, J. Zhang, X. Guo, X. Cao, Single image deraining: A comprehensive benchmark analysis, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 3838–3847.

[59]

Zhang K., Luo W., Yu Y., Ren W., Zhao F., Li C., Ma L., Liu W., Li H., Beyond monocular deraining: Parallel stereo deraining network via semantic prior, Int. J. Comput. Vis. 130 (7) (2022) 1754–1769.

[60]

Zhang K., Li D., Luo W., Ren W., Dual attention-in-attention model for joint rain streak and raindrop removal, IEEE Trans. Image Process. 30 (2021) 7608–7619.

[61]

J.M.J. Valanarasu, R. Yasarla, V.M. Patel, Transweather: Transformer-based restoration of images degraded by adverse weather conditions, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 2353–2363.

[62]

R. Li, L.-F. Cheong, R.T. Tan, Heavy rain image restoration: Integrating physics model and conditional adversarial learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 1633–1642.

[63]

Geiger A., Lenz P., Stiller C., Urtasun R., Vision meets robotics: The kitti dataset, Int. J. Robot. Res. 32 (11) (2013) 1231–1237.

Digital Library

[64]

Yang Y., Zhang Y., Cui Z., Zhao H., Ouyang T., Single image deraining using scale constraint iterative update network, Expert Syst. Appl. 236 (2024).

[65]

Zhang K., Li D., Luo W., Ren W., Liu W., Enhanced spatio-temporal interaction learning for video deraining: faster and better, IEEE Trans. Pattern Anal. Mach. Intell. 45 (1) (2022) 1287–1293.

Cited By

Zhang DXu YMa LLi XZhang XPeng YChen Y(2024)SRENet: Structure recovery ensemble network for single image derainingApplied Intelligence10.1007/s10489-024-05382-554:5(4425-4442)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1007/s10489-024-05382-5

Index Terms

Progressive network based on detail scaling and texture extraction: A more general framework for image deraining
1. Computing methodologies

Index terms have been assigned to the content through auto-classification.

Recommendations

HDRD-Net: High-resolution detail-recovering image deraining network
Abstract
Image deraining aims to restore the clean scenes of rainy images, which facilitates a number of outdoor vision systems, such as autonomous driving, unmanned aerial vehicles and surveillance systems. This paper proposes a high-resolution detail-...
Read More
Single image deraining using multi-scales context information and attention network
Highlights
- We proposed a novel CNN-based framework for image deraining.
- We designed a ...
Abstract
The existing deraining methods based on convolutional neural networks (CNNs) have made great success, but some remaining rain streaks can degrade images drastically. In this work, we proposed an end-to-end multi-scale context ...
Read More
Progressive dense feature fusion network for single image deraining
Abstract
Deep Learning (DL) has achieved significant progress in single image deraining methods. Most of current DL methods, however, are still weak in image detail recovery and feature inherent correlation learning. In this work, we explore the detail ...
Highlights
- PFFM can excavate inherent features correlations and intensify the details feature.
- Detail perceptual loss is used to better measure the details difference.
- Contextual loss on target regions can recover the details of objects.
- ...
Read More

Comments

Information & Contributors

Information

Published In

cover image Neurocomputing

Neurocomputing Volume 568, Issue C

Feb 2024

249 pages

ISSN:0925-2312

Issue’s Table of Contents

Elsevier B.V.

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 14 March 2024

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Other Metrics

View Author Metrics

Citations

Cited By

Zhang DXu YMa LLi XZhang XPeng YChen Y(2024)SRENet: Structure recovery ensemble network for single image derainingApplied Intelligence10.1007/s10489-024-05382-554:5(4425-4442)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1007/s10489-024-05382-5

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents