research-article

Non-local feature aggregation quaternion network for single image deraining

Authors:

Chengli SunAuthors Info & Claims

Volume 103, Issue C

https://doi.org/10.1016/j.jvcir.2024.104250

Published: 01 August 2024 Publication History

Abstract

The existing deraining methods are based on convolutional neural networks (CNN) learning the mapping relationship between rainy and clean images. However, the real-valued CNN processes the color images as three independent channels separately, which fails to fully leverage color information. Additionally, sliding-window-based neural networks cannot effectively model the non-local characteristics of an image. In this work, we proposed a non-local feature aggregation quaternion network (NLAQNet), which is composed of two concurrent sub-networks: the Quaternion Local Detail Repair Network (QLDRNet) and the Multi-Level Feature Aggregation Network (MLFANet). Furthermore, in the subnetwork of QLDRNet, the Local Detail Repair Block (LDRB) is proposed to repair the backdrop of an image that has not been damaged by rain streaks. Finally, within the MLFANet subnetwork, we have introduced two specialized blocks, namely the Non-Local Feature Aggregation Block (NLAB) and the Feature Aggregation Block (Mix), specifically designed to address the restoration of rain-streak-damaged image backgrounds. Extensive experiments demonstrate that the proposed network delivers strong performance in both qualitative and quantitative evaluations on existing datasets. The code is available at https://github.com/xionggonghe/NLAQNet.

References

[1]

Xu K., Tian X., Yang X., Yin B., Lau R.W.H., Intensity-aware single-image deraining with semantic and color regularization, IEEE Trans. Image Process. 30 (2021) 8497–8509.

[2]

S. Deng, M. Wei, J. Wang, L. Liang, H. Xie, M. Wang, Detail-recovery Image Deraining via Context Aggregation Networks, in: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2019, pp. 14548–14557.

[3]

Y. Luo, Y. Xu, H. Ji, Removing Rain from a Single Image via Discriminative Sparse Coding, in: 2015 IEEE International Conference on Computer Vision, ICCV, 2015, pp. 3397–3405.

[4]

Y.-L. Chen, C.-T. Hsu, A Generalized Low-Rank Appearance Model for Spatio-temporally Correlated Rain Streaks, in: 2013 IEEE International Conference on Computer Vision, 2013, pp. 1968–1975.

[5]

Kang L.-W., Lin C.-W., Fu Y.-H., Automatic single-image-based rain streaks removal via image decomposition, IEEE Trans. Image Process. 21 (2012) 1742–1755.

[6]

X. Fu, J. Huang, D. Zeng, Y. Huang, X. Ding, J.W. Paisley, Removing Rain from Single Images via a Deep Detail Network, in: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2017, pp. 1715–1723.

[7]

Zhang H., Sindagi V.A., Patel V.M., Image de-raining using a conditional generative adversarial network, IEEE Trans. Circuits Syst. Video Technol. 30 (2017) 3943–3956.

[8]

K. Jiang, Z. Wang, P. Yi, C. Chen, B. Huang, Y. Luo, J. Ma, J. Jiang, Multi-Scale Progressive Fusion Network for Single Image Deraining, in: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2020, pp. 8343–8352.

[9]

H. Zhang, V.M. Patel, Density-Aware Single Image De-raining Using a Multi-stream Dense Network, in: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 695–704.

[10]

Jin L., Zhu Z., Song E., Xu X., An effective vector filter for impulse noise reduction based on adaptive quaternion color distance mechanism, Signal Process. 155 (2019) 334–345.

[11]

Xiao X., Chen Y., jiao Gong Y., Zhou Y., 2D quaternion sparse discriminant analysis, IEEE Trans. Image Process. 29 (2020) 2271–2286.

[12]

Xu Y., Yu L., Xu H., Zhang H., Nguyen T., Vector sparse representation of color image using quaternion matrix analysis, IEEE Trans. Image Process. 24 (2015) 1315–1329.

[13]

Gai S., Theory of reduced biquaternion sparse representation and its applications, Expert Syst. Appl. 213 (2023).

[14]

C.J. Gaudet, A. Maida, Deep Quaternion Networks, in: 2018 International Joint Conference on Neural Networks, IJCNN, 2017, pp. 1–8.

[15]

Gai S., Huang X., Reduced biquaternion convolutional neural network for color image processing, IEEE Trans. Circuits Syst. Video Technol. 32 (2021) 1061–1075.

[16]

Yin Q., Wang J., Luo X., Zhai J., Jha S.K., Shi Y.Q., Quaternion convolutional neural network for color image classification and forensics, IEEE Access 7 (2019) 20293–20301.

[17]

Zhou Y., Jin L., Liu H., Song E., Color facial expression recognition by quaternion convolutional neural network with gabor attention, IEEE Trans. Cogn. Dev. Syst. 13 (2021) 969–983.

[18]

X. Zhu, Y. Xu, H. Xu, C. Chen, Quaternion Convolutional Neural Networks, in: European Conference on Computer Vision, 2018.

[19]

Mehta S., Rastegari M., MobileViT: Light-weight, general-purpose, and mobile-friendly vision transformer, 2021, ArXiv arXiv:2110.02178.

[20]

Mehta S., Rastegari M., Separable self-attention for mobile vision transformers, 2022, ArXiv arXiv:2206.02680.

[21]

Dosovitskiy A., Beyer L., Kolesnikov A., Weissenborn D., Zhai X., Unterthiner T., Dehghani M., Minderer M., Heigold G., Gelly S., Uszkoreit J., Houlsby N., An image is worth 16x16 words: Transformers for image recognition at scale, 2020, ArXiv arXiv:2010.11929.

[22]

J.-H. Kim, C. Lee, J.-Y. Sim, C.-S. Kim, Single-image deraining using an adaptive nonlocal means filter, in: 2013 IEEE International Conference on Image Processing, 2013, pp. 914–917.

[23]

Y. Li, R.T. Tan, X. Guo, J. Lu, M.S. Brown, Rain Streak Removal Using Layer Priors, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2016, pp. 2736–2744.

[24]

J. Xiao, M. Zhou, X. Fu, A. Liu, Z. Zha, Improving De-raining Generalization via Neural Reorganization, in: 2021 IEEE/CVF International Conference on Computer Vision, ICCV, 2021, pp. 4967–4976.

[25]

Huang Z.X., Zhang J., Contrastive unfolding deraining network., IEEE Trans. Neural Netw. Learn. Syst. PP (2022).

[26]

Y. Ye, C.W. Yu, Y. Chang, L. Zhu, X. Zhao, L. Yan, Y. Tian, Unsupervised Deraining: Where Contrastive Learning Meets Self-similarity, in: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2022, pp. 5811–5820.

[27]

Jiang K., Wang Z., Yi P., Chen C., Han Z., Lu T., Huang B., Jiang J., Decomposition makes better rain removal: An improved attention-guided deraining network, IEEE Trans. Circuits Syst. Video Technol. 31 (2021) 3981–3995.

[28]

Jiang K., Wang Z., Yi P., Chen C., Wang Z., Wang X., Jiang J., Lin C.-W., Rain-free and residue hand-in-hand: A progressive coupled network for real-time image deraining, IEEE Trans. Image Process. 30 (2021) 7404–7418.

[29]

S.W. Zamir, A. Arora, S.H. Khan, M. Hayat, F.S. Khan, M.-H. Yang, L. Shao, Multi-Stage Progressive Image Restoration, in: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2021, pp. 14816–14826.

[30]

Y. Nanba, H. Miyata, X. Han, Dual Heterogeneous Complementary Networks for Single Image Deraining, in: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW, 2022, pp. 567–576.

[31]

Hao Z., Gai S., Li P., Multi-scale self-calibrated dual-attention lightweight residual dense deraining network based on monogenic wavelets, IEEE Trans. Circuits Syst. Video Technol. (2022).

[32]

Dabov K., Foi A., Katkovnik V., Egiazarian K.O., Image denoising by sparse 3-D transform-domain collaborative filtering, IEEE Trans. Image Process. 16 (2007) 2080–2095.

[33]

Lefkimmiatis S., Universal denoising networks : A novel CNN-based network architecture for image denoising, 2017, ArXiv arXiv:1711.07807.

[34]

S. Lefkimmiatis, Non-local Color Image Denoising with Convolutional Neural Networks, in: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2016, pp. 5882–5891.

[35]

Ulyanov D., Vedaldi A., Lempitsky V.S., Deep image prior, Int. J. Comput. Vis. 128 (2017) 1867–1888.

[36]

T. Plötz, S. Roth, Neural Nearest Neighbors Networks, in: Neural Information Processing Systems, 2018.

[37]

Y. Mei, Y. Fan, Y. Zhou, Image Super-Resolution with Non-Local Sparse Attention, in: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2021, pp. 3516–3525.

[38]

Wang Z., Ng M.K., Zhuang L., Gao L., Zhang B., Nonlocal self-similarity-based hyperspectral remote sensing image denoising with 3-D convolutional neural network, IEEE Trans. Geosci. Remote Sens. 60 (2022) 1–17.

[39]

Hu Y., Li J., Huang Y., Gao X., Image super-resolution with self-similarity prior guided network and sample-discriminating learning, IEEE Trans. Circuits Syst. Video Technol. 32 (2022) 1966–1985.

[40]

H. Chen, Y. Wang, T. Guo, C. Xu, Y. Deng, Z. Liu, S. Ma, C. Xu, C. Xu, W. Gao, Pre-Trained Image Processing Transformer, in: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2020, pp. 12294–12305.

[41]

Z. Wang, X. Cun, J. Bao, J. Liu, Uformer: A General U-Shaped Transformer for Image Restoration, in: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2021, pp. 17662–17672.

[42]

S.W. Zamir, A. Arora, S.H. Khan, M. Hayat, F.S. Khan, M.-H. Yang, Restormer: Efficient Transformer for High-Resolution Image Restoration, in: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2021, pp. 5718–5729.

[43]

Yan L., Zhao M., Liu S., Shi S., Chen J., Cascaded transformer U-net for image restoration, Signal Process. 206 (2022) URL https://api.semanticscholar.org/CorpusID:255089439.

[44]

Hu J., Shen L., Albanie S., Sun G., Wu E., Squeeze-and-excitation networks, IEEE Trans. Pattern Anal. Mach. Intell. 42 (2017) 2011–2023.

Digital Library

[45]

H. Wu, Y. Qu, S. Lin, J.J. Zhou, R. Qiao, Z. Zhang, Y. Xie, L. Ma, Contrastive Learning for Compact Single Image Dehazing, in: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2021, pp. 10546–10555.

[46]

Fu X., Huang J., Ding X., Liao Y., Paisley J.W., Clearing the skies: A deep network architecture for single-image rain removal, IEEE Trans. Image Process. 26 (2016) 2944–2956.

Digital Library

[47]

W. Yang, R.T. Tan, J. Feng, J. Liu, Z. Guo, S. Yan, Deep Joint Rain Detection and Removal from a Single Image, in: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2016, pp. 1685–1694.

[48]

Wu Q., Wang L., Ngan K.N., Li H., Meng F., Xu L., Subjective and objective de-raining quality assessment towards authentic rain image, IEEE Trans. Circuits Syst. Video Technol. 30 (2019) 3883–3897.

[49]

W. Wei, D. Meng, Q. Zhao, Z. Xu, Y. Wu, Semi-Supervised Transfer Learning for Image Rain Removal, in: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2018, pp. 3872–3881.

[50]

X. Li, J. Wu, Z. Lin, H. Liu, H. Zha, Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining, in: European Conference on Computer Vision, 2018.

[51]

R. Yasarla, V.M. Patel, Uncertainty Guided Multi-Scale Residual Learning-Using a Cycle Spinning CNN for Single Image De-Raining, in: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2019, pp. 8397–8406.

[52]

D. Ren, W. Zuo, Q. Hu, P.F. Zhu, D. Meng, Progressive Image Deraining Networks: A Better and Simpler Baseline, in: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2019, pp. 3932–3941.

[53]

Kingma D.P., Ba J., Adam: A method for stochastic optimization, 2014, CoRR arXiv:1412.6980.

[54]

Loshchilov I., Hutter F., SGDR: Stochastic gradient descent with warm restarts, 2016, arXiv: Learning.

Index Terms

Non-local feature aggregation quaternion network for single image deraining
1. Computing methodologies

Index terms have been assigned to the content through auto-classification.

Recommendations

Edge-preserving image deraining network using cumulative feature aggregation
Abstract
This study proposes an edge-preserving image deraining network using a wavelet feature aggregation method. Wavelet subbands re correlated with each other, and the high-frequency subband in the horizontal direction is the least affected by rain ...
Highlights
- This paper presents a single image deraining network that cumulatively aggregates wavelet subband features.
- The wavelet feature aggregation block integrates long short-term memory-based multiscale attentive rain streak removal blocks.
Haze transfer and feature aggregation network for real-world single image dehazing
Abstract
The absence of ground truth for hazy images in real scenes results in most dehazing models being trained only by synthesized datasets, which is not feasible for real hazy images due to the domain shift. To address this problem, a haze ...
Highlights
- We propose a haze transfer model for haze information from a real hazy image to a clear image.
Local multi-scale feature aggregation network for real-time image dehazing
Highlights
- We found the local high correlation mapping relationship between hazeclear image pairs.
Graphical abstract

Display Omitted

Abstract
Haze causes visual degradation and obscures image information, which gravely affects the reliability of computer vision tasks in real-time systems. Leveraging an enormous number of learning parameters as the restoration costs, learning-...

Comments

Information & Contributors

Information

Published In

cover image Journal of Visual Communication and Image Representation

Journal of Visual Communication and Image Representation Volume 103, Issue C

Aug 2024

474 pages

Issue’s Table of Contents

Elsevier Inc.

Publisher

Academic Press, Inc.

United States

Publication History

Published: 01 August 2024

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Feb 2025

Other Metrics

View Author Metrics

Citations

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents