Progressive Local and Non-Local Interactive Networks with Deeply Discriminative Training for Image Deraining
Authors: 
Cong Wang, Liyan Wang, Jie Mu, Chengjin Yu, Wei WangAuthors Info & Claims
Cong Wang, Liyan Wang, Jie Mu, Chengjin Yu, Wei WangAuthors Info & ClaimsPages 10326 - 10335
Abstract
In this paper, we develop a progressive local and non-local interactive network with multi-scale cross-content deeply discriminative learning to solve image deraining. The proposed model contains two key techniques: 1) Progressive Local and Non-Local Interactive Network (PLNLIN) and 2) Multi-Scale Cross-Content Deeply Discriminative Learning (MCDDL). The PLNLIN is a U-shaped encoder-decoder network, where the proposed new Progressive Local and Non-Local Interactive Module (PLNLIM) is the basic unit in the encoder-decoder framework. The PLNLIM fully explores local and non-local learning in convolution and Transformer operation respectively and the local and non-local content are further interactively learned in a progressive manner. The proposed MCDDL not only discriminates the output of the generator but also receives the deep content from the generator to distinguish real and fake features at each side layer of the discriminator in a multi-scale manner. We show that the proposed MCDDL has fast and stable convergence properties that lack in existing discriminative learning manners. Extensive experiments demonstrate that the proposed method outperforms state-of-the-art methods on five public synthetic datasets and one real-world data. The source codes will be made available at https://github.com/supersupercong/PLNLIN-MCDDL.
References
[1]
Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. 2021. Pre-Trained Image Processing Transformer. In IEEE CVPR. 12299--12310.
[2]
Xiang Chen, Jinshan Pan, Jiyang Lu, Zhentao Fan, and Hao Li. 2023. Hybrid CNN-Transformer Feature Fusion for Single Image Deraining. In Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7--14, 2023, Brian Williams, Yiling Chen, and Jennifer Neville (Eds.). AAAI Press, 378--386.
[3]
Yi-Lei Chen and Chiou-Ting Hsu. 2013. A Generalized Low-Rank Appearance Model for Spatio-temporally Correlated Rain Streaks. In IEEE ICCV. 1968--1975.
[4]
Xin Cui, Cong Wang, Dongwei Ren, Yunjin Chen, and Pengfei Zhu. 2022. Semi-Supervised Image Deraining Using Knowledge Distillation. IEEE Trans. Circuits Syst. Video Technol., Vol. 32, 12 (2022), 8327--8341.
[5]
Qili Deng, Ziling Huang, Chung-Chi Tsai, and Chia-Wen Lin. 2020. HardGAN: A Haze-Aware Representation Distillation GAN for Single Image Dehazing. In ECCV, Vol. 12351. 722--738.
[6]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR.
[7]
Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley. 2017. Removing Rain from Single Images via a Deep Detail Network. In IEEE CVPR. 1715--1723.
[8]
Xueyang Fu, Qi Qi, Zheng-Jun Zha, Yurui Zhu, and Xinghao Ding. 2021. Rain Streak Removal via Dual Graph Convolutional Network. In AAAI. 1352--1360.
[9]
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In NeurIPS. 2672--2680.
[10]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In IEEE CVPR. 770--778.
[11]
Haobo Ji, Xin Feng, Wenjie Pei, Jinxing Li, and Guangming Lu. 2021. U2-Former: A Nested U-shaped Transformer for Image Restoration. arXiv preprint arXiv:2112.02279 (2021).
[12]
Kui Jiang, Zhongyuan Wang, Peng Yi, Chen Chen, Baojin Huang, Yimin Luo, Jiayi Ma, and Junjun Jiang. 2020. Multi-Scale Progressive Fusion Network for Single Image Deraining. In IEEE CVPR. 8343--8352.
[13]
Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, and Zhangyang Wang. 2021. EnlightenGAN: Deep Light Enhancement Without Paired Supervision. IEEE Trans. Image Process., Vol. 30 (2021), 2340--2349.
[14]
Yeying Jin, Beibei Lin, Wending Yan, Yuan Yuan, Wei Ye, and Robby T Tan. 2023. Enhancing visibility in nighttime haze images using guided apsf and gradient adaptive convolution. In ACM MM. 2446--2457.
[15]
Yeying Jin, Aashish Sharma, and Robby T Tan. 2021. Dc-shadownet: Single-image hard and soft shadow removal using unsupervised domain-classifier guided network. In ICCV. 5027--5036.
[16]
Yeying Jin, Wenhan Yang, and Robby T Tan. 2022. Unsupervised night image enhancement: When layer decomposition meets light-effects suppression. In ECCV. 404--421.
[17]
Yeying Jin, Wei Ye, Wenhan Yang, Yuan Yuan, and Robby T Tan. 2024. DeS3: Adaptive Attention-Driven Self and Soft Shadow Removal Using ViT Similarity. In AAAI, Vol. 38. 2634--2642.
[18]
Yakun Ju, Muwei Jian, Cong Wang, Cong Zhang, Junyu Dong, and Kin-Man Lam. 2023. Estimating high-resolution surface normals via low-resolution photometric stereo images. IEEE TCSVT (2023).
[19]
Yakun Ju, Kin-Man Lam, Wuyuan Xie, Huiyu Zhou, Junyu Dong, and Boxin Shi. 2024. Deep Learning Methods for Calibrated Photometric Stereo and Beyond. IEEE TPAMI (2024).
[20]
Yakun Ju, Boxin Shi, Yang Chen, Huiyu Zhou, Junyu Dong, and Kin-Man Lam. 2023. GR-PSN: Learning to estimate surface normal and reconstruct photometric stereo images. IEEE TVCG (2023).
[21]
Animesh Karnewar and Oliver Wang. 2020. MSG-GAN: Multi-Scale Gradients for Generative Adversarial Networks. In IEEE CVPR. 7796--7805.
[22]
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR.
[23]
Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiri Matas. 2018. DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks. In IEEE CVPR. 8183--8192.
[24]
Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. 2019. DeblurGAN-v2: Deblurring (Orders-of-Magnitude) Faster and Better. In IEEE ICCV. 8877--8886.
[25]
Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P. Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. 2017. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In IEEE CVPR. 105--114.
[26]
Guanbin Li, Xiang He, Wei Zhang, Huiyou Chang, Le Dong, and Liang Lin. 2018. Non-locally Enhanced Encoder-Decoder Network for Single Image De-raining. In ACM MM. 1056--1064.
[27]
Runde Li, Jinshan Pan, Zechao Li, and Jinhui Tang. 2018. Single Image Dehazing via Conditional Generative Adversarial Network. In IEEE CVPR. 8202--8211.
[28]
Siyuan Li, Iago Breno Araujo, Wenqi Ren, Zhangyang Wang, Eric K. Tokuda, Roberto Hirata Junior, Roberto Cesar-Junior, Jiawan Zhang, Xiaojie Guo, and Xiaochun Cao. 2019. Single Image Deraining: A Comprehensive Benchmark Analysis. In IEEE CVPR. 3838--3847.
[29]
Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, and Hongbin Zha. 2018. Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining. In ECCV. 262--277.
[30]
Yu Li, Robby T. Tan, Xiaojie Guo, Jiangbo Lu, and Michael S. Brown. 2016. Rain Streak Removal Using Layer Priors. In IEEE CVPR. 2736--2744.
[31]
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. 2021. SwinIR: Image Restoration Using Swin Transformer. In IEEE ICCVW. 1833--1844.
[32]
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. 2021. SwinIR: Image Restoration Using Swin Transformer. In IEEE ICCVW. 1833--1844.
[33]
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows. In IEEE ICCV. 10012--10022.
[34]
Yu Luo, Yong Xu, and Hui Ji. 2015. Removing Rain from a Single Image via Discriminative Sparse Coding. In IEEE ICCV. 3397--3405.
[35]
Jinshan Pan, Jiangxin Dong, Yang Liu, Jiawei Zhang, Jimmy S. J. Ren, Jinhui Tang, Yu-Wing Tai, and Ming-Hsuan Yang. 2021. Physics-Based Generative Adversarial Models for Image Restoration and Beyond. IEEE TPAMI (2021).
[36]
Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng. 2019. Progressive Image Deraining Networks: A Better and Simpler Baseline. In IEEE CVPR. 3937--3946.
[37]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. In NeurIPS.
[38]
Cong Wang, Jinshan Pan, Wei Wang, Jiangxin Dong, Mengzhu Wang, Yakun Ju, and Junyang Chen. 2024. PromptRestorer: A Prompting Image Restoration Method with Degradation Perception. NeurIPS, Vol. 36 (2024).
[39]
Cong Wang, Jinshan Pan, Wei Wang, Gang Fu, Siyuan Liang, Mengzhu Wang, Xiao-Ming Wu, and Jun Liu. 2024. Correlation Matching Transformation Transformers for UHD Image Restoration. In AAAI, Vol. 38. 5336--5344.
[40]
Cong Wang, Jinshan Pan, and Xiao-Ming Wu. 2022. Online-Updated High-Order Collaborative Networks for Single Image Deraining. In AAAI. 2406--2413.
[41]
Cong Wang, Wei Wang, Chengjin Yu, and Jie Mu. 2024. Explore Internal and External Similarity for Single Image Deraining with Graph Neural Networks. In IJCAI.
[42]
Cong Wang, Yutong Wu, Yu Cai, Guangle Yao, Zhixun Su, and Hongyan Wang. 2020. Single image deraining via deep pyramid network with spatial contextual information aggregation. Appl. Intell., Vol. 50, 5 (2020), 1437--1447.
[43]
Cong Wang, Yutong Wu, Zhixun Su, and Junyang Chen. 2020. Joint Self-Attention and Scale-Aggregation for Self-Calibrated Deraining Network. In ACM MM. 2517--2525.
[44]
Cong Wang, Xiaoying Xing, Yutong Wu, Zhixun Su, and Junyang Chen. 2020 d. DCSFN: Deep Cross-scale Fusion Network for Single Image Rain Removal. In ACM MM. 1643--1651.
[45]
Cong Wang, Xiaoying Xing, Guangle Yao, and Zhixun Su. 2021. Single image deraining via deep shared pyramid network. Vis. Comput., Vol. 37, 7 (2021), 1851--1865.
[46]
Cong Wang, Man Zhang, Zhixun Su, Yutong Wu, Guangle Yao, and Hongyan Wang. 2019. Learning a multi-level guided residual network for single image deraining. Signal Process. Image Commun., Vol. 78 (2019), 206--215.
[47]
Cong Wang, Man Zhang, Zhixun Su, Guangle Yao, Yan Wang, Xiyan Sun, and Xiaonan Luo. 2019. From Coarse to Fine: A Stage-Wise Deraining Net. IEEE Access, Vol. 7 (2019), 84420--84428.
[48]
Hong Wang, Qi Xie, Qian Zhao, and Deyu Meng. 2020. A Model-Driven Deep Neural Network for Single Image Rain Removal. In IEEE CVPR. 3100--3109.
[49]
Lei Wang, Qingbo Wu, Desen Yuan, King Ngi Ngan, Hongliang Li, Fanman Meng, and Linfeng Xu. 2023. Learning with Noisy Low-Cost MOS for Image Quality Assessment via Dual-Bias Calibration. arXiv preprint arXiv:2311.15846 (2023).
[50]
Tianyu Wang, Xin Yang, Ke Xu, Shaozhe Chen, Qiang Zhang, and Rynson W. H. Lau. 2019. Spatial Attentive Single-Image Deraining With a High Quality Real Rain Dataset. In IEEE CVPR. 12270--12279.
[51]
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. 2021. Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction Without Convolutions. In IEEE ICCV. 568--578.
[52]
Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. 2018. Recovering Realistic Texture in Image Super-Resolution by Deep Spatial Feature Transform. In IEEE CVPR.
[53]
Zhendong Wang, Xiaodong Cun, Jianmin Bao, and Jianzhuang Liu. 2021. Uformer: A general u-shaped transformer for image restoration. arXiv preprint arXiv:2106.03106 (2021).
[54]
Wei Wei, Deyu Meng, Qian Zhao, Zongben Xu, and Ying Wu. 2019. Semi-Supervised Transfer Learning for Image Rain Removal. In IEEE CVPR. 3877--3886.
[55]
Yanyan Wei, Zhao Zhang, Mingliang Xu, Richang Hong, Jicong Fan, and Shuicheng Yan. 2022. Robust Attention Deraining Network for Synchronous Rain Streaks and Raindrops Removal. In MM '22: The 30th ACM International Conference on Multimedia, Lisboa, Portugal, October 10 - 14, 2022. ACM, 6464--6472.
[56]
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. 2021. SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers. In NeurIPS.
[57]
Kepeng Xu and Gang He. 2022. DNAS:A Decoupled Global Neural Architecture Search Method. In CVPR Workshops. 1979--1985.
[58]
Kepeng Xu, Li Xu, Gang He, Chang Wu, Zijia Ma, Ming Sun, and Yu-Wing Tai. 2022. Sdrtv-to-hdrtv conversion via spatial-temporal feature fusion. arXiv preprint arXiv:2211.02297 (2022).
[59]
Kepeng Xu, Li Xu, Gang He, Wenxin Yu, and Yunsong Li. 2024. Beyond Alignment: Blind Video Face Restoration via Parsing-Guided Temporal-Coherent Transformer. In IJCAI.
[60]
Wenhan Yang, Robby T. Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. 2017. Deep Joint Rain Detection and Removal from a Single Image. In IEEE CVPR. 1685--1694.
[61]
Wenhan Yang, Shiqi Wang, Dejia Xu, Xiaodong Wang, and Jiaying Liu. 2020. Towards Scale-Free Rain Streak Removal via Self-Supervised Fractal Band Learning. In AAAI. 12629--12636.
[62]
Youzhao Yang and Hong Lu. 2019. Single Image Deraining via Recurrent Hierarchy Enhancement Network. In ACM MM. 1814--1822.
[63]
Rajeev Yasarla, Vishwanath A. Sindagi, and Vishal M. Patel. 2020. Syn2Real Transfer Learning for Image Deraining Using Gaussian Processes. In IEEE CVPR. 2723--2733.
[64]
Desen Yuan. 2024. Balancing Easy and Hard Distortions: A Multi-Rate Knowledge Distillation Strategy for Blind Image Quality Assessment. In ICASSP. 8265--8269.
[65]
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. 2021. Multi-Stage Progressive Image Restoration. In IEEE CVPR.
[66]
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. 2021. Restormer: Efficient Transformer for High-Resolution Image Restoration. arXiv preprint arXiv:2111.09881 (2021).
[67]
He Zhang and Vishal M. Patel. 2018. Densely Connected Pyramid Dehazing Network. In IEEE CVPR. 3194--3203.
[68]
He Zhang and Vishal M. Patel. 2018. Density-Aware Single Image De-Raining Using a Multi-Stream Dense Network. In IEEE CVPR. 695--704.
[69]
He Zhang, Vishwanath Sindagi, and Vishal M. Patel. 2020. Image De-Raining Using a Conditional Generative Adversarial Network. IEEE TCSVT, Vol. 30, 11 (2020), 3943--3956.
[70]
Wenlong Zhang, Yihao Liu, Chao Dong, and Yu Qiao. 2019. RankSRGAN: Generative Adversarial Networks With Ranker for Image Super-Resolution. In IEEE ICCV. 3096--3105.
[71]
Hongyuan Zhu, Xi Peng, Joey Tianyi Zhou, Songfan Yang, Vijay Chanderasekh, Liyuan Li, and Joo-Hwee Lim. 2019. Singe Image Rain Removal with Unpaired Information: A Differentiable Programming Perspective. In AAAI. 9332--9339.
[72]
Honghe Zhu, Cong Wang, Yajie Zhang, Zhixun Su, and Guohui Zhao. 2020. Physical model guided deep image deraining. In ICME. 1--6.
Index Terms
- Progressive Local and Non-Local Interactive Networks with Deeply Discriminative Training for Image Deraining
Recommendations
Multi-level wavelet network based on CNN-Transformer hybrid attention for single image deraining
AbstractRemoving rain streaks from rainy images can improve the accuracy of computer vision applications such as object detection. In order to make full use of the frequency domain analysis characteristics of wavelet and combine the advantages of ...
Semi-supervised Single Image Deraining with Discrete Wavelet Transform
PRICAI 2021: Trends in Artificial IntelligenceAbstractIn recent years, single image deraining has received considerable research interests. Supervised learning is widely adopted for training dedicated deraining networks to achieve promising results on synthetic datasets, while limiting in handling ...
Non-local feature aggregation quaternion network for single image deraining
AbstractThe existing deraining methods are based on convolutional neural networks (CNN) learning the mapping relationship between rainy and clean images. However, the real-valued CNN processes the color images as three independent channels separately, ...
Comments
Information & Contributors
Information
Published In
October 2024
11719 pages
ISBN:9798400706868
DOI:10.1145/3664647
- General Chairs:
- Jianfei Cai,
- Mohan Kankanhalli,
- Balakrishnan Prabhakaran,
- Susanne Boll,
- Program Chairs:
- Ramanathan Subramanian,
- Liang Zheng,
- Vivek K. Singh,
- Pablo Cesar,
- Lexing Xie,
- Dong Xu
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 28 October 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Shenzhen Science and Technology Program
- Fundamental Research Funds for the Central Universities, Sun Yat-sen University
- National Natural Science Foundation of China
Conference
MM '24
Sponsor:
MM '24: The 32nd ACM International Conference on Multimedia
October 28 - November 1, 2024
Melbourne VIC, Australia
Acceptance Rates
MM '24 Paper Acceptance Rate 1,150 of 4,385 submissions, 26%;
Overall Acceptance Rate 2,145 of 8,556 submissions, 25%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 45Total Downloads
- Downloads (Last 12 months)45
- Downloads (Last 6 weeks)10
Reflects downloads up to 11 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in