Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content

Advertisement

Springer Nature Link
Log in

Frequency domain-enhanced transformer for single image deraining

  • Research
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Since Transformers show a strong capability of building long-range dependencies, the relevant methods are extensively employed for image deraining tasks. However, the intrinsic limitations of Transformers, including costly computational complexity and insufficient ability to capture high-frequency components of the image, hinder the the utilization of Transformers in high-resolution images and lead to the unsatisfactory recovery of local edges and textures. To overcome these limitations, we propose an simple but effective Frequency Domain Enhanced Transformer (FDEFormer) for the image deraining. Firstly, drawing inspiration from the convolution theorem, we devise an efficient approach called frequency domain enhanced multi-head self-attention. The proposed approach replaces traditional matrix multiplication with element-wise product operations in the frequency domain, leading to a substantial reduction in computational complexity. Secondly, the existing spatial domain Transformers-based methods only focus on low-frequency features but pay less attention to high-frequency components, which can adversely affect the quality of the reconstructed images. Therefore, to integrate the content of different frequency levels, we propose a dual domain-complemented feed-forward network. Besides, we further present an attention feature fusion module to facilitate a more effective fusion of features across different layers. Extensive experiments on several datasets demonstrate that our FDEFormer performs favorably against state-of-the-art methods while taking acceptable computational costs. The source code and pre-trained models are available at https://github.com/bobozy1999/FDEFormer.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Sun, H., Ang, M.H., Rus, D.: A convolutional network for joint deraining and dehazing from a single image for autonomous driving in rain. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 962–969 (2019). https://doi.org/10.1109/IROS40897.2019.8967644

  2. Wang, W., Zhang, J., Zhai, W., Cao, Y., Tao, D.: Robust object detection via adversarial novel style exploration. IEEE Trans. Image Process. 31, 1949–1962 (2022). https://doi.org/10.1109/TIP.2022.3146017

    Article  Google Scholar 

  3. Yang, W., Tan, R.T., Feng, J., Liu, J., Guo, Z., Yan, S.: Deep joint rain detection and removal from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1357–1366 (2017). https://doi.org/10.1109/CVPR.2017.183

  4. Xiao, J., Fu, X., Liu, A., Wu, F., Zha, Z.-J.: Image de-raining transformer. IEEE Trans. Pattern Anal. Mach. Intell., 1–18 (2022). https://doi.org/10.1109/TPAMI.2022.3183612

  5. Deng, L., Huang, T., Zhao, X., Jiang, T.: A directional global sparse model for single image rain removal. Appl. Math. Model. 59, 662–679 (2018). https://doi.org/10.1016/j.apm.2018.03.001

    Article  MathSciNet  Google Scholar 

  6. Xu, J., Zhao, W., Liu, P., Tang, X.: Removing rain and snow in a single image using guided filter. In: 2012 IEEE International Conference on Computer Science and Automation Engineering (CSAE), vol. 2, pp. 304–307 (2012). https://doi.org/10.1109/CSAE.2012.6272780

  7. Zheng, X., Liao, Y., Guo, W., Fu, X., Ding, X.: Single-image-based rain and snow removal using multi-guided filter. In: Neural Information Processing: 20th International Conference, ICONIP 2013, Daegu, Korea, November 3–7, 2013. Proceedings, Part III 20, pp. 258–265 (2013). https://doi.org/10.1007/978-3-642-42051-1-33

  8. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.-H.: Restormer: efficient transformer for high-resolution image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5728–5739 (2022). https://doi.org/10.1109/CVPR52688.2022.00564

  9. Wan, Y., Shao, M., Bao, Z., Cheng, Y.: Global-local transformer for single-image rain removal. Pattern Anal. Appl. 1–12 (2023). https://doi.org/10.1007/s10044-023-01184-6

  10. Wang, Z., Cun, X., Bao, J., Zhou, W., Liu, J., Li, H.: Uformer: a general u-shaped transformer for image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17683–17693 (2022). https://doi.org/10.1109/CVPR52688.2022.01716

  11. Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., Timofte, R.: Swinir: image restoration using swin transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1833–1844 (2021). https://doi.org/10.1109/ICCVW54120.2021.00210

  12. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Adv. Neural Inf. Process. Syst. 30 (2017)

  13. Blackledge, J.M.: Digital Image Processing: Mathematical and Computational Methods, pp. 44–45. Elsevier, England (2005)

  14. Wang, Y., Liu, S., Chen, C., Zeng, B.: A hierarchical approach for rain or snow removing in a single color image. IEEE Trans. Image Process. 26(8), 3936–3950 (2017). https://doi.org/10.1109/TIP.2017.2708502

    Article  MathSciNet  Google Scholar 

  15. Chen, D., Chen, C., Kang, L.: Visual depth guided color image rain streaks removal using sparse coding. IEEE Trans. Circuits Syst. Video Technol. 24(8), 1430–1455 (2014). https://doi.org/10.1109/TCSVT.2014.2308627

    Article  Google Scholar 

  16. Zhu, L., Fu, C., Lischinski, D., Heng, P.: Joint bi-layer optimization for single-image rain streak removal. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2526–2534 (2017). https://doi.org/10.1109/ICCV.2017.276

  17. Jiang, T., Huang, T., Zhao, X., Deng, L., Wang, Y.: Fastderain: a novel video rain streak removal method using directional gradient priors. IEEE Trans. Image Process. 28(4), 2089–2102 (2018). https://doi.org/10.1109/TIP.2018.2880512

    Article  MathSciNet  Google Scholar 

  18. Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., Paisley, J.: Removing rain from single images via a deep detail network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3855–3863 (2017). https://doi.org/10.1109/CVPR.2017.186

  19. Chen, X., Pan, J., Jiang, K., Li, Y., Huang, Y., Kong, C., Dai, L., Fan, Z.: Unpaired deep image deraining using dual contrastive learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2017–2026 (2022). https://doi.org/10.1109/CVPR52688.2022.00206

  20. Du, Y., Xu, J., Zhen, X., Cheng, M.-M., Shao, L.: Conditional variational image deraining. IEEE Trans. Image Process. 29, 6288–6301 (2020). https://doi.org/10.1109/TIP.2020.2990606

    Article  Google Scholar 

  21. Li, B., Liu, X., Hu, P., Wu, Z., Lv, J., Peng, X.: All-in-one image restoration for unknown corruption. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17452–17462 (2022). https://doi.org/10.1109/CVPR52688.2022.01693

  22. Wan, Y., Shao, M., Cheng, Y., Liu, Y., Bao, Z., Meng, D.: Restoring images captured in arbitrary hybrid adverse weather conditions in one go. arXiv preprint arXiv:2305.09996 (2023)

  23. Wan, Y., Cheng, Y., Shao, M., Gonzàlez, J.: Image rain removal and illumination enhancement done in one go. Knowl.-Based Syst. 252, 109244 (2022). https://doi.org/10.1016/j.knosys.2022.109244

    Article  Google Scholar 

  24. Shao, M., Qiao, Y., Meng, D., Zuo, W.: Uncertainty-guided hierarchical frequency domain transformer for image restoration. Knowl.-Based Syst. 263, 110306 (2023). https://doi.org/10.1016/j.knosys.2023.110306

    Article  Google Scholar 

  25. Yang, H., Zhou, D., Li, M., Zhao, Q.: A two-stage network with wavelet transformation for single-image deraining. Vis. Comput., 1pp. –17 (2022). https://doi.org/10.1007/s00371-022-02533-yv

  26. Chen, M., Wang, P., Shang, D., Wang, P.: Cycle-attention-derain: unsupervised rain removal with cyclegan. Vis. Comput., 1–13 (2023). https://doi.org/10.1007/s00371-023-02947-2

  27. Luo, Y., Wu, M., Huang, Q., Zhu, J., Ling, J., Sheng, B.: Joint feedback and recurrent deraining network with ensemble learning. Vis. Comput. 38(9–10), 3109–3119 (2022). https://doi.org/10.1007/s00371-022-02567-2

    Article  Google Scholar 

  28. Li, R., Cheong, L.-F., Tan, R.T.: Heavy rain image restoration: integrating physics model and conditional adversarial learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1633–1642 (2019). https://doi.org/10.1109/CVPR.2019.00173

  29. Jiang, K., Wang, Z., Yi, P., Chen, C., Huang, B., Luo, Y., Ma, J., Jiang, J.: Multi-scale progressive fusion network for single image deraining. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8346–8355 (2020). https://doi.org/10.1109/CVPR42600.2020.00837

  30. Huang, H., Yu, A., He, R.: Memory oriented transfer learning for semi-supervised image deraining. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7732–7741 (2021). https://doi.org/10.1109/CVPR46437.2021.00764

  31. Yasarla, R., Sindagi, V.A., Patel, V.M.: Syn2real transfer learning for image deraining using gaussian processes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2726–2736 (2020). https://doi.org/10.1109/CVPR42600.2020.00280

  32. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.-H., Shao, L.: Multi-stage progressive image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14821–14831 (2021). https://doi.org/10.1109/CVPR46437.2021.01458

  33. Lin, X., Ma, L., Sheng, B., Wang, Z.-J., Chen, W.: Utilizing two-phase processing with fbls for single image deraining. IEEE Trans. Multimedia 23, 664–676 (2020). https://doi.org/10.1109/TMM.2020.2987703

    Article  Google Scholar 

  34. Qian, R., Tan, R.T., Yang, W., Su, J., Liu, J.: Attentive generative adversarial network for raindrop removal from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2482–2491 (2018). https://doi.org/10.1109/CVPR.2018.00263

  35. Peng, J., Xu, Y., Chen, T., Huang, Y.: Single-image raindrop removal using concurrent channel-spatial attention and long-short skip connections. Pattern Recognit. Lett. 131, 121–127 (2020). https://doi.org/10.1016/j.patrec.2019.12.012

    Article  Google Scholar 

  36. Shao, M., Li, L., Meng, D., Zuo, W.: Uncertainty guided multi-scale attention network for raindrop removal from a single image. IEEE Trans. Image Process. 30, 4828–4839 (2021). https://doi.org/10.1109/TIP.2021.3076283

    Article  Google Scholar 

  37. Lin, X., Sun, S., Huang, W., Sheng, B., Li, P., Feng, D.D.: Eapt: efficient attention pyramid transformer for image processing. IEEE Trans. Multimedia (2021)

  38. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022 (2021). https://doi.org/10.1109/ICCV48922.2021.00986

  39. Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., Gao, W.: Pre-trained image processing transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12299–12310 (2021). https://doi.org/10.1109/CVPR46437.2021.01212

  40. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16\(\times \)16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

  41. Li, Y., Fan, Y., Xiang, X., Demandolx, D., Ranjan, R., Timofte, R., Van Gool, L.: Efficient and explicit modelling of image hierarchies for image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18278–18289 (2023). https://doi.org/10.1109/CVPR52729.2023.01753

  42. Chen, H., Gu, J., Liu, Y., Magid, S.A., Dong, C., Wang, Q., Pfister, H., Zhu, L.: Masked image training for generalizable deep image denoising. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1692–1703 (2023). https://doi.org/10.1109/CVPR52729.2023.00169

  43. Kong, L., Dong, J., Ge, J., Li, M., Pan, J.: Efficient frequency domain-based transformers for high-quality image deblurring. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5886–5895 (2023). https://doi.org/10.1109/CVPR52729.2023.00570

  44. Kamgar-Parsi, B., Rosenfeld, A.: Optimally isotropic Laplacian operator. IEEE Trans. Image Process. 8(10), 1467–1472 (1999). https://doi.org/10.1109/83.791975

    Article  MathSciNet  Google Scholar 

  45. Cao, J., Li, Y., Sun, M., Chen, Y., Lischinski, D., Cohen-Or, D., Chen, B., Tu, C.: Do-conv: depthwise over-parameterized convolutional layer. IEEE Trans. Image Process. 31, 3726–3736 (2022). https://doi.org/10.1109/TIP.2022.3175432

    Article  Google Scholar 

  46. Loshchilov, I., Hutter, F.: Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 (2016)

  47. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.69803 (2014)

  48. Luo, Y., Xu, Y., Ji, H.: Removing rain from a single image via discriminative sparse coding. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3397–3405 (2015). https://doi.org/10.1109/ICCV.2015.388

  49. Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017). https://doi.org/10.1109/CVPR.2017.632

  50. Liu, X., Suganuma, M., Sun, Z., Okatani, T.: Dual residual networks leveraging the potential of paired operations for image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7007–7016 (2019). https://doi.org/10.1109/CVPR.2019.00717

  51. Quan, Y., Deng, S., Chen, Y., Ji, H.: Deep learning for seeing through window with raindrops. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2463–2471 (2019). https://doi.org/10.1109/ICCV.2019.00255

  52. Tu, Z., Talebi, H., Zhang, H., Yang, F., Milanfar, P., Bovik, A., Li, Y.: Maxim: Multi-axis mlp for image processing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5769–5780 (2022). https://doi.org/10.1109/CVPR52688.2022.00568

  53. Purohit, K., Suin, M., Rajagopalan, A., Boddeti, V.N.: Spatially-adaptive image restoration using distortion-guided networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2309–2319 (2021). https://doi.org/10.1109/ICCV48922.2021.00231

  54. Zhang, H., Sindagi, V., Patel, V.M.: Image de-raining using a conditional generative adversarial network. IEEE Trans. Circuits Syst. Video Technol. 30(11), 3943–3956 (2019). https://doi.org/10.1109/TCSVT.2019.2920407

    Article  Google Scholar 

  55. Zhang, H., Patel, V.M.: Density-aware single image de-raining using a multi-stream dense network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 695–704 (2018). https://doi.org/10.1109/CVPR.2018.00079

  56. Quan, R., Yu, X., Liang, Y., Yang, Y.: Removing raindrops and rain streaks in one go. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9147–9156 (2021). https://doi.org/10.1109/CVPR46437.2021.00903

  57. Huynh-Thu, Q., Ghanbari, M.: Scope of validity of psnr in image/video quality assessment. Electron. Lett. 44(13), 800–801 (2008). https://doi.org/10.1049/el:20080522

    Article  Google Scholar 

  58. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004). https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  59. Fu, X., Xiao, J., Zhu, Y., Liu, A., Wu, F., Zha, Z.-J.: Continual image deraining with hypergraph convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. (2023). https://doi.org/10.1109/TPAMI.2023.3241756

    Article  Google Scholar 

  60. Fu, X., Huang, J., Ding, X., Liao, Y., Paisley, J.: Clearing the skies: a deep network architecture for single-image rain removal. IEEE Trans. Image Process. 26(6), 2944–2956 (2017). https://doi.org/10.1109/TIP.2017.2691802

    Article  MathSciNet  Google Scholar 

  61. Wei, W., Meng, D., Zhao, Q., Xu, Z., Wu, Y.: Semi-supervised transfer learning for image rain removal. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3877–3886 (2019). https://doi.org/10.1109/CVPR.2019.00400

  62. Yasarla, R., Patel, V.M.: Uncertainty guided multi-scale residual learning-using a cycle spinning cnn for single image de-raining. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8405–8414 (2019). https://doi.org/10.1109/CVPR.2019.00860

  63. Li, X., Wu, J., Lin, Z., Liu, H., Zha, H.: Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 254–269 (2018). https://doi.org/10.1007/978-3-030-01234-2-16

  64. Ren, D., Zuo, W., Hu, Q., Zhu, P., Meng, D.: Progressive image deraining networks: a better and simpler baseline. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3937–3946 (2019). https://doi.org/10.1109/CVPR.2019.00406

  65. Chen, L., Lu, X., Zhang, J., Chu, X., Chen, C.: Hinet: half instance normalization network for image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 182–192 (2021). https://doi.org/10.1109/CVPRW53098.2021.00027

Download references

Acknowledgements

The authors are very indebted to the anonymous referees for their critical comments and suggestions for the improvement of this paper. This work was supported by National Key Research and development Program of China (2021YFA1000102), and in part by the grants from the National Natural Science Foundation of China (Nos. 62376285, 62272375, 61673396), Natural Science Foundation of Shandong Province, China (No. ZR2022MF260).

Author information

Authors and Affiliations

  1. National Science Digital Industry College, Quanzhou Vocational and Technical University, Jinjiang, 362000, Fujian, China

    Mingwen Shao

  2. College of Computer Science and Technology, China University of Petroleum (East China), Qingdao, 266000, Shandong, China

    Mingwen Shao, Zhiyuan Bao, Weihan Liu, Yuanjian Qiao & Yecong Wan

Authors
  1. Mingwen Shao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhiyuan Bao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Weihan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yuanjian Qiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yecong Wan
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors did not receive support from any organization for the submitted work. The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request. All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by Weihan Liu, Yuanjian Qiao and Yecong Wan. The first draft of the manuscript was written by Mingwen Shao and Zhiyuan Bao and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Mingwen Shao.

Ethics declarations

Conflict of interest

All authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shao, M., Bao, Z., Liu, W. et al. Frequency domain-enhanced transformer for single image deraining. Vis Comput 40, 6723–6738 (2024). https://doi.org/10.1007/s00371-023-03252-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-03252-8

Keywords

Search

Navigation