Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content

Advertisement

Springer Nature Link
Log in

Event-Driven Heterogeneous Network for Video Deraining

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Restoring clear frames from rainy videos poses a significant challenge due to the swift motion of rain streaks. Traditional frame-based visual sensors, which record dense scene content synchronously, struggle to capture the fast-moving rain information. Conversely, the novel bio-inspired event camera, known for its high temporal resolution and low latency, effectively records the motion trajectories of rapidly falling rain through asynchronously generated sparse event sequences. In light of these attributes, we introduce a novel event-driven convolutional spiking network for video deraining. For video restoration, we employ a Convolutional Neural Network as the backbone, extracting dense features from video sequences to map rain-soaked frames to clear ones. To remove rain, we meticulously design a bio-inspired Spiking Neural Network that adapts to the sparse event sequences, capturing features of falling rain. We then establish a bimodal feature fusion module that combines dense convolutional features with sparse spiking features. This fusion aids the backbone in accurately pinpointing rain streaks across spatiotemporal dimensions. Thus, our diverse network extracts and collaborates information from both events and videos, enhancing deraining performance. Experiments conducted on synthetic and real-world datasets prove that our network markedly surpasses existing video deraining techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  • Ardizzone, L., Kruse, J., Rother, C., & Köthe, U. (2019). Analyzing inverse problems with invertible neural networks. In ICLR.

  • Barnum, P. C., Narasimhan, S., & Kanade, T. (2010). Analysis of rain and snow in frequency space. International Journal of Computer Vision, 86, 256–274.

    Article  Google Scholar 

  • Bossu, J., Hautiere, N., & Tarel, J.-P. (2011). Rain or snow detection in image sequences through use of a histogram of orientation of streaks. International Journal of Computer Vision, 93, 348–367.

    Article  Google Scholar 

  • Bouvier, M., Valentian, A., Mesquida, T., Rummens, F., Reyboz, M., Vianello, E., & Beigne, E. (2019). Spiking neural networks hardware implementations and challenges: A survey. ACM Journal on Emerging Technologies in Computing Systems, 15(2), 1–35.

    Article  Google Scholar 

  • Brandli, C., Berner, R., Yang, M., Liu, S.-C., & Delbruck, T. (2014). A 240\(\times \) 180 130 db 3 \(\mu \)s latency global shutter spatiotemporal vision sensor. IEEE Journal of Solid-State Circuits, 49(10), 2333–2341.

    Article  Google Scholar 

  • Chen, J., Tan, C. -H., Hou, J., Chau, L. -P., & Li, H. (2018). Robust video content alignment and compensation for rain removal in a CNN framework. In CVPR.

  • Chen, X., Kundu, K., Zhang, Z., Ma, H., Fidler, S., & Urtasun, R. (2016). Monocular 3d object detection for autonomous driving. In CVPR.

  • Chen, Y. -L., & Hsu, C. -T. (2013). A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In ICCV.

  • Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In CVPR.

  • Dosovitskiy, A., Beyer, L., Kolesnikov, Al. et al. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR.

  • Fang, W., Yu, Z., Chen, Y., Huang, T., Masquelier, T., & Tian, Y. (2021a). Deep residual learning in spiking neural networks. In NeurIPS.

  • Feng, R., Li, C., Chen, H., Li, Shuai, L., Chen, C., & Gu, J. (2021b). Removing diffraction image artifacts in under-display camera via dynamic skip connection network. In CVPR.

  • Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., & Paisley, J. (2017). Removing rain from single images via a deep detail network. In CVPR.

  • Gallego, G., Delbrück, T., Orchard, G., et al. (2020). Event-based vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1), 154–180.

    Article  Google Scholar 

  • Garg, K., & Nayar, S. K. (2004). Detection and removal of rain from videos. In CVPR.

  • Garg, K., & Nayar, S. K. (2005). When does a camera see rain?. In ICCV.

  • Garg, K., & Nayar, S. K. (2006). Photorealistic rendering of rain streaks. ACM Transactions on Graphics, 25(3), 996–1002.

    Article  Google Scholar 

  • Garg, K., & Nayar, S. K. (2007). Vision and rain. International Journal of Computer Vision, 75, 3–27.

    Article  Google Scholar 

  • Gehrig, M., Millhäusler, M., Gehrig, D., & Scaramuzza, D. (2021a). E-raft: Dense optical flow from event cameras. In 3DV.

  • Gehrig, M., Millhäusler, M., Gehrig, D., & Scaramuzza, D. (2021b). E-RAFT: Dense optical flow from event cameras. arXiv:2108.10552 [cs]

  • Hagenaars, J. J., & Paredes-Vallés, F. (2021). Self-supervised learning of event-based optical flow with spiking neural networks. In NeurIPS.

  • Han, B., Srinivasan, G., & Roy, K. (2020). RMP-SNN: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. In CVPR.

  • Han, J., Yang, Y., Zhou, C., Xu, C., & Shi, B. (2021). EvIntSR-Net: Event guided multiple latent frames reconstruction and super-resolution. In ICCV.

  • Hu, Y., Liu, S. -C., & Delbruck, T. (2021). V2E: From video frames to realistic DVS events. In CVPR.

  • Jiang, H., Sun, D., Jampani, V., Yang, M. -H., Learned-Miller, E., & Kautz, J. (2018). Super slomo: High quality estimation of multiple intermediate frames for video interpolation. In CVPR.

  • Jiang, T. -X., Huang, T. -Z., Zhao, X. -L., Deng, L. -J., & Wang, Y. (2017a). A novel tensor-based video rain streaks removal approach via utilizing discriminatively intrinsic priors. In CVPR.

  • Jiang, T. -X., Huang, T. -Z., Zhao, X. -L., Deng, L. -J., & Wang, Y. (2017b). A novel tensor-based video rain streaks removal approach via utilizing discriminatively intrinsic priors. In CVPR.

  • Kim, J., Bae, J., Park, G., Zhang, D., & Kim, Y. M. (2021). N-imagenet: Towards robust, fine-grained object recognition with event cameras. In ICCV.

  • Kingma, D. P., & Ba, J. (2017). Adam: A method for stochastic optimization. NeurIPS.

  • Lee, C., Kosta, A. K., Zhu, A. Z., Chaney, K., Daniilidis, K., & Roy, K. (2020). Spike-FlowNet: Event-based optical flow estimation with energy-efficient hybrid neural networks. In ECCV.

  • Lee, C., Srinivasan, G., Panda, P., & Roy, K. (2018). Deep spiking convolutional neural network trained with unsupervised spike-timing-dependent plasticity. IEEE Transactions on Cognitive and Developmental Systems, 11(3), 384–394.

    Article  Google Scholar 

  • Li, C., Anwar, S., Hou, J., Cong, R., Guo, C., & Ren, W. (2021a). Underwater image enhancement via medium transmission-guided multi-color space embedding. IEEE Transactions on Image Processing, 30, 4985–5000.

  • Li, C., Guo, C., Guo, J., Han, P., Huazhu, F., & Cong, R. (2019). PDR-Net: Perception-inspired single image dehazing network with refinement. IEEE Transactions on Multimedia, 22(3), 704–716.

    Article  Google Scholar 

  • Li, C., Guo, C., Han, L., Jiang, J., Cheng, M.-M., Jinwei, G., & Loy, C. C. (2021b). Low-light image and video enhancement using deep learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12), 9396–9416.

  • Li, C., Guo, C., & Loy, C. C. (2021c). Learning to enhance low-light image via zero-reference deep curve estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8), 4225–4238.

  • Li, C., Guo, C., Ren, W., Cong, R., Hou, J., Kwong, S., & Tao, D. (2019). An underwater image enhancement benchmark dataset and beyond. IEEE Transactions on Image Processing, 29, 4376–4389.

    Article  Google Scholar 

  • Li, M., Cao, X., Zhao, Q., Zhang, L., & Meng, D. (2021d). Online rain/snow removal from surveillance videos. IEEE Transactions on Image Processing, 30, 2029–2044.

  • Li, M., Xie, Q., Zhao, Q., Wei, W., Gu, S., Tao, J., & Meng, D. (2018). Video rain streak removal by multiscale convolutional sparse coding. In CVPR.

  • Li, S., Ren, W., Wang, F., Araujo, I. B., Tokuda, E. K., Junior, R. H., Cesar-Jr, R. M., Wang, Z., & Cao, X. (2021e). A comprehensive benchmark analysis of single image deraining: Current challenges and future perspectives. International Journal of Computer Vision, 129, 1301–1322.

  • Li, Y., Tan, R. T., Guo, X., Lu, J., & Brown, M. S. (2016). Rain streak removal using layer priors. In CVPR.

  • Liu, J., Yang, W., Yang, S., & Guo, Z. (2018). Erase or fill? Deep joint recurrent rain removal and reconstruction in videos. In CVPR.

  • Liu, J., Yang, W., Yang, S., & Guo, Z. (2019). D3R-Net: Dynamic Routing residue recurrent network for video rain removal. IEEE Transactions on Image Processing, 28(2), 699–712.

    Article  MathSciNet  Google Scholar 

  • Luo, Y., Xu, Y., & Ji, H. (2015). Removing rain from a single image via discriminative sparse coding. In ICCV.

  • Mostafavi, M., Nam, Y., Choi, J., & Yoon, K.-J. (2022). E2SRI: Learning to super-resolve intensity images from events. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10), 6890–6909.

    Article  Google Scholar 

  • Pan, M., Liu, Z., Liu, Y., Liu, R., & Fan, X. (2021). Triple-level model inferred collaborative network architecture for video deraining. IEEE Transactions on Image Processing, 31, 239–250.

    Google Scholar 

  • Qian, R., Tan, R. T., Yang, W., Su, J., & Liu, J. (2018). Attentive generative adversarial network for raindrop removal from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2482–2491).

  • Rebecq, H., Ranftl, R., Koltun, V., & Scaramuzza, D. (2019). Events-to-video: Bringing modern computer vision to event cameras. In CVPR.

  • Ren, D., Zuo, W., Hu, Q., Zhu, P., & Meng, D. (2019). Progressive image deraining networks: A better and simpler baseline. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 3937–3946).

  • Ren, W., Pan, J., Zhang, H., Cao, X., & Yang, M.-H. (2020). Single image dehazing via multi-scale convolutional neural networks with holistic edges. International Journal of Computer Vision, 128, 240–259.

    Article  Google Scholar 

  • Ren, W., Tian, J., Han, Z., Chan, A., & Tang, Y. (2017). Video desnowing and deraining based on matrix decomposition. In CVPR.

  • Santhaseelan, V., & Asari, V. K. (2015). Utilizing local phase information to remove rain from video. International Journal of Computer Vision, 112, 71–89.

    Article  Google Scholar 

  • Scheerlinck, C., Rebecq, H., Gehrig, D., Barnes, N., Mahony, R. E., & Scaramuzza, D. (2020). Fast image reconstruction with an event camera. In WACV.

  • Shi, X., Chen, Z., Wang, H., Yeung, D. -Y., Wong, W. -K., & Woo, W. -C. (2015). Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In NeurIPS.

  • Sun, S., Ren, W., Li, J., Zhang, K., Liang, M., & Cao, X. (2023). Event-aware video deraining via multi-patch progressive learning. IEEE Transactions on Image Processing, 32, 3040–3053.

    Article  Google Scholar 

  • Tavanaei, A., Ghodrati, M., Kheradpisheh, S. R., Masquelier, T., & Maida, A. S. (2019). Deep learning in spiking neural networks. Neural Networks, 111, 47–63.

    Article  Google Scholar 

  • Valanarasu, J. M. J., Yasarla, R., & Patel, V. M. (2022). Transweather: Transformer-based restoration of images degraded by adverse weather conditions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2353–2363).

  • Wang, C., Wu, Y., Su, Z., & Chen, J. (2020a). Joint self-attention and scale-aggregation for self-calibrated deraining network. In ACM Multimedia.

  • Wang, C., Xing, X., Wu, Y., Su, Z., & Chen, J. (2020b). DCSFN: Deep cross-scale fusion network for single image rain removal. In ACM Multimedia.

  • Wang, G., Sun, C., & Sowmya, A. (2021). Context-enhanced representation learning for single image deraining. International Journal of Computer Vision, 129, 1650–1674.

    Article  Google Scholar 

  • Wang, H., Xie, Q., Yichen, W., Zhao, Q., & Meng, D. (2020). Single image rain streaks removal: A review and an exploration. International Journal of Machine Learning and Cybernetics, 11, 853–872.

    Article  Google Scholar 

  • Wang, H., Xie, Q., Zhao, Q., & Meng, D. (2020d). A model-driven deep neural network for single image rain removal. In CVPR.

  • Wang, T., Zhang, K., Shao, Z., Luo, W., Stenger, B., Lu, T., Kim, T. -K., Liu, W., & Li, H. (2023). Gridformer: Residual dense transformer with grid structure for image restoration in adverse weather conditions. arXiv preprint arXiv:2305.17863.

  • Wang, X., Girshick, R., Gupta, A., & He, K. (2018). Non-local neural networks. In CVPR.

  • Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600–612.

    Article  Google Scholar 

  • Wang, Z. W., Jiang, W., & He, K. (2019). Event-driven video frame synthesis. In ICCVW.

  • Wei, W., Yi, L., Xie, Q., Zhao, Q., Meng, D., & Xu, Z. (2017). Should we encode rain streaks in video as deterministic or stochastic?. In ICCV.

  • Wu, Z., Liu, Z., Lin, J., Lin, Y., & Han, S. (2020). Lite transformer with long-short range attention. In ICLR.

  • Xin, J., Wang, N., Jiang, X., Li, J., & Gao, X. (2023). Advanced binary neural network for single image super resolution. International Journal of Computer Vision, 131(7), 1808–1824.

    Article  Google Scholar 

  • Xu, F., Yu, L., & Wang, B. (2021). Motion deblurring with real events. In ICCV.

  • Xueyang, F., Qi, Q., Zha, Z.-J., Ding, X., Feng, W., & Paisley, J. (2021). Successive graph convolutional network for image de-raining. International Journal of Computer Vision, 129, 1691–1711.

    Article  Google Scholar 

  • Yan, W., Tan, R. T., Yang, W., & Dai, D. (2021). Self-aligned video deraining with transmission-depth consistency. In CVPR.

  • Yang, W., Liu, J., & Feng, J. (2019a). Frame-consistent recurrent video deraining with dual-level flow. In CVPR.

  • Yang, W., Liu, J., & Feng, J. (2019b). Frame-consistent recurrent video deraining with dual-level flow. In CVPR.

  • Yang, W., Tan, R. T., Feng, J., Liu, J., Guo, Z., & Yan, S. (2017). Deep joint rain detection and removal from a single image. In CVPR.

  • Yang, W., Tan, R. T., Wang, S., & Liu, J. (2020). Self-learning video rain streak removal: When cyclic consistency meets temporal correspondence. In CVPR.

  • Yang, W., Tan, R. T., Wang, S., Fang, Y., & Liu, J. (2020). Single image deraining: From model-based to data-driven and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(11), 4059–4077.

    Article  Google Scholar 

  • Yao, M., Gao, H., Zhao, G., Wang, D., Lin, Y., Yang, Z., & Li, G. (2021). Temporal-wise attention spiking neural networks for event streams classification. In ICCV.

  • Ye, C. (2019). Learning of dense optical flow, motion and depth, from sparse event cameras. PhD thesis, University of Maryland, College Park.

  • Ye, C., Mitrokhin, A., Fermüller, C., Yorke, J. A., & Aloimonos, Y. (2018). Unsupervised learning of dense optical flow, depth and egomotion from sparse event data. arXiv preprint arXiv:1809.08625

  • Yue, Z., Xie, J., Zhao, Q., & Meng, D. (2021). Semi-supervised video deraining with dynamical rain generator. In CVPR.

  • Zhang, H., & Patel, V. M. (2017). Convolutional sparse and low-rank coding-based rain streak removal. In WACV.

  • Zhang, H, & Patel, V. M. (2018a). Density-aware single image de-raining using a multi-stream dense network. In CVPR.

  • Zhang, K., Li, D., Luo, W., & Ren, W. (2021). Dual attention-in-attention model for joint rain streak and raindrop removal. IEEE Transactions on Image Processing, 30, 7608–7619.

    Article  Google Scholar 

  • Zhang, K., Li, D., Luo, W., Ren, W., & Liu, W. (2022). Enhanced spatio-temporal interaction learning for video deraining: A faster and better framework. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1), 1287–1293.

    Article  Google Scholar 

  • Zhang, K., Luo, W., Yanjiang, Y., Ren, W., Zhao, F., Li, C., Ma, L., Liu, W., & Li, H. (2022). Beyond monocular deraining: Parallel stereo deraining network via semantic prior. International Journal of Computer Vision, 130(7), 1754–1769.

    Article  Google Scholar 

  • Zhang, K., Ren, W., Luo, W., Lai, W.-S., Stenger, B., Yang, M.-H., & Li, H. (2022). Deep image deblurring: A survey. International Journal of Computer Vision, 130(9), 2103–2130.

    Article  Google Scholar 

  • Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018b). The unreasonable effectiveness of deep features as a perceptual metric. In CVPR.

  • Zhou, S., Li, C., & Change Loy, C. (2022) Lednet: Joint low-light enhancement and deblurring in the dark. In ECCV.

  • Zhu, A. Z., Yuan, L., Chaney, K., & Daniilidis, K. (2018). EV-FlowNet: Self-supervised optical flow estimation for event-based cameras. Robotics: Science and Systems XIV.

  • Zhu, L., Fu, C. -W., Lischinski, D., & Heng, P. -A. (2017). Joint bi-layer optimization for single-image rain streak removal. In ICCV.

  • Zhu, L., Li, J., Wang, X., Huang, T., & Tian, Y. (2021). NeuSpike-Net: High speed video reconstruction via bio-inspired neuromorphic cameras. In ICCV.

  • Zou, D. (2020). Learning event-driven video deblurring and interpolation. In ECCV.

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (NSFC) under Grant 62225207 and Grant 62276243.

Author information

Authors and Affiliations

  1. School of Information Science and Technology and MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China, Hefei, 230026, China

    Xueyang Fu, Chengzhi Cao, Senyan Xu, Fanrui Zhang, Kunyu Wang & Zheng-Jun Zha

Authors
  1. Xueyang Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chengzhi Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Senyan Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fanrui Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kunyu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Zheng-Jun Zha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zheng-Jun Zha.

Additional information

Communicated by Yasuyuki Matsushita.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fu, X., Cao, C., Xu, S. et al. Event-Driven Heterogeneous Network for Video Deraining. Int J Comput Vis 132, 5841–5861 (2024). https://doi.org/10.1007/s11263-024-02148-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-024-02148-x

Keywords

Search

Navigation