Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

Fast simultaneous image super-resolution and motion deblurring with decoupled cooperative learning

  • Special Issue Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

In recent years, deep convolutional neural networks (CNNs) have been widely applied to handle low-level vision problems. However, most existing CNN-based approaches can either handle single degeneration each time or treat them jointly through feature entangling, thus likely leading to poor performance when the actual degradation is inconsistent with hypothetical degradation condition. Furthermore, feature coupling will bring a large amount of computation, which may make the methods impractical to real-time mobile scenarios. In order to address these problems, we propose a deep decoupled cooperative learning model which can not only develop the corresponding recover network to deal with each degradation, but also flexibly handle multiple degradations at the same time. Thus, our approach can achieve disentangling and synthesizing single image super-resolution and motion deblurring, which has high practicability. We evaluate the proposed approach on various benchmark datasets, covering both natural images and synthetic images. The results demonstrate its superiority, compared to the state-of-the-art, where image SR and motion deblurring can be accomplished effectively concurrently. The source code of the work is available at https://github.com/hengliusky/Cooperative-Learning-Deblur-SR.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Ding, G., Guo, Y., Chen, K., Chu, C., Han, J., Dai, Q.: Decode: deep confidence network for robust image classification. IEEE Trans. Image Process. 28(8), 3752–3765 (2019)

    Article  MathSciNet  Google Scholar 

  2. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Proceedings of European Conference on Computer Vision, pp. 184–199 (2014)

  3. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)

    Article  Google Scholar 

  4. Dong, W., Zhang, L., Shi, G., Li, X.: Nonlocally centralized sparse representation for image restoration. IEEE Trans. Image Process. 22(4), 1620–1630 (2013)

    Article  MathSciNet  Google Scholar 

  5. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)

  6. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: Convolutional architecture for fast feature embedding. In: Proceedings of ACM International Conference on Multimedia, pp. 675–678 (2014)

  7. Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)

  8. Lai, W.S., Huang, J.B., Hu, Z., Ahuja, N., Yang, M.H.: A comparative study for single image blind deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1701–1709 (2016)

  9. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A.P., Tejani, A., Totz, J., Wang, Z., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4681–4690 (2017)

  10. Li, L., Pan, J., Lai, W.S., Gao, C., Sang, N., Yang, M.H.: Learning a discriminative prior for blind image deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6616–6625 (2018)

  11. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 136–144 (2017)

  12. Liu, H., Dai, L., Hou, S., Han, J., Liu, H.: Are mid-air dynamic gestures applicable to user identification? Pattern Recognit. Lett. 117, 179–185 (2019)

    Article  Google Scholar 

  13. Liu, H., Fu, Z., Han, J., Shao, L., Hou, S., Chu, Y.: Single image super-resolution using multi-scale deep encoder-decoder with phase congruency edge map guidance. Inf. Sci. 473, 44–58 (2019)

    Article  MathSciNet  Google Scholar 

  14. Luan, S., Chen, C., Zhang, B., Han, J., Liu, J.: Gabor convolutional networks. IEEE Trans. Image Process. 27(9), 4357–4366 (2018)

    Article  MathSciNet  Google Scholar 

  15. Nah, S., Kim, T.H., Lee, K.M.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3883–3891 (2017)

  16. Patrick, H.: Super-resolution on satellite imagery using deep learning part 1. The DownLinQ (2016)

  17. Qiao, T., Ren, J., Wang, Z., Zabalza, J., Sun, M., Zhao, H., Li, S., Benediktsson, J.A., Dai, Q., Marshall, S.: Effective denoising and classification of hyperspectral images using curvelet transform and singular spectrum analysis. IEEE Trans. Geosci. Remote Sens. 55(1), 119–133 (2016)

    Article  Google Scholar 

  18. Schuler, C., Hirsch, M., Harmeling, S., Scholkopf, B.: Learning to deblur. IEEE Trans. Pattern Anal. Mach. Intell. 38(7), 1439–1451 (2016)

    Article  Google Scholar 

  19. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)

  20. Su, S., Delbracio, M., Wang, J., Sapiro, G., Heidrich, W., Wang, O.: Deep video deblurring for hand-held cameras. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1279–1288 (2017)

  21. Sun, J., Cao, W., Xu, Z., Ponce, J.: Learning a convolutional neural network for non-uniform motion blur removal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 769–777 (2015)

  22. Tao, X., Gao, H., Shen, X., Wang, J., Jia, J.: Scale-recurrent network for deep image deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8174–8182 (2018)

  23. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  24. Wang, Z., Ren, J., Zhang, D., Sun, M., Jiang, J.: A deep-learning based feature hybrid framework for spatiotemporal saliency detection inside videos. Neurocomputing 287, 68–83 (2018)

    Article  Google Scholar 

  25. Wu, G., Han, J., Guo, Y., Liu, L., Ding, G., Ni, Q., Shao, L.: Unsupervised deep video hashing via balanced code for large-scale video retrieval. IEEE Trans. Image Process. 28(4), 1993–2007 (2018)

    Article  MathSciNet  Google Scholar 

  26. Wu, G., Han, J., Lin, Z., Ding, G., Zhang, B., Ni, Q.: Joint image-text hashing for fast large-scale cross-media retrieval using self-supervised deep learning. IEEE Trans. Ind. Electron. 66(12), 9868–9877 (2018)

    Article  Google Scholar 

  27. Xu, L., Ren, J.S., Liu, C., Jia, J.: Deep convolutional neural network for image deconvolution. In: Advances in Neural Information Processing Systems, pp. 1790–1798 (2014)

  28. Xu, L., Zheng, S., Jia, J.: Unnatural l0 sparse representation for natural image deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1107–1114 (2013)

  29. Xu, X., Sun, D., Pan, J., Zhang, Y., Pfister, H., Yang, M.H.: Learning to super-resolve blurry face and text images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 251–260 (2017)

  30. Yan, C., Tu, Y., Wang, X., Zhang, Y., Hao, X., Zhang, Y., Dai, Q.: Stat: spatial-temporal attention mechanism for video captioning. IEEE Trans. Multimed. (2019)

  31. Yan, Y., Ren, J., Sun, G., Zhao, H., Han, J., Li, X., Marshall, S., Zhan, J.: Unsupervised image saliency detection with gestalt-laws guided optimization and visual attention based refinement. Pattern Recognit. 79, 65–78 (2018)

    Article  Google Scholar 

  32. Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010)

    Article  MathSciNet  Google Scholar 

  33. Yang, W., Feng, J., Yang, J., Zhao, F., Liu, J., Guo, Z., Yan, S.: Deep edge guided recurrent residual learning for image super-resolution. IEEE Trans. Image Process. 26(12), 5895–5907 (2017)

    Article  MathSciNet  Google Scholar 

  34. Zhang, K., Wang, B., Zuo, W., Zhang, H., Zhang, L.: Joint learning of multiple regressors for single image super-resolution. IEEE Signal Process. Lett. 23(1), 102–106 (2016)

    Article  Google Scholar 

  35. Zhang, K., Zuo, W., Zhang, L.: Learning a single convolutional super-resolution network for multiple degradations. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3262–3271 (2018)

  36. Zhang, X., Dong, H., Hu, Z., Lai, W.S., Wang, F., Yang, M.H.: Gated fusion network for joint image deblurring and super-resolution. In: BMVC (2018)

  37. Zhang, X., Wang, F., Dong, H., Guo, Y.: A deep encoder–decoder networks for joint deblurring and super-resolution. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1448–1452. IEEE (2018)

  38. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2472–2481 (2018)

Download references

Acknowledgements

This work is supported in part by the National Natural Science Foundation of China under Grant No. 61971004, by the Key Project of Natural Science of Anhui Provincial Department of Education under Grant No. KJ2019A0083) and by the Natural Science Foundation of Anhui University of Technology under Grant No. RD18100244.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Anhui University of Technology, Ma’anshan, 243032, China

    Heng Liu, Jiajun Qin, Zilin Fu & Xue Li

  2. Warwick Manufacturing Group, The University of Warwick, Conventry, CV4 7AL, UK

    Jungong Han

Authors
  1. Heng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiajun Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zilin Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xue Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jungong Han
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Heng Liu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, H., Qin, J., Fu, Z. et al. Fast simultaneous image super-resolution and motion deblurring with decoupled cooperative learning. J Real-Time Image Proc 17, 1787–1800 (2020). https://doi.org/10.1007/s11554-020-00976-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-020-00976-x

Keywords

Search

Navigation