Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
SpringerLink
Log in

Lightweight single image super-resolution based on multi-path progressive feature fusion and attention mechanism

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Single image super-resolution (SISR) is a fundamental image processing task, which aims to generate a high-resolution (HR) image from its low-resolution (LR) counterpart. Deep convolution neural networks (CNNs) have significantly improved the performance of SISR, and dominated the current research on SISR techniques. However, the performance improvement depends heavily on the size of the networks. In general, the deeper the networks, the better the performance, which limits their use to devices with limited computing and memory resources. The challenge is to find the optimal balance between network model complexity and SISR performance. In this paper, we propose a new lightweight SISR algorithm based on CNNs, which uses multi-path progressive feature fusion and attention mechanism. Our main contributions are as follows: (1) We propose a multi-path progressive feature fusion block (PFF), which can use the feature from the previous path to gradually guide the feature learning of the next path step by step in multiple paths. (2) We propose a multi-path feature attention mechanism (FAM), which can adaptively weigh the multi-path feature channels to be concatenated, improve the utilization of feature information and feature representation capability. The experimental results show that whether it is an objective measurement or a subjective measurement, our method is better than other similar state of the art methods, and has a better model complexity and performance balance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data availability

The datasets generated and analyzed during the current study are available from the corresponding author on reasonable request. The data we used are all publicly available, and the sources of all datasets can be found in the references of the original article.

References

  1. Agarap A (2018) Deep learning using rectified linear units (relu). Preprint at arXiv:1803.08375v2

  2. Ahn N, Kang B, Sohn KA (2018) Fast, accurate, and lightweight super-resolution with cascading residual network. In: 15th European Conference on computer vision (ECCV). Springer, Munich, pp 256–272. https://doi.org/10.1007/978-3-030-01249-6_16

  3. Bevilacqua M, Roumy A, Guillemot C, et al (2012) Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: 23rd British Amchine Vision Conference (BMVC). https://doi.org/10.5244/C.26.135

  4. Cao C, Liu X, Yang Y, et al (2015) Look and think twice: capturing top-down visual attention with feedback convolutional neural networks. In: IEEE International Conference on computer vision (ICCV). Springer, Santiago, pp 2956–2964. https://doi.org/10.1109/ICCV.2015.338

  5. Dai T, Cai JR, Zhang YB, et al (2019) Second-order attention network for single image super-resolution. In: IEEE/CVF Conference on computer vision and pattern recognition (CVPR). IEEE, pp 11057–11066. https://doi.org/10.1109/CVPR.2019.01132

  6. Dong C, Loy CC, He KM et al (2015) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307. https://doi.org/10.1109/TPAMI.2015.2439281

    Article  Google Scholar 

  7. Dong C, Loy CC, Tang XO (2016) Accelerating the super-resolution convolutional neural network. In: 14th European Conference on computer vision (ECCV). Springer, Amsterdam, pp 391–407. https://doi.org/10.1007/978-3-319-46475-6_257

  8. Freeman WT, Pasztor EC, Carmichael OT (2000) Learning low-level vision. Int J Comput Vis 40(1):25–47. https://doi.org/10.1109/ICCV.1999.790414

    Article  MATH  Google Scholar 

  9. Gao H, Yuan H, Wang Z et al (2019) Pixel transposed convolutional networks. IEEE Trans Pattern Anal Mach Intell 42(5):1218–1227. https://doi.org/10.1109/TPAMI.2019.2893965

    Article  Google Scholar 

  10. Goodfellow I, Pouget-Abadie J, Mirza M, et al (2014) Generative adversarial nets. In: Ghahramani Z, Welling M, Cortes C, et al (eds) Advances in neural information processing systems, vol 27. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf. Accessed 17 Dec 2022

  11. Han W, Chang S, Liu D, et al (2018) Image super-resolution via dual-state recurrent networks. In: 2018 IEEE Conference on computer vision and pattern recognition. IEEE, pp 1654–1663. https://doi.org/10.1109/CVPR.2018.00178

  12. He K, Zhang X, Ren S, et al (2015) Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: 2015 IEEE International Conference on computer vision (ICCV). IEEE, pp 1026–1034. https://doi.org/10.1109/ICCV.2015.123

  13. Howard AG, Zhu M, Chen B, et al (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. Preprint at arXiv:1704.04861

  14. Hu J, Shen L, Sun G (2019) Squeeze-and-excitation networks. IEEE Trans Pattern Anal Mach Intell 42:2011–2023. https://doi.org/10.1109/TPAMI.2019.2913372. (IEEE)

    Article  Google Scholar 

  15. Huang G, Liu Z, Van Der Maaten L, et al (2017) Densely connected convolutional networks. In: 2017 IEEE Conference on computer vision and pattern recognition. IEEE, pp 2261–2269. https://doi.org/10.1109/CVPR.2017.243

  16. Huang JB, Singh A, Ahuja N (2015) Single image super-resolution from transformed self-exemplars. In: 2015 IEEE Conference on computer vision and pattern recognition. IEEE, pp 5197–5206. https://doi.org/10.1109/CVPR.2015.7299156

  17. Huang Z, Li W, Li J et al (2021) Dual-path attention network for single image super-resolution. Expert Syst Appl 169(114):450. https://doi.org/10.1016/j.eswa.2020.114450

    Article  Google Scholar 

  18. Hui Z, Wang XM, Gao XB (2018) Fast and accurate single image super-resolution via information distillation network. In: 2018 IEEE/CVF Conference on computer vision and pattern recognition. IEEE, pp 723–731. https://doi.org/10.1109/CVPR.2018.00082

  19. Hui Z, Gao X, Yang Y, et al (2019) Lightweight image super-resolution with information multi-distillation network. In: 27th ACM International Conference on multimedia. association for computing machinery, pp 2024–2032. https://doi.org/10.1145/3343031.3351084

  20. Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: 2016 IEEE Conference on computer vision and pattern recognition. IEEE, pp 1646–1654. https://doi.org/10.1109/CVPR.2016.182

  21. Kim J, Lee JK, Lee KM (2016) Deeply-recursive convolutional network for image super-resolution. In: 2016 IEEE Conference on computer vision and pattern recognition. IEEE, pp 1637–1645. https://doi.org/10.1109/CVPR.2016.181

  22. Kim J, Lee JK, Lee KM (2017) Image super-resolution using dense skip connections. In: 2017 IEEE International Conference on computer vision. IEEE, pp 4799–4807. https://doi.org/10.1109/CVPR.2016.182

  23. Kingma D, Ba J (2014) Adam: a method for stochastic optimization. Comput Sci Preprint at arXiv:1412.6980

  24. Lai WS, Huang JB, Ahuja N, et al (2017) Deep Laplacian pyramid networks for fast and accurate super-resolution. In: 2017 IEEE Conference on computer vision and pattern recognition. IEEE, pp 5835–5843. https://doi.org/10.1109/CVPR.2017.618

  25. Ledig C, Theis L, Huszar F, et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on computer vision and pattern recognition (CVPR)

  26. Li W, Zhou K, Qi L, et al (2021) Lapar: inearly-assembled pixel-adaptive regression network for single image super-resolution and beyond. Adv Neural Inform Process Syst 33:20343–20355. Preprint at arXiv:2105.10422

  27. Lim B, Son S, Kim H, et al (2017) Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on computer vision and pattern recognition workshops. IEEE, Honolulu, pp 136–144. https://doi.org/10.1109/CVPRW.2017.151

  28. Liu J, Tang J, Wu G (2020) Residual feature distillation network for lightweight image super-resolution. In: Proceedings of the 16th European Conference on computer vision. Springer, Glasgow, pp 41–55. https://doi.org/10.1007/978-3-030-67070-2_2

  29. Liu JJ, Hou Q, Cheng MM, et al (2020) Improving convolutional networks with self-calibrated convolutions. In: 2020 IEEE/CVF Conference on computer vision and pattern recognition. IEEE, pp 10096–10105. https://doi.org/10.1109/CVPR42600.2020.01011

  30. Luo X, Xie Y, Zhang Y, et al (2020) Latticenet: towards lightweight image super-resolution with lattice block. In: 16th European Conference on computer vision. Springer, Glasgow, pp 272–289. https://doi.org/10.1007/978-3-030-58542-6_17

  31. Martin D, Fowlkes C, Tal D, et al (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: 8th IEEE International Conference on computer vision, vol 2. IEEE, pp 416–423. https://doi.org/10.1109/ICCV.2001.937655

  32. Matsui Y, Ito K, Aramaki Y et al (2017) Single image super-resolution from transformed self-exemplars. Multimed Tools Appl 76(20):21811–21838. https://doi.org/10.1007/s11042-016-4020-z

    Article  Google Scholar 

  33. Muqeet A, Hwang J, Yang S, et al (2020) Multi-attention based ultra lightweight image super-resolution. In: 16th European Conference on computer vision. Springer, Glasgow, pp 103–118. https://doi.org/10.1007/978-3-030-67070-2_6

  34. Park SC, Park MK, Kang MG (2003) Super-resolution image reconstruction a technical overview. IEEE Signal Process Mag 20(3):21–36. https://doi.org/10.1109/MSP.2003.1203207

    Article  Google Scholar 

  35. Paszke A, Gross S, Chintala S et al (2017) Automatic differentiation in pytorch. 31st Conference on Neural Information Processing Systems (NIPS 2017), 4–9 December 2017, Long Beach, pp 1–4

  36. Peyré G, Bougleux S, Cohen L (2008) Non-local regularization of inverse problems. In: European Conference on computer vision. Springer, Marseille, pp 57–68. https://doi.org/10.1007/978-3-540-88690-7_5

  37. Shi WZ, Caballero J, Ledig C, et al (2013) Cardiac image super-resolution with global correspondence using multi-atlas patchmatch. In: International Conference on medical image computing and computer-assisted intervention. Springer, Nagoya, pp 9–16, https://doi.org/10.1007/978-3-642-40760-4_2

  38. Sun L, Pan J, Tang J (2022) Shufflemixer: an efficient convnet for image super-resolution. In: Advances in Neural Information Processing Systems, vol 35

  39. Tai Y, Liu XM, Xu CY (2017) Memnet: a persistent memory network for image restoration. In: 2017 IEEE International Conference on computer vision. IEEE, pp 4539–4547. https://doi.org/10.1109/ICCV.2017.486

  40. Tai Y, Yang J, Liu X (2017) Image super-resolution via deep recursive residual network. In: 2017 IEEE Conference on computer vision and pattern recognition. Springer, Munich, pp 3147–3155. https://doi.org/10.1109/CVPR.2017.298

  41. Thornton WM, Atkinson PM, Holland DA (2006) Sub-pixel mapping of rural land cover objects from fine spatial resolution satellite sensor imagery using super-resolution pixel-swapping. Int J Remote Sens 27(3):473–491. https://doi.org/10.1080/01431160500207088

    Article  Google Scholar 

  42. Timofte R, Agustsson E, Van Gool L, et al (2017) Ntire 2017 challenge on single image super-resolution: methods and results. In: 2017 IEEE Conference on computer vision and pattern recognition workshops. https://doi.org/10.1109/CVPRW.2017.149

  43. Wang C, Li Z, Shi J (2019) Lightweight image super-resolution with adaptive weighted learning network. Preprint at arXiv:1904.02358

  44. Wang X, Wang Q, Zhao Y, et al (2020) Lightweight single-image super-resolution network with attentive auxiliary feature learning. In: 15th Asian Conference on computer vision. Springer, Kyoto, pp 268–285. https://doi.org/10.1007/978-3-030-69532-3_17

  45. Wang XT, Yu K, Wu SX, et al (2018) Esrgan: enhanced super-resolution generative adversarial networks. In: 2018 European Conference on computer vision (ECCV). Springer, Munich, pp 63–79. https://doi.org/10.1007/978-3-030-11021-5_5

  46. Wang Z, Bovik AC, Sheikh HR et al (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  47. Woo S, Park JY, Lee JY, et al (2018) Cbam: convolutional block attention module. In: 2018 European conference on computer vision (ECCV). Springer, Munich, pp 3–19. https://doi.org/10.1007/978-3-030-01234-2_1

  48. Ymff0592 (2021) Super-resolution. https://github.com/ymff0592/super-resolution. Accessed 31 Mar 2022

  49. Yu J, Fan Y, Huang T (2019) Wide activation for efficient image and video super-resolution. In: Sidorov K, Hicks Y (eds) Proceedings of the British Machine Vision Conference (BMVC). BMVA Press, pp 52.1–52.13. https://doi.org/10.5244/C.33.52

  50. Www Z, Yuen PC (2010) Very low resolution face recognition problem. Int J Remote Sens 21(1):327–340. https://doi.org/10.1109/BTAS.2010.5634490

    Article  MathSciNet  Google Scholar 

  51. Zeyde R, Elad M, Protter M (2010) On single image scale-up using sparse-representations. In: 7th International Conference on curves and surfaces. Springer, Avignon, pp 711–730. https://doi.org/10.1007/978-3-642-27413-8_47

  52. Zhang J, Long C, Wang Y et al (2021) A two-stage attentive network for single image super-resolution. IEEE Trans Circ Syst Video Technol 32(3):1020–1033. https://doi.org/10.1109/TCSVT.2021.3071191

    Article  Google Scholar 

  53. Zhang K, Zuo W, Zhang L (2018) Learning a single convolutional super-resolution network for multiple degradations. In: 2018 IEEE/CVF Conference on computer vision and pattern recognition. IEEE, pp 3262–3271. https://doi.org/10.1109/CVPR.2018.00344

  54. Zhang X, Zhou X, Lin M, et al (2018) Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: 2018 IEEE/CVF conference on computer vision and pattern recognition. IEEE, pp 6848–6856. https://doi.org/10.1109/CVPR.2018.00716

  55. Zhang YL, Li KP, Li K, et al (2018) Image super-resolution using very deep residual channel attention networks. In: 15th European Conference on computer vision (ECCV). Springer, Munich, pp 294–310. https://doi.org/10.1007/978-3-030-01234-2_18

  56. Zhang YL, Li KP, Li K, et al (2018) Residual dense network for image super-resolution. In: 15th IEEE Conference on computer vision and pattern recognition. Springer, Munich, pp 294–310. https://doi.org/10.1007/978-3-030-01234-2_18

  57. Zhao H, Kong X, He J, et al (2020) Efficient image super-resolution using pixel attentio. In: 16th European conference on computer vision. Springer, Glasgow, pp 56–72, https://doi.org/10.1007/978-3-030-67070-2_3

  58. Zhao X, Liao Y, He T, et al (2019) F \(c^{2}\) n:fully channel-concatenated network for single image super-resolution. Preprint at arXiv:1907.03221v2

  59. Zhou D, Duan R, Zhao L et al (2020) Single image super-resolution reconstruction based on multi-scale feature mapping adversarial network. Signal Process 166(107):251. https://doi.org/10.1016/j.sigpro.2019.107251

    Article  Google Scholar 

  60. Zhou D, Chen Y, Li W et al (2021) Image super-resolution based on adaptive cascading attention network. Expert Syst Appl 186(115):815. https://doi.org/10.1016/j.eswa.2021.115815

    Article  Google Scholar 

  61. Zhu X, Guo K, Ren S, et al (2022) Lightweight image super-resolution with expectation-maximization attention mechanism. IEEE Transactions on Circuits and Systems for Video Technology 32(3):1273C1284. https://doi.org/10.1109/TCSVT.2021.3078436

Download references

Author information

Authors and Affiliations

  1. School of Control and Computer Engineering, North China Electric Power University, Beijing, 102206, People’s Republic of China

    Shanshan Li, Dengwen Zhou, Yukai Liu, Dandan Gao & Wanjun Wang

Authors
  1. Shanshan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dengwen Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yukai Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dandan Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wanjun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dengwen Zhou.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, S., Zhou, D., Liu, Y. et al. Lightweight single image super-resolution based on multi-path progressive feature fusion and attention mechanism. Int. J. Mach. Learn. & Cyber. 14, 3517–3528 (2023). https://doi.org/10.1007/s13042-023-01847-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-023-01847-0

Keywords

Search

Navigation