Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
SpringerLink
Log in

Hierarchical feature fusion network for light field spatial super-resolution

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Light field (LF) spatial super-resolution (SR) aims to restore a high-resolution LF image from a degraded low-resolution one. However, due to the complexity of high-dimensional LF images, the existing LF spatial SR methods failed to fully incorporate the correlation between sub-aperture images of the LF. To mitigate this problem, we propose a hierarchical feature fusion network (LF-HFNet) for LF spatial SR with two novel components, namely feature interaction module and residual spatial and channel attention block. By cascading several residual spatial-angular separable convolution blocks with concatenation connections, the former can fully utilize the hierarchical and complementary information between SAIs. And the latter can adaptively rescale the feature responses for emphasizing informative features. Experimental results on both synthetic and real-world LF datasets demonstrate that the proposed method outperforms other state-of-the-art methods with higher PSNR/SSIM.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Alain, M., Smolic, A.: Light field super-resolution via lfbm5d sparse coding. In: 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 2501–2505. IEEE (2018)

  2. BiBi, S., Abbas, M., Misro, M.Y., Hu, G.: A novel approach of hybrid trigonometric Bézier curve to the modeling of symmetric revolutionary curves and symmetric rotation surfaces. IEEE Access 7, 165779–165792 (2019)

    Article  Google Scholar 

  3. BiBi, S., Abbas, M., Miura, K.T., Misro, M.Y.: Geometric modeling of novel generalized hybrid trigonometric Bézier-like curve with shape parameters and its applications. Mathematics 8(6), 967 (2020)

    Article  Google Scholar 

  4. Bishop, T.E., Favaro, P.: The light field camera: extended depth of field, aliasing, and superresolution. IEEE Trans. Pattern Anal. Mach. Intell. 34(5), 972–986 (2011)

    Article  Google Scholar 

  5. Cheng, Z., Xiong, Z., Chen, C., Liu, D., Zha, Z.J.: Light field super-resolution with zero-shot learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10010–10019 (2021)

  6. Dai, T., Cai, J., Zhang, Y., Xia, S.T., Zhang, L.: Second-order attention network for single image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11065–11074 (2019)

  7. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: European Conference on Computer Vision, pp. 184–199. Springer (2014)

  8. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: European Conference on Computer Vision, pp. 391–407. Springer (2016)

  9. Egiazarian, K., Katkovnik, V.: Single image super-resolution via bm3d sparse coding. In: 2015 23rd European Signal Processing Conference (EUSIPCO), pp. 2849–2853. IEEE (2015)

  10. Honauer, K., Johannsen, O., Kondermann, D., Goldluecke, B.: A dataset and evaluation methodology for depth estimation on 4d light fields. In: Asian Conference on Computer Vision, pp. 19–34. Springer (2016)

  11. Jia, X., De Brabandere, B., Tuytelaars, T., Van Gool, L.: Dynamic filter networks. In: NIPS (2016)

  12. Jin, J., Hou, J., Chen, J., Kwong, S.: Light field spatial super-resolution via deep combinatorial geometry embedding and structural consistency regularization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2260–2269 (2020)

  13. Jin, J., Hou, J., Chen, J., Kwong, S., Yu, J.: Light field super-resolution via attention-guided fusion of hybrid lenses. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 193–201 (2020)

  14. Jin, J., Hou, J., Yuan, H., Kwong, S.: Learning light field angular super-resolution via a geometry-aware network. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 11141–11148 (2020)

  15. Kalantari, N.K., Wang, T.C., Ramamoorthi, R.: Learning-based view synthesis for light field cameras. ACM Trans Graphics (TOG) 35(6), 1–10 (2016)

    Article  Google Scholar 

  16. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)

  17. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 136–144 (2017)

  18. Maqsood, S., Abbas, M., Hu, G., Ramli, A.L.A., Miura, K.T.: A novel generalization of trigonometric Bézier curve and surface with shape parameters and its applications. Math. Prob. Eng. 2020, (2020). https://doi.org/10.1155/2020/4036434

  19. Meng, N., Wu, X., Liu, J., Lam, E.: High-order residual network for light field super-resolution. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 11757–11764 (2020)

  20. Mitra, K., Veeraraghavan, A.: Light field denoising, light field superresolution and stereo camera based refocussing using a gmm light field patch prior. In: 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 22–28. IEEE (2012)

  21. Raj, A.S., Lowney, M., Shah, R., Wetzstein, G.: Stanford lytro light field archive (2016)

  22. Rossi, M., Frossard, P.: Geometry-consistent light field super-resolution via graph-based regularization. IEEE Trans. Image Process. 27(9), 4207–4218 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  23. Shi, J., Jiang, X., Guillemot, C.: A framework for learning depth from a flexible subset of dense and sparse light field views. IEEE Trans. Image Process. 28(12), 5867–5880 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  24. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)

  25. Shi, W., Du, H., Mei, W., Ma, Z.: (sarn) spatial-wise attention residual network for image super-resolution. The Visual Computer pp. 1–12 (2020)

  26. Shin, C., Jeon, H.G., Yoon, Y., Kweon, I.S., Kim, S.J.: Epinet: A fully-convolutional neural network using epipolar geometry for depth from light field images. IEEE (2018)

  27. Sun, Q., Zhang, S., Chang, S., Zhu, L., Lin, Y.: Multi-dimension fusion network for light field spatial super-resolution using dynamic filters. arXiv preprint arXiv:2008.11449 (2020)

  28. Usman, M., Abbas, M., Miura, K.T.: Some engineering applications of new trigonometric cubic Bézier-like curves to free-form complex curve modeling. J. Adv. Mech. Des. Syst. Manuf. 14(4), JAMDSM0048–JAMDSM0048 (2020)

    Article  Google Scholar 

  29. Wang, Y., Liu, F., Zhang, K., Hou, G., Sun, Z., Tan, T.: Lfnet: a novel bidirectional recurrent convolutional neural network for light-field image super-resolution. IEEE Trans. Image Process. 27(9), 4274–4286 (2018)

    Article  MathSciNet  Google Scholar 

  30. Wang, Y., Wang, L., Yang, J., An, W., Yu, J., Guo, Y.: Spatial-angular interaction for light field image super-resolution. In: European Conference on Computer Vision, pp. 290–308. Springer (2020)

  31. Wang, Y., Yang, J., Wang, L., Ying, X., Wu, T., An, W., Guo, Y.: Light field image super-resolution using deformable convolution. IEEE Trans. Image Process. 30, 1057–1071 (2020)

    Article  Google Scholar 

  32. Wanner, S., Goldluecke, B.: Variational light field analysis for disparity estimation and super-resolution. IEEE Trans. Pattern Anal. Mach. Intell. 36(3), 606–619 (2013)

    Article  Google Scholar 

  33. Yeung, H.W.F., Hou, J., Chen, X., Chen, J., Chen, Z., Chung, Y.Y.: Light field spatial super-resolution using deep efficient spatial-angular separable convolution. IEEE Trans. Image Process. 28(5), 2319–2330 (2018)

    Article  MathSciNet  Google Scholar 

  34. Yoon, Y., Jeon, H.G., Yoo, D., Lee, J.Y., Kweon, I.S.: Light-field image super-resolution using convolutional neural network. IEEE Signal Process. Lett. 24(6), 848–852 (2017)

    Article  Google Scholar 

  35. Yoon, Y., Jeon, H.G., Yoo, D., Lee, J.Y., So Kweon, I.: Learning a deep convolutional network for light-field image super-resolution. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 24–32 (2015)

  36. Yuan, Y., Cao, Z., Su, L.: Light-field image superresolution using a combined deep cnn based on epi. IEEE Signal Process. Lett. 25(9), 1359–1363 (2018)

    Article  Google Scholar 

  37. Yücer, K., Sorkine-Hornung, A., Wang, O., Sorkine-Hornung, O.: Efficient 3d object segmentation from densely sampled light fields with applications to 3d reconstruction. ACM Trans. Graphics (TOG) 35(3), 1–15 (2016)

    Article  Google Scholar 

  38. Zhang, J., Liu, Y., Zhang, S., Poppe, R., Wang, M.: Light field saliency detection with deep convolutional networks. IEEE Trans. Image Process. 29, 4421–4434 (2020)

  39. Zhang, S., Lin, Y., Sheng, H.: Residual networks for light field image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11046–11055 (2019)

  40. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 286–301 (2018)

  41. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2472–2481 (2018)

Download references

Acknowledgements

This work was supported in part by the Science and Technology Plan Project of Sichuan Province under Grant 2021YFG0350, in part by the National Key Research and Development Program of China under Grant 2016YFB0700802, in part by the Innovative Youth Fund Program of the State Oceanic Administration of China under Grant 2015001 and in part by Sichuan Dazhou Intelligent Manufacturing Industry Technology Research Institute under Grant ZNZZ21.

Author information

Authors and Affiliations

  1. College of Computer Science, Sichuan University, Chengdu, China

    Xiyao Hua, Minghui Wang & Xia Liu

  2. College of Intelligent Manufacturing, Sichuan University of Arts and Sciences, Dazhou, China

    Boni Su

Authors
  1. Xiyao Hua
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Minghui Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Boni Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xia Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Minghui Wang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hua, X., Wang, M., Su, B. et al. Hierarchical feature fusion network for light field spatial super-resolution. Vis Comput 39, 267–279 (2023). https://doi.org/10.1007/s00371-021-02327-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-021-02327-8

Keywords

Search

Navigation