Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

Dual-Fisheye Image Stitching via Unsupervised Deep Learning

  • Conference paper
  • First Online:
MultiMedia Modeling (MMM 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14556))

Included in the following conference series:

  • 905 Accesses

Abstract

Constructing panoramic images from a dual-fisheye lens has been increasingly used along with the recent booming of new computer vision applications, such as virtual reality (VR) and augmented reality(AR). The recent development of deep learning (DL) techniques has shed new light on the field of image stitching, but little research has been conducted on DL-based dual-fisheye image stitching. In this work, we propose an unsupervised deep learning method for dual-fisheye image stitching. Specifically, we construct a stitching system consisting of fisheye distortion correction, unsupervised image reconstruction, and image edge rectangularization blocks. Experiment results show that the proposed scheme can perform accurate and natural stitching of two images, and exceed the traditional method in PSNR, SSIM, RMSE, MSE, and other performance indicators.

This work is supported in part by the National Natural Science Foundation of China (NSFC) under Grant 62272256, the Shandong Provincial Natural Science Foundation under Grants ZR2021MF026 and ZR2023MF040, the Innovation Capability Enhancement Program for Small and Medium-sized Technological Enterprises of Shandong Province under Grants 2022TSGC2180 and 2022TSGC2123, the Innovation Team Cultivating Program of Jinan under Grant 202228093, the Piloting Fundamental Research Program for the Integration of Scientific Research, Education and Industry of Qilu University of Technology (Shandong Academy of Sciences) under Grants 2021JC02014 and 2022XD001, the Talent Cultivation Promotion Program of Computer Science and Technology in Qilu University of Technology (Shandong Academy of Sciences) under Grants 2021PY05001 and 2023PY059.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Chalfoun, J., et al.: Mist: accurate and scalable microscopy image stitching tool with stage modeling and error minimization. Sci. Rep. 7(1), 1–10 (2017)

    Article  Google Scholar 

  2. Duan, H., Min, X., Sun, W., Zhu, Y., Zhang, X-P., Zhai, G.: Attentive deep image quality assessment for omnidirectional stitching, IEEE J. Se. Top. Sign. Process. 2023

    Google Scholar 

  3. Jia, Q., Feng, X., Liu, Y., Fan, X., Latecki, L.J.: Learning pixel-wise alignment for unsupervised image stitching. Network 1(1), 1 (2023)

    Google Scholar 

  4. Lo, I-C., Chen,H.H. : Acquiring 360\(\circ \) light field by a moving dual-fisheye camera, IEEE Transactions on Image Processing, 2023

    Google Scholar 

  5. Lo, I.-C., Shih, K.-T., Chen, H.H.: Efficient and accurate stitching for 360\(\circ \) dual-fisheye images and videos. IEEE Trans. Image Process. 31, 251–262 (2021)

    Article  Google Scholar 

  6. Kweon, H., Kim, H., Kang, Y., Yoon, Y., Jeong, W., Yoon, K.-J.: Pixel-wise warping for deep image stitching. Proc. AAAI Conf. Artif. Intell. 37(1), 1196–1204 (2023)

    Google Scholar 

  7. Lo, Y.C. Huang, C.C., Tsai, Y.F., Lo, I.C., Wu, A.Y.A., Chen, H.H.: Face recognition for fisheye images, in 2022 IEEE International Conference on Image Processing (ICIP). IEEE, 2022, pp. 146–150

    Google Scholar 

  8. Li, Y.-H., Lo, I.-C., Chen, H.H.: Deep face rectification for 360 dual-fisheye cameras. IEEE Trans. Image Process. 30, 264–276 (2020)

    Article  Google Scholar 

  9. Souza, T., et al.: 360 stitching from dual-fisheye cameras based on feature cluster matching, In: 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). IEEE 2018, pp.313–320 (2018)

    Google Scholar 

  10. Ni, G., Chen, X., Zhu, Y., He, L., “Dual-fisheye lens stitching and error correction, In : 10th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI). IEEE 2017, pp .1–6 (2017)

    Google Scholar 

  11. Lo, I.-C., Shih, K.-T., Chen, H.H.: Image stitching for dual fisheye cameras, In: 2018 25th IEEE International Conference on Image Processing (ICIP). IEEE, 2018, pp. 3164–3168

    Google Scholar 

  12. Dong, Y., Pei, M., Zhang, L., Xu, B., Wu, Y., Jia, Y.: Stitching videos from a fisheye lens camera and a wide-angle lens camera for telepresence robots. Int. J. Soc. Robot. 14(3), 733–745 (2022)

    Article  Google Scholar 

  13. Brown, M., Lowe, D.G.: Automatic panoramic image stitching using invariant features. Int. J. Comput. Vision 74, 59–73 (2007)

    Article  Google Scholar 

  14. Lo, I.-C., Shih, K.-T., Chen, H.H.: 360\(\circ \) video stitching for dual fisheye cameras, In 2019 IEEE International Conference on Image Processing (ICIP). IEEE, 2019, pp. 3522–3526

    Google Scholar 

  15. Cheng, H., Xu, C. Wang, J. Zhao,L. : Quad-fisheye image stitching for monoscopic panorama reconstruction, In: Computer Graphics Forum, vol. 41, no. 6. Wiley Online Library, 2022, pp. 94–109

    Google Scholar 

  16. Ufer, N., Ommer, B.: Deep semantic feature matching, In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 6914–6923

    Google Scholar 

  17. Rocco,I. Arandjelovic, R. Sivic,J.: Convolutional neural network architecture for geometric matching, In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 6148–6157

    Google Scholar 

  18. Jeon, Sangryul, Kim, Seungryong, Min, Dongbo, Sohn, Kwanghoon: PARN: Pyramidal Affine Regression Networks for Dense Semantic Correspondence. In: Ferrari, Vittorio, Hebert, Martial, Sminchisescu, Cristian, Weiss, Yair (eds.) ECCV 2018. LNCS, vol. 11210, pp. 355–371. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01231-1_22

    Chapter  Google Scholar 

  19. Song, D.-Y., Lee, G., Lee, H., Um, G.-M., Cho, D.: Weakly-supervised stitching network for real-world panoramic image generation, In: European Conference on Computer Vision. Springer, 2022, pp. 54–71

    Google Scholar 

  20. Harris, C., Stephens, M., et al.,: A combined corner and edge detector, In; Alvey Vision Conference, vol. 15, no. 50. Citeseer, 1988, pp. 10–5244

    Google Scholar 

  21. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60(2), 91–110 (2004)

    Article  Google Scholar 

  22. Bay, H., Ess, A., Tuytelaars, T., Van Gool, L.: Speeded-up robust features (surf). Comput. Vis. Image Underst. 110(3), 346–359 (2008)

    Article  Google Scholar 

  23. Rosten, Edward, Drummond, Tom: Machine Learning for High-Speed Corner Detection. In: Leonardis, Aleš, Bischof, Horst, Pinz, Axel (eds.) ECCV 2006. LNCS, vol. 3951, pp. 430–443. Springer, Heidelberg (2006). https://doi.org/10.1007/11744023_34

    Chapter  Google Scholar 

  24. Roberto, R., et al.: Using local refinements on 360 stitching from dual-fisheye cameras. In: VISIGRAPP (5: VISAPP), 2020, pp. 17–26

    Google Scholar 

  25. Zhang, Z.: Flexible camera calibration by viewing a plane from unknown orientations, In: Proceedings of the seventh IEEE International Conference on Computer Vision, vol. 1. IEEE, 1999, pp. 666–673

    Google Scholar 

  26. —, A flexible new technique for camera calibration, IEEE Trans. Pattern Anal. Mach. Intell. vol. 22, no. 11, pp. 1330–1334, 2000

    Google Scholar 

  27. Tareen, S.A.K., Saleem, Z.: A comparative analysis of sift, surf, kaze, akaze, orb, and brisk, in,: International Conference on Computing, Mathematics and Engineering Technologies (iCoMET). IEEE 2018, pp. 1–10 (2018)

    Google Scholar 

  28. Nie, L., Lin, C., Liao, K., Liu, S., Zhao, Y.: Unsupervised deep image stitching: reconstructing stitched features to images. IEEE Trans. Image Process. 30, 6184–6197 (2021)

    Article  MathSciNet  Google Scholar 

  29. Jaderberg, M., Simonyan, K., Zisserman, A., et al.: Spatial transformer networks, Adv. Neural Inf. process. syst. vol. 28, 2015

    Google Scholar 

  30. He, K., Chang, H., Sun, J.: Rectangling panoramic images via warping. ACM Trans. Graph. (TOG) 32(4), 1–10 (2013)

    Google Scholar 

  31. Li, D., He, K., Sun, J., Zhou, K., : A geodesic-preserving method for image warping, In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 213–221

    Google Scholar 

  32. Nie, L., Lin, C., Liao, K., Liu, S., Zhao, Y.: Deep rectangling for image stitching: a learning baseline, In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5740–5748

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences), Jinan, 250353, China

    Zhanjie Jin & Anming Dong

  2. School of Information Science and Technology, Qilu University of Technology (Shandong Academy of Sciences), Jinan, 25353, China

    Zhanjie Jin & Anming Dong

  3. Big Data Institute, Qilu University of Technology (Shandong Academy of Sciences), Jinan, 25353, China

    Jiguo Yu

  4. Advanced Technology Research Institute, Beijing Institute of Technology, Jinan, China

    Shuxiang Dong

  5. echnology Department, Shandong HiCon New Media Institute Co., Ltd, Jinan, China

    You Zhou

Authors
  1. Zhanjie Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anming Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiguo Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shuxiang Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. You Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anming Dong .

Editor information

Editors and Affiliations

  1. University of Amsterdam, Amsterdam, The Netherlands

    Stevan Rudinac

  2. Delft University of Technology, Delft, The Netherlands

    Alan Hanjalic

  3. Delft University of Technology, Delft, The Netherlands

    Cynthia Liem

  4. University of Amsterdam, Amsterdam, The Netherlands

    Marcel Worring

  5. Reykjavik University, Reykjavik, Iceland

    Björn Þór Jónsson

  6. Microsoft Research Lab – Asia, Beijing, China

    Bei Liu

  7. The University of Tokyo, Tokyo, Japan

    Yoko Yamakata

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jin, Z., Dong, A., Yu, J., Dong, S., Zhou, Y. (2024). Dual-Fisheye Image Stitching via Unsupervised Deep Learning. In: Rudinac, S., et al. MultiMedia Modeling. MMM 2024. Lecture Notes in Computer Science, vol 14556. Springer, Cham. https://doi.org/10.1007/978-3-031-53311-2_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-53311-2_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-53310-5

  • Online ISBN: 978-3-031-53311-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation