research-article

RT-libSGM: An Implementation of a Real-time Stereo Matching System on FPGA

Authors:

Masatoshi Arai,

Hideharu AmanoAuthors Info & Claims

HEART '22: Proceedings of the 12th International Symposium on Highly-Efficient Accelerators and Reconfigurable Technologies

Pages 1 - 9

https://doi.org/10.1145/3535044.3535045

Published: 09 June 2022 Publication History

Abstract

Stereo depth estimation has become an attractive topic in the computer vision field. Although various algorithms strive to optimize the speed and the precision of estimation, the energy cost of a system is also an essential metric for an embedded system. Among these various algorithms, Semi-Global Matching (SGM) has been a popular choice for some real-world applications because of its accuracy-and-speed balance. However, its power consumption makes it difficult to be applied to an embedded system. Thus, we propose a robust stereo matching system, RT-libSGM, working on the Xilinx Field-programmable gate array (FPGA) platforms. The dedicated design of each module optimizes the speed of the entire system while ensuring the flexibility of the system structure. Through an evaluation running on a Zynq FPGA board called M-KUBOS, RT-libSGM achieves state-of-the-art performance with lower power consumption. Compared with the original design (libSGM), when working on the Tegra X2 GPU, RT-libSGM runs 2 × faster at a lower energy cost.

References

[1]

Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. 2020. AdaBins: Depth Estimation using Adaptive Bins. CoRR abs/2011.14141(2020). arXiv:2011.14141https://arxiv.org/abs/2011.14141

[2]

D. V. S. X. De Silva, W. A. C. Fernando, H. Kodikaraarachchi, S. T. Worrall, and A. M. Kondoz. 2011. Improved depth map filtering for 3D-TV systems. In 2011 IEEE International Conference on Consumer Electronics (ICCE). 645–646. https://doi.org/10.1109/ICCE.2011.5722787

[3]

L. Di Stefano, M. Marchionni, S. Mattoccia, and G. Neri. 2002. Dense stereo based on the uniqueness constraint. In 2002 International Conference on Pattern Recognition, Vol. 3. 657–661 vol.3. https://doi.org/10.1109/ICPR.2002.1048024

[4]

James Diebel and Sebastian Thrun. 2006. An Application of Markov Random Fields to Range Sensing. In Advances in Neural Information Processing Systems, Y. Weiss, B. Sch¥”olkopf, and J. Platt (Eds.). Vol. 18. MIT Press. https://proceedings.neurips.cc/paper/2005/file/353de26971b93af88da102641069b440-Paper.pdf

[5]

[5] Fixstars.2021. https://github.com/fixstars/segmentation-sgm/wiki/Segmentation-SGM

[6]

Fixstars. 2022. A CUDA implementation performing Semi-Global Matching. https://github.com/fixstars/libSGM

[7]

Hayato Hagiwara, Yasufumi Touma, Kenichi Asami, and Mochimitsu Komori. 2015. FPGA-Based Stereo Vision System Using Gradient Feature Correspondence. Journal of Robotics and Mechatronics 27, 6 (2015), 681–690. https://doi.org/10.20965/jrm.2015.p0681

[8]

Istvan Haller and Sergiu Nedevschi. 2010. GPU optimization of the SGM stereo algorithm. In Proceedings of the 2010 IEEE 6th International Conference on Intelligent Computer Communication and Processing. 197–202. https://doi.org/10.1109/ICCP.2010.5606438

Digital Library

[9]

Yihui He. 2017. Estimated Depth Map Helps Image Classification. CoRR abs/1709.07077(2017). arXiv:1709.07077http://arxiv.org/abs/1709.07077

[10]

D. Hernandez-Juarez, A. Chacón, A. Espinosa, D. Vázquez, J.C. Moure, and A.M. López. 2016. Embedded Real-time Stereo Estimation via Semi-global Matching on the GPU. Procedia Computer Science 80 (2016), 143–153. https://doi.org/10.1016/j.procs.2016.05.305 International Conference on Computational Science 2016, ICCS 2016, 6-8 June 2016, San Diego, California, USA.

Digital Library

[11]

Heiko Hirschmuller. 2008. Stereo Processing by Semiglobal Matching and Mutual Information. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 2(2008), 328–341. https://doi.org/10.1109/TPAMI.2007.1166

Digital Library

[12]

Martin Humenberger, Tobias Engelke, and Wilfried Kubinger. 2010. A census-based stereo vision algorithm using modified Semi-Global Matching and plane fitting to improve matching quality. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops. 77–84. https://doi.org/10.1109/CVPRW.2010.5543769

[13]

lackmagicdesign. 2022. Teranex Mini 12G Converters. https://www.blackmagicdesign.com/products/teranexmini/models

[14]

Jin Han Lee, Myung-Kyu Han, Dong Wook Ko, and Il Hong Suh. 2019. From big to small: Multi-scale local planar guidance for monocular depth estimation. arXiv preprint arXiv:1907.10326(2019).

[15]

Stephen Longfield Jr. and Mark L. Chang. 2009. A Parameterized Stereo Vision Core for FPGAs. In 2009 17th IEEE Symposium on Field Programmable Custom Computing Machines. 263–266. https://doi.org/10.1109/FCCM.2009.32

Digital Library

[16]

Guibo Luo and Yuesheng Zhu. 2018. Hole Filling with Depth-Guided Global Optimization for View Synthesis. IEEE Access PP (06 2018), 1–1. https://doi.org/10.1109/ACCESS.2018.2847312

[17]

David McKinnon, Ryan N. Smith, and Ben Upcroft. 2012. A semi-local method for iterative depth-map refinement. In 2012 IEEE International Conference on Robotics and Automation. 758–763. https://doi.org/10.1109/ICRA.2012.6224614

[18]

Matthias Michael, Jan Salmen, Johannes Stallkamp, and Marc Schlipsing. 2013. Real-time stereo vision: Optimizing Semi-Global Matching. In 2013 IEEE Intelligent Vehicles Symposium (IV). 1197–1202. https://doi.org/10.1109/IVS.2013.6629629

[19]

Renatus Mushi and Faith Shimba. 2012. A critical performance analysis of Thin Client platforms. 2012 2nd International Conference on Digital Information and Communication Technology and its Applications, DICTAP 2012 (05 2012). https://doi.org/10.1109/DICTAP.2012.6215389

[20]

Katsuki Ohata, Yukitoshi Sanada, Tetsuro Ogaki, Kento Matsuyama, Takanori Ohira, Satoshi Chikuda, Masaki Igarashi, Masayuki Ikebe, Tetsuya Asai, Masato Motomura, and Tadahiro Kuroda. 2013. Hardware-oriented stereo vision algorithm based on 1-D guided filtering and its FPGA implementation. 2013 IEEE 20th International Conference on Electronics, Circuits, and Systems (ICECS) (2013), 169–172.

[21]

Prathmesh Sawant, Yashwant Temburu, Mandar Datar, Imran Ahmed, Vinayak Shriniwas, and Sachin B. Patkar. 2020. Single Storage Semi-Global Matching for Real Time Depth Processing. CoRR abs/2007.03269(2020). arXiv:2007.03269https://arxiv.org/abs/2007.03269

[22]

Akihito Seki and Marc Pollefeys. 2017. SGM-Nets: Semi-Global Matching with Neural Networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 6640–6649. https://doi.org/10.1109/CVPR.2017.703

[23]

Raden Setyawan, Rudy Sunoko, Moch Choiron, and Panca Mudjirahardjo. 2018. Implementation of Stereo Vision Semi-Global Block Matching Methods for Distance Measurement. Indonesian Journal of Electrical Engineering and Computer Science 12 (11 2018), 585–591. https://doi.org/10.11591/ijeecs.v12.i2.pp585-591

[24]

John L. Smith. 1996. Implementing Median Filters in XC4000E FPGAs.

[25]

Stereolabs. 2021. ZED camera. https://www.stereolabs.com/docs/

[26]

Leyuan Wang, Kunbo Zhang, Yunlong Wang, and Zhenan Sun. 2021. An End-to-End Autofocus Camera for Iris on the Move. 2021 IEEE International Joint Conference on Biometrics (IJCB) (2021), 1–8.

[27]

E. Wong. 2006. A New Method for Creating a Depth Map for Camera Auto Focus Using an All in Focus Picture and 2D Scale Space Matching. In 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, Vol. 3. III–III. https://doi.org/10.1109/ICASSP.2006.1660871

[28]

Xilinx. 2022. Vitis Libraries. https://github.com/Xilinx/Vitis_Libraries/tree/master/vision

[29]

Feng Xue, Guirong Zhuo, Ziyuan Huang, Wufei Fu, Zhuoyue Wu, and Marcelo H. Ang Jr.2020. Toward Hierarchical Self-Supervised Monocular Absolute Depth Estimation for Autonomous Driving Applications. CoRR abs/2004.05560(2020). arXiv:2004.05560https://arxiv.org/abs/2004.05560

[30]

Guanglei Yang, Hao Tang, Mingli Ding, Nicu Sebe, and Elisa Ricci. 2021. Transformers Solve the Limited Receptive Field for Monocular Depth Prediction. CoRR abs/2103.12091(2021). arXiv:2103.12091https://arxiv.org/abs/2103.12091

[31]

Jieru Zhao, Tingyuan Liang, Liang Feng, Wenchao Ding, Sharad Sinha, Wei Zhang, and Shaojie Shen. 2020. FP-Stereo: Hardware-Efficient Stereo Vision for Embedded Applications. CoRR abs/2006.03250(2020). arXiv:2006.03250https://arxiv.org/abs/2006.03250

Cited By

WEI KKUNO YARAI MAMANO H(2023)RT-libSGM: FPGA-Oriented Real-Time Stereo Matching System with High ScalabilityIEICE Transactions on Information and Systems10.1587/transinf.2022EDP7131E106.D:3(337-348)Online publication date: 1-Mar-2023
https://doi.org/10.1587/transinf.2022EDP7131
Wang YGu MZhu YChen GXu ZGuo Y(2022)Improvement of AD-Census Algorithm Based on Stereo VisionSensors10.3390/s2218693322:18(6933)Online publication date: 13-Sep-2022
https://doi.org/10.3390/s22186933
Chen YWei KNishi HAmano H(2022)An Implementation of a 3D Image Filter for Motion Vector Generation on an FPGA Board2022 Tenth International Symposium on Computing and Networking (CANDAR)10.1109/CANDAR57322.2022.00018(83-89)Online publication date: Nov-2022
https://doi.org/10.1109/CANDAR57322.2022.00018

Recommendations

A real-time global stereo-matching on FPGA

An improved global stereo matching algorithm is implemented on a single FPGA for real-time applications. Stereo matching is widely used in stereo vision systems, i.e. objects detection and autonomous vehicles. Global algorithms have much more accurate ...
Real-time high-definition stereo matching on FPGA
FPGA '11: Proceedings of the 19th ACM/SIGDA international symposium on Field programmable gate arrays

Although many fast stereo matching designs have been proposed in the past decades, it is still very challenging to achieve real-time speed at high definition resolution while maintaining high matching accuracy. In this paper, we propose a real-time high ...
FPGA implementation of an efficient similarity-based adaptive window algorithm for real-time stereo matching

Stereo matching is one of the most widely used algorithms in real-time image processing applications such as positioning systems for mobile robots, three-dimensional building mapping and both recognition, detection and three-dimensional reconstruction ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

HEART '22: Proceedings of the 12th International Symposium on Highly-Efficient Accelerators and Reconfigurable Technologies

June 2022

114 pages

ISBN:9781450396608

DOI:10.1145/3535044

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 June 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

JST CREST
JST SPRING

Conference

HEART2022

HEART2022: International Symposium on Highly-Efficient Accelerators and Reconfigurable Technologies

June 9 - 10, 2022

Tsukuba, Japan

Acceptance Rates

HEART '22 Paper Acceptance Rate 10 of 21 submissions, 48%;

Overall Acceptance Rate 22 of 50 submissions, 44%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
148
Total Downloads

Downloads (Last 12 months)42
Downloads (Last 6 weeks)0

Reflects downloads up to 22 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

WEI KKUNO YARAI MAMANO H(2023)RT-libSGM: FPGA-Oriented Real-Time Stereo Matching System with High ScalabilityIEICE Transactions on Information and Systems10.1587/transinf.2022EDP7131E106.D:3(337-348)Online publication date: 1-Mar-2023
https://doi.org/10.1587/transinf.2022EDP7131
Wang YGu MZhu YChen GXu ZGuo Y(2022)Improvement of AD-Census Algorithm Based on Stereo VisionSensors10.3390/s2218693322:18(6933)Online publication date: 13-Sep-2022
https://doi.org/10.3390/s22186933
Chen YWei KNishi HAmano H(2022)An Implementation of a 3D Image Filter for Motion Vector Generation on an FPGA Board2022 Tenth International Symposium on Computing and Networking (CANDAR)10.1109/CANDAR57322.2022.00018(83-89)Online publication date: Nov-2022
https://doi.org/10.1109/CANDAR57322.2022.00018
Castells-Rufas DNgo VBorrego-Carazo JCodina MSanchez CGil DCarrabina J(2022)A Survey of FPGA-Based Vision Systems for Autonomous CarsIEEE Access10.1109/ACCESS.2022.323028210(132525-132563)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3230282

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents