Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article

Reconfigurable System-on-Chip Architectures for Robust Visual SLAM on Humanoid Robots

Published: 09 February 2023 Publication History

Abstract

Visual Simultaneous Localization and Mapping (vSLAM) is the method of employing an optical sensor to map the robot’s observable surroundings while also identifying the robot’s pose in relation to that map. The accuracy and speed of vSLAM calculations can have a very significant impact on the performance and effectiveness of subsequent tasks that need to be executed by the robot, making it a key building component for current robotic designs. The application of vSLAM in the area of humanoid robotics is particularly difficult due to the robot’s unsteady locomotion. This paper introduces a pose graph optimization module based on RGB (ORB) features, as an extension of the KinectFusion pipeline (a well-known vSLAM algorithm), to assist in recovering the robot’s stance during unstable gait patterns when the KinectFusion tracking system fails. We develop and test a wide range of embedded MPSoC FPGA designs, and we investigate numerous architectural improvements, both precise and approximation, to study their impact on performance and accuracy. Extensive design space exploration reveals that properly designed approximations, which exploit domain knowledge and efficient management of CPU and FPGA fabric resources, enable real-time vSLAM at more than 30 fps in humanoid robots with high energy-efficiency and without compromising robot tracking and map construction. This is the first FPGA design to achieve robust, real-time dense SLAM operation targeting specifically humanoid robots. An open source release of our implementations and data can be found in [1].

References

[2]
Mohamed Abouzahir, Abdelhafid Elouardi, Rachid Latif, Samir Bouaziz, and Abdelouahed Tajer. 2018. Embedding SLAM algorithms: Has it come of age? Robotics and Autonomous Systems 100 (2018), 14–26.
[3]
Nikolay A. Atanasov, J. L. Ny, Kostas Daniilidis, and George J. Pappas. 2015. Decentralized active information acquisition: Theory and application to multi-robot SLAM. IEEE International Conference on Robotics and Automation (ICRA) (2015), 4775–4782.
[4]
Raghav Bansal, Gaurav Raj, and Tanupriya Choudhury. 2016. Blur image detection using Laplacian operator and Open-CV. 2016 International Conference System Modeling & Advancement in Research Trends (SMART) (2016), 63–67.
[5]
Paul J. Besl et al. 1992. A method for registration of 3-D shapes. IEEE Trans. Pattern Analysis and Machine Intelligence 14, 2 (1992).
[6]
Bruno Bodin, Harry Wagstaff, Sajad Saeedi, Luigi Nardi, Emanuele Vespa, John Mawer, Andy Nisbet, Mikel Luján, Steve B. Furber, Andrew J. Davison, Paul H. J. Kelly, and Michael F. P. O’Boyle. 2018. SLAMBench2: Multi-objective head-to-head benchmarking for visual SLAM. CoRR abs/1808.06820.
[7]
Konstantinos Boikos and Christos-Savvas Bouganis. 2016. Semi-dense SLAM on an FPGA SoC. In 26th International Conference on Field Programmable Logic and Applications, (FPL), Lausanne, Switzerland, August 29–September 2, 2016, Paolo Ienne, Walid A. Najjar, Jason Helge Anderson, Philip Brisk, and Walter Stechele (Eds.). IEEE, 1–4.
[8]
Konstantinos Boikos and Christos Savvas Bouganis. 2017. A high-performance system-on-chip architecture for direct tracking for SLAM. In 27th International Conference on Field Programmable Logic and Applications, (FPL), Ghent, Belgium, September 4–8. 1–7.
[9]
Konstantinos Boikos and Christos Savvas Bouganis. 2019. A scalable FPGA-based architecture for depth estimation in SLAM. In 15th International Symposium on Applied Reconfigurable Computing, (ARC) Darmstadt, Germany, April 9–11. 181–196.
[10]
Vanderlei Bonato, Eduardo Marques, and George A. Constantinides. 2009. A floating-point extended Kalman filter implementation for autonomous mobile robots. J. Signal Process. Syst. 56, 1 (2009), 41–50.
[11]
Guillaume Bresson, Z. Alsayed, Li Yu, and S. Glaser. 2017. Simultaneous localization and mapping: A survey of current trends in autonomous driving. IEEE Transactions on Intelligent Vehicles 2 (2017), 194–220.
[12]
David J. Bruemmer and Mark S. Swinson. 2003. Humanoid robots. In Encyclopedia of Physical Science and Technology (Third Edition), Robert A. Meyers (Ed.). Academic Press, New York, 401–425.
[13]
Mihai Bujanca, Paul Gafton, Sajad Saeedi, Andy Nisbet, Bruno Bodin, Michael F. P. O’Boyle, Andrew J. Davison, Paul H. J. Kelly, Graham D. Riley, Barry Lennox, Mikel Luján, and Steve B. Furber. 2019. SLAMBench 3.0: Systematic automated reproducible evaluation of SLAM systems for robot vision challenges and scene understanding. In International Conference on Robotics and Automation, ICRA 2019, Montreal, QC, Canada, May 20–24, 2019. IEEE, 6351–6358.
[14]
Cesar Cadena, Luca Carlone, Henry Carrillo, Yasir Latif, Davide Scaramuzza, José Neira, Ian Reid, and John J. Leonard. 2016. Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age. IEEE Transactions on Robotics 32, 6 (2016), 1309–1332.
[15]
Brian Curless and Marc Levoy. 1996. A volumetric method for building complex models from range images. In 23rd Annual Conference on Computer Graphics and Interactive Techniques, (SIGGRAPH), New Orleans, LA, USA, August 4–9, 1996. 303–312.
[16]
Igor Cvisic, Josip Cesic, Ivan Marković, and I. Petrović. 2018. SOFT-SLAM: Computationally efficient stereo visual simultaneous localization and mapping for autonomous unmanned aerial vehicles. J. Field Robotics 35 (2018), 578–595.
[17]
A. Davison, I. Reid, N. Molton, and O. Stasse. 2007. MonoSLAM: Real-time single camera SLAM. IEEE Transactions on Pattern Analysis and Machine Intelligence 29 (2007), 1052–1067.
[18]
Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. 2018. SuperPoint: Self-supervised interest point detection and description. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2018, Salt Lake City, UT, USA, June 18–22, 2018. 224–236.
[19]
Jakob Engel, Thomas Schöps, and Daniel Cremers. 2014. LSD-SLAM: Large-scale direct monocular SLAM. In Computer Vision - ECCV 2014–13th European Conference, Zurich, Switzerland, September 6–12, 2014, Proceedings, Part II (Lecture Notes in Computer Science), Vol. 8690. Springer, 834–849.
[20]
Weikang Fang, Yanjun Zhang, Bo Yu, and Shaoshan Liu. 2017. FPGA-based ORB feature extraction for real-time visual SLAM. In International Conference on Field Programmable Technology, (FPT), Melbourne, Australia, December 11–13, 2017. 275–278.
[21]
Marcel Flottmann, Marc Eisoldt, Julian Gaal, Marc Rothmann, Marco Tassemeier, Thomas Wiemann, and Mario Porrmann. 2021. Energy-efficient FPGA-accelerated LiDAR-based SLAM for embedded robotics. In International Conference on Field-Programmable Technology, (FPT) Auckland, New Zealand, December 6–10, 2021. IEEE, 1–6.
[22]
Christian Forster, Simon Lynen, L. Kneip, and D. Scaramuzza. 2013. Collaborative monocular SLAM with multiple Micro Aerial Vehicles. 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (2013), 3962–3970.
[23]
Quentin Gautier, Alric Althoff, and Ryan Kastner. 2019. FPGA architectures for real-time dense SLAM. In 30th IEEE International Conference on Application-specific Systems, Architectures and Processors, (ASAP), New York, NY, USA, July 15–17, 2019. 83–90.
[24]
Quentin Gautier, Alexandria Shearer, Janarbek Matai, Dustin Richmond, Pingfan Meng, and Ryan Kastner. 2014. Real-time 3D reconstruction for FPGAs: A case study for evaluating the performance, area, and programmability trade-offs of the Altera OpenCL SDK. In 2014 International Conference on Field-Programmable Technology, FPT Shanghai, China, December 10–12, 2014. 326–329.
[25]
A. Gil, Ó. Reinoso, M. Ballesta, and Miguel Juliá. 2010. Multi-robot visual SLAM using a Rao-Blackwellized particle filter. Robotics Auton. Syst. 58 (2010), 68–80.
[26]
Maria Rafaela Gkeka, Alexandros Patras, Christos D. Antonopoulos, Spyros Lalis, and Nikolaos Bellas. 2021. FPGA architectures for approximate dense SLAM computing. In 24th Conference on Design, Automation and Test in Europe (DATE) Virtual Conference, February 1–3, 2021.
[27]
Mengyuan Gu, Kaiyuan Guo, Wenqiang Wang, Yu Wang, and Huazhong Yang. 2015. An FPGA-based real-time simultaneous localization and mapping system. In International Conference on Field Programmable Technology, (FPT) Queenstown, New Zealand, December 7–9, 2015. 200–203.
[28]
Ankur Handa et al. 2014. A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM. In ICRA Hong Kong, China, May.
[29]
A. Hornung, Kai Wurm, and Maren Bennewitz. 2010. Humanoid robot localization in complex indoor environments. IEEE/RSJ 2010 International Conference on Intelligent Robots and Systems, IROS 2010 - Conference Proceedings, 1690–1695. DOI:
[30]
Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, and Andrew Fitzgibbon. 2011. KinectFusion: Real-time 3D reconstruction and interaction using a moving depth camera. UIST’11 - Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, 559–568.
[31]
Olaf Kähler et al. 2015. Very high frame rate volumetric integration of depth images on mobile devices. IEEE Trans. Vis. Comput. Graph. 21, 11 (2015).
[32]
C. Kerl, Jürgen Sturm, and D. Cremers. 2013. Dense visual SLAM for RGB-D cameras. 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (2013), 2100–2106.
[33]
Rainer Kümmerle, Giorgio Grisetti, Hauke Strasdat, Kurt Konolige, and Wolfram Burgard. 2011. G2o: A general framework for graph optimization. Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA), 3607–3613. DOI:
[34]
Jinyu Li, Bangbang Yang, Danpeng Chen, Nan Wang, Guofeng Zhang, and Hujun Bao. 2019. Survey and evaluation of monocular visual-inertial SLAM algorithms for augmented reality. Virtual Real. Intell. Hardw. 1 (2019), 386–410.
[35]
Haomin Liu, Mingyu Chen, Guofeng Zhang, Hujun Bao, and Yingze Bao. 2018. ICE-BA: Incremental, consistent and efficient bundle adjustment for visual-inertial SLAM. In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), Salt Lake City, UT, USA, June 18–22, 2018. Computer Vision Foundation / IEEE Computer Society, 1974–1982.
[36]
Runze Liu, Jianlei Yang, Yiran Chen, and Weisheng Zhao. 2019. eSLAM: An energy-efficient accelerator for real-time ORB-SLAM on FPGA platform. In 56th Annual Design Automation Conference, (DAC), Las Vegas, NV, USA, June 02–06, 2019. ACM, 193.
[37]
Tanwi Mallick, Parthapratim Das, and Arun Majumdar. 2014. Characterizations of noise in Kinect depth images: A review. IEEE Sensors Journal 14, 6 (2014), 1731–1740.
[38]
Sparsh Mittal. 2016. A survey of techniques for approximate computing. ACM Comput. Surv. 48, 4 (2016).
[39]
Michael Montemerlo, Sebastian Thrun, Daphne Koller, and Ben Wegbreit. 2003. FastSLAM 2.0: An improved particle filtering algorithm for simultaneous localization and mapping that provably converges. In IJCAI.
[40]
Marius Muja and David Lowe. 2009. Fast approximate nearest neighbors with automatic algorithm configuration. VISAPP 2009 - Proceedings of the 4th International Conference on Computer Vision Theory and Applications 1, 331–340.
[41]
Raul Mur-Artal et al. 2015. ORB-SLAM: A versatile and accurate monocular SLAM system. IEEE Trans. Robotics 31, 5 (2015).
[42]
Raul Mur-Artal and J. D. Tardós. 2017. ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-D cameras. IEEE Transactions on Robotics 33 (2017), 1255–1262.
[43]
Luigi Nardi, Bruno Bodin, M. Zeeshan Zia, John Mawer, Andy Nisbet, Paul H. J. Kelly, Andrew J. Davison, Mikel Luján, Michael F. P. O’Boyle, Graham D. Riley, Nigel P. Topham, and Stephen B. Furber. 2015. Introducing SLAMBench, a performance and accuracy benchmarking methodology for SLAM. In International Conference on Robotics and Automation, (ICRA), Seattle, WA, USA, 26–30 May. 5783–5790.
[44]
J. Nikolic, J. Rehder, M. Burri, Pascal Gohl, Stefan Leutenegger, P. Furgale, and R. Siegwart. 2014. A synchronized visual-inertial sensor system with FPGA pre-processing for accurate real-time SLAM. In International Conference on Robotics and Automation, (ICRA), Hong Kong, China, May 31–June 7, 2014.
[45]
Jinwook Oh, Jungwook Choi, Guilherme C. Januario, and Kailash Gopalakrishnan. 2016. Energy-efficient simultaneous localization and mapping via compounded approximate computing. In IEEE International Workshop on Signal Processing Systems (SiPS), Dallas, TX, USA, October 26–28, 2016.
[46]
Giuseppe Oriolo, Antonio Paolillo, Lorenzo Rosa, and Marilena Vendittelli. 2012. Vision-based odometric localization for humanoids using a kinematic EKF. In 12th IEEE-RAS International Conference on Humanoid Robots (Humanoids 2012). 153–158.
[47]
R. Ozawa, Y. Takaoka, Y. Kida, K. Nishiwaki, J. Chestnutt, J. Kuffner, J. Kagami, H. Mizoguch, and H. Inoue. 2005. Using visual odometry to create 3D maps for online footstep planning. In 2005 IEEE International Conference on Systems, Man and Cybernetics, Vol. 3. 2643–2648.
[48]
Yan Pei, Swarnendu Biswas, Donald S. Fussell, and Keshav Pingali. 2019. SLAMBooster: An application-aware online controller for approximation in dense SLAM. In 28th International Conference on Parallel Architectures and Compilation Techniques, (PACT), Seattle, WA, USA, September 23–26, 2019.
[49]
Yan Pei, Swarnendu Biswas, Donald S. Fussell, and Keshav Pingali. 2020. A methodology for principled approximation in visual SLAM. In International Conference on Parallel Architectures and Compilation Techniques (PACT), Virtual Event, GA, USA, October 3–7, 2020. 373–386.
[50]
Stylianos Piperakis, Nikolaos Tavoularis, Emmanouil Hourdakis, Dimitrios Kanoulas, and Panos Trahanias. 2019. Humanoid Robot Dense RGB-D SLAM for Embedded Devices. (042019). DOI:
[51]
Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. 2011. ORB: An efficient alternative to SIFT or SURF. Proceedings of the IEEE International Conference on Computer Vision, 2564–2571.
[52]
Sajad Saeedi et al. 2018. Navigating the landscape for real-time localization and mapping for robotics and virtual and augmented reality. Proc. IEEE 106, 11 (2018), 2020–2039.
[53]
Renato F. Salas-Moreno, Richard A. Newcombe, H. Strasdat, P. Kelly, and A. Davison. 2013. SLAM++: Simultaneous localisation and mapping at the level of objects. 2013 IEEE Conference on Computer Vision and Pattern Recognition (2013), 1352–1359.
[54]
Patrik Schmuck. 2017. Multi-UAV collaborative monocular SLAM. 2017 IEEE International Conference on Robotics and Automation (ICRA) (2017), 3863–3870.
[55]
Thomas Schöps, Torsten Sattler, and M. Pollefeys. 2019. BAD SLAM: Bundle adjusted direct RGB-D SLAM. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019), 134–144.
[56]
Stelios Sidiroglou-Douskos et al. 2011. Managing performance vs. accuracy trade-offs with loop perforation. In ESEC/FSE, Szeged, Hungary, Sept. 2011.
[57]
Edgar Simo-Serra, Eduard Trulls, Luis Ferraz, Iasonas Kokkinos, Pascal Fua, and Francesc Moreno-Noguer. 2015. Discriminative learning of deep convolutional feature point descriptors. In 2015 IEEE International Conference on Computer Vision, ICCV Santiago, Chile, December 7–13, 2015. 118–126.
[58]
Randall C. Smith, Matthew Self, and Peter C. Cheeseman. 1987. Estimating uncertain spatial relationships in robotics. Proceedings. 1987 IEEE International Conference on Robotics and Automation 4 (1987), 850–850.
[59]
Amr Suleiman et al. 2019. Navion: A 2-mW fully integrated real-time visual-inertial odometry accelerator for autonomous navigation of nano drones. IEEE Journal of Solid-State Circuits 54, 4 (2019).
[60]
Y. Takaoka, Y. Kida, S. Kagami, H. Mizoguchi, and T. Kanade. 2004. 3D map building for a humanoid robot by using visual odometry. In 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583), Vol. 5. 4444–4449. DOI:
[61]
Shaharyar Ahmed Khan Tareen and Zahra Saleem. 2018. A comparative analysis of SIFT, SURF, KAZE, AKAZE, ORB, and BRISK. International Conference on Computing, Mathematics and Engineering Technologies (iCoMET) (2018), 1–10.
[62]
Daniel Tortei Tertei, Jonathan Piat, and Michel Devy. 2014. FPGA design and implementation of a matrix multiplier based accelerator for 3D EKF SLAM. In International Conference on ReConFigurable Computing and FPGAs, ReConFig14, Cancun, Mexico, December 8–10, 2014. IEEE, 1–6.
[63]
Daniel Tortei Tertei, Jonathan Piat, and Michel Devy. 2016. FPGA design of EKF block accelerator for 3D visual SLAM. Comput. Electr. Eng. 55 (2016), 123–137.
[64]
Simon Thompson, Satoshi Kagami, and Koichi Nishiwaki. 2006. Localisation for autonomous humanoid navigation. In 2006 6th IEEE-RAS International Conference on Humanoid Robots. 13–19.
[65]
R. Tibshirani. 1996. Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society (Series B) 58 (1996).
[66]
Carlo Tomasi and Roberto Manduchi. 1998. Bilateral filtering for gray and color images. In 6th International Conference on Computer Vision (ICCV), Bombay, India, January 4–7, 1998.
[67]
Bastien Vincke, Abdelhafid Elouardi, and Alain Lambert. 2012. Real time simultaneous localization and mapping: Towards low-cost multiprocessor embedded systems. EURASIP J. Embed. Syst. 2012 (2012), 5.
[68]
Thomas Whelan, Stefan Leutenegger, Renato F. Salas-Moreno, B. Glocker, and A. Davison. 2015. ElasticFusion: Dense SLAM without a pose graph. In Robotics: Science and Systems.
[69]
Émilie Wirbel, Bruno Steux, Silvere Bonnabel, and Arnaud de La Fortelle. 2013. Humanoid robot navigation: From a visual SLAM to a visual compass. In 10th IEEE International Conference on Networking, Sensing and Control, ICNSC 2013, Evry, France, April 10–12, 2013. 678–683.
[70]
Zhilin Xu, Jincheng Yu, Chao Yu, Hao Shen, Yu Wang, and Huazhong Yang. 2020. CNN-based feature-point extraction for real-time visual SLAM on embedded FPGA. In 28th IEEE Annual International Symposium on Field-Programmable Custom Computing Machines, (FCCM), Fayetteville, AR, USA, May 3–6, 2020. 33–37.
[71]
Tianwei Zhang, Emiko Uchiyama, and Yoshihiko Nakamura. 2018. Dense RGB-D SLAM for humanoid robots in the dynamic humans environment. In IEEE-RAS International Conference on Humanoid Robots, Humanoids. Beijing, China, November 6–9, 2018. 270–276.

Cited By

View all
  • (2024)TrinitySLAM: On-board Real-time Event-image Fusion SLAM System for DronesACM Transactions on Sensor Networks10.1145/369642020:6(1-22)Online publication date: 20-Sep-2024
  • (2024)Enhancing Real-time Simultaneous Localization and Mapping with FPGA-based EKF-SLAM's Hardware ArchitectureProcedia Computer Science10.1016/j.procs.2024.05.061236(517-526)Online publication date: 2024
  • (2023)Research on Visual SLAM Navigation Techniques for Dynamic EnvironmentsInternational Journal of Distributed Sensor Networks10.1155/2023/20258442023(1-10)Online publication date: 1-Sep-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Embedded Computing Systems
ACM Transactions on Embedded Computing Systems  Volume 22, Issue 2
March 2023
560 pages
ISSN:1539-9087
EISSN:1558-3465
DOI:10.1145/3572826
  • Editor:
  • Tulika Mitra
Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Journal Family

Publication History

Published: 09 February 2023
Online AM: 09 November 2022
Accepted: 04 September 2022
Revised: 06 August 2022
Received: 17 February 2022
Published in TECS Volume 22, Issue 2

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Simultaneous localization and mapping
  2. humanoid robots
  3. FPGA
  4. approximate computing

Qualifiers

  • Research-article
  • Refereed

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)271
  • Downloads (Last 6 weeks)36
Reflects downloads up to 23 Dec 2024

Other Metrics

Citations

Cited By

View all
  • (2024)TrinitySLAM: On-board Real-time Event-image Fusion SLAM System for DronesACM Transactions on Sensor Networks10.1145/369642020:6(1-22)Online publication date: 20-Sep-2024
  • (2024)Enhancing Real-time Simultaneous Localization and Mapping with FPGA-based EKF-SLAM's Hardware ArchitectureProcedia Computer Science10.1016/j.procs.2024.05.061236(517-526)Online publication date: 2024
  • (2023)Research on Visual SLAM Navigation Techniques for Dynamic EnvironmentsInternational Journal of Distributed Sensor Networks10.1155/2023/20258442023(1-10)Online publication date: 1-Sep-2023
  • (2023)Electronic Sensor Multi-Modal Slam Algorithm Based on Information Fusion Technology2023 International Conference on Ambient Intelligence, Knowledge Informatics and Industrial Electronics (AIKIIE)10.1109/AIKIIE60097.2023.10390439(1-6)Online publication date: 2-Nov-2023
  • (2023)Exploring Sparse Visual Odometry Acceleration With High-Level SynthesisIEEE Access10.1109/ACCESS.2023.326899211(70741-70763)Online publication date: 2023

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

Full Text

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media