Just Accepted
OmniCharger: CNN-based Hand Gesture Interface to Operate an Electric Car Charging Robot through Teleconference
Abstract
The automation of the car charging process is motivated by the rapid development of technologies for self-driving cars and the increasing importance of ecological transportation units. Automation of this process requires the implementation of Computer Vision (CV) techniques. However, it remains challenging to precisely position the charger plug autonomously due to the sensitivity of CV algorithms to lighting and weather conditions. We introduce a novel robotic operation system based on hand gesture recognition through teleconferencing software. The users, connected by teleconference, use their hand gestures to teleoperate the electric plug located on the collaborative robot end-effector. We conducted a user study to evaluate the system performance and suitability using OmniCharger and two baseline interfaces (a UR10 Teach Pendant and a Logitech F710 Wireless Gamepad). Except for two trials, all the users were able to locate the plug inside of a 5 cm target using the interfaces. The distance to the target and the orientation error did not present statistically significant differences ( \(p=0.1099\,{\gt}\,0.05\) and \(p=0.0903\,{\gt}\,0.05\) , respectively) in the use of the three interfaces. The NASA TLX questionnaire results showed low values in all the sub-classes, the SUS results rated the usability of the proposed interface above average (68%), and the UEQ showed excellent performance of the OmniCharger interface in the attractiveness, stimulation, and novelty attributes.
References
[1]
Xue Bai, Changqiang Li, Keyan Chen, Yongjie Feng, Zhaowei Yu, and Ming Xu. 2018. Kinect-Based Hand Tracking for First-Person-Perspective Robotic Arm Teleoperation. In 2018 IEEE International Conference on Information and Automation (ICIA). 684–691. https://doi.org/10.1109/ICInfA.2018.8812561
[2]
Stefano Bonaiuto, Alberto Cannavò, Giovanni Piumatti, Gianluca Paravati, and Fabrizio Lamberti. 2017. Tele-operation of Robot Teams: A Comparison of Gamepad, Mobile Device and Hand Tracking-Based User Interfaces. In 2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC), Vol. 2. 555–560. https://doi.org/10.1109/COMPSAC.2017.278
[3]
John Brooke. 1996. SUS: A quick and dirty usability scale. Usability Eval. Ind. 189(3) (1996), 189–194.
[4]
European Commission. 2022. Zero emission vehicles: first ‘Fit for 55’ deal will end the sale of new CO2 emitting cars in Europe by 2035. https://ec.europa.eu/commission/presscorner/detail/en/ip_22_6462
[5]
The Entertainment Software Association (ESA). 2022. 2022 Essential Facts About the Video Game Industry. https://www.theesa.com/wp-content/uploads/2022/06/2022-Essential-Facts-About-the-Video-Game-Industry.pdf
[6]
Video Games Europe. 2022. All About Video Games European Key Facts 2022. https://www.videogameseurope.eu/wp-content/uploads/2023/08/Video-Games-Europe_Key-Facts-2022_FINAL.pdf
[7]
Research Institute for Disabled Costumers. 2021. Going Electric? https://www.ridc.org.uk/transport/going-electric
[8]
Xingang Fu, Jiang Lu, Ting Zhang, Chadwell Bonair, and Marvin L. Coats. 2015. Wavelet Enhanced Image Preprocessing and Neural Networks for Hand Gesture Recognition. In 2015 IEEE International Conference on Smart City/SocialCom/SustainCom (SmartCity). 838–843. https://doi.org/10.1109/SmartCity.2015.172
[9]
Roman Golovanov, Dmitry Vorotnev, and Darina Kalina. 2020. Combining Hand Detection and Gesture Recognition Algorithms for Minimizing Computational Cost. In 2020 22th International Conference on Digital Signal Processing and its Applications (DSPA). 1–4. https://doi.org/10.1109/DSPA48919.2020.9213273
[10]
Sandra G. Hart. 1987. Background description and application of the NASA task load index (TLX). In Department of Defens Human Engineering Technical Advisory Group Workshop on Workload, Newport, RI.
[11]
Jobair Hossain Gourob, Sourav Raxit, and Afnan Hasan. 2021. A Robotic Hand: Controlled With Vision Based Hand Gesture Recognition System. In 2021 International Conference on Automation, Control and Mechatronics for Industry 4.0 (ACMI). 1–4. https://doi.org/10.1109/ACMI53878.2021.9528192
[12]
IEA. 2023. Electric car sales. https://www.iea.org/data-and-statistics/charts/electric-car-sales-2016-2023
[13]
Kuka. 2022. Kuka smartPAD. https://www.kuka.com/en-us/products/robotics-systems/robot-controllers/smartpad
[14]
Bettina Laugwitz, Theo Held, and Martin Schrepp. 2008. Construction and Evaluation of a User Experience Questionnaire. In HCI and Usability for Education and Work. 63–76. https://doi.org/10.1007/978-3-540-89350-9_6
[15]
Shuntao Liu, Chicheng Liu, Zhi Liu, Ying Xie, Jing Xu, and Ken Chen. 2014. Laser tracker-based control for peg-in-hole assembly robot. In The 4th Annual IEEE International Conference on Cyber Technology in Automation, Control and Intelligent. 569–573. https://doi.org/10.1109/CYBER.2014.6917526
[16]
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg. 2016. SSD: Single Shot MultiBox Detector. In Computer Vision – ECCV 2016. 21–37. https://doi.org/10.1007/978-3-319-46448-0_2
[17]
Xincan Lv, Guangzeng Chen, Haopeng Hu, and Yunjiang Lou. 2019. A Robotic Charging Scheme for Electric Vehicles Based on Monocular Vision and Force Perception. In 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO). 2958–2963. https://doi.org/10.1109/ROBIO49542.2019.8961689
[18]
Justinas Miseikis, Matthias Ruther, Bernhard Walzel, Mario Hirz, and Helmut Brunner. 2017. 3D vision guided robotic charging station for electric and plug-in hybrid vehicles. In Proceedings of the OAGM&ARW Joint Workshop 2017. https://doi.org/10.3217/978-3-85125-524-9-13
[19]
National Association of Insurance Commissioners. 2022. Autonomous Vehicles. https://content.naic.org/cipr-topics/autonomous-vehicles#:~:text=The%20Insurance%20Institute%20for%20Highway,and%204.5 %20million%20by%202030.
[20]
Hyeonjun Park, Ji-Hun Bae, Jae-Han Park, Moon-Hong Baeg, and Jaeheung Park. 2013. Intuitive peg-in-hole assembly strategy with a compliant manipulator. In IEEE ISR 2013. 1–5. https://doi.org/10.1109/ISR.2013.6695699
[21]
Pengkun Quan, Ya’Nan Lou, Haoyu Lin, Zhuo Liang, and Shichun Di. 2022. Research on Fast Identification and Location of Contour Features of Electric Vehicle Charging Port in Complex Scenes. IEEE Access 10 (2022), 26702–26714. https://doi.org/10.1109/ACCESS.2021.3092210
[22]
Universal Robots. 2022. 3PE Teach Pendant for E-series. https://www.universal-robots.com/plus/products/universal-robots/3pe-teach-pendant-for-e-series/
[23]
Elena Sánchez-Nielsen, Luis Antón-Canalís, and Mario Hernández-Tejera. 2004. Hand gesture recognition for human-machine interaction. In Journal of WSCG, Vol. 12. 395–402.
[24]
Jeff Sauro and James R. Lewis. 2012. Chapter 8 - Standardized Usability Questionnaires. In Quantifying the User Experience. Morgan Kaufmann, Boston, 185–240. https://doi.org/10.1016/B978-0-12-384968-7.00008-4
[25]
Martin Schrepp, Andreas Hinderks, and Jörg Thomaschewski. 2017. Construction of a Benchmark for the User Experience Questionnaire (UEQ). International Journal of Interactive Multimedia and Artificial Intelligence 4 (06 2017), 40–44. https://doi.org/10.9781/ijimai.2017.445
[26]
Valerii Serpiva, Ekaterina Karmanova, Aleksey Fedoseev, Stepan Perminov, and Dzmitry Tsetserukou. 2021. DronePaint: Swarm Light Painting with DNN-based Gesture Recognition. In ACM SIGGRAPH 2021 Emerging Technologies (Virtual Event, USA). Article 6. https://doi.org/10.1145/3450550.3465349
[27]
Jamie Shotton, Andrew Fitzgibbon, Mat Cook, Toby Sharp, Mark Finocchio, Richard Moore, Alex Kipman, and Andrew Blake. 2011. Real-time human pose recognition in parts from single depth images. In CVPR 2011. 1297–1304. https://doi.org/10.1109/CVPR.2011.5995316
[28]
T. Simon, H. Joo, I. Matthews, and Y. Sheikh. 2017. Hand Keypoint Detection in Single Images Using Multiview Bootstrapping. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 4645–4653. https://doi.org/10.1109/CVPR.2017.494
[29]
J. Ernesto Solanes, Adolfo Muños, Luis Garcia, Ana Martí, Vicent Girbés-Juan, and Josep Tornero. 2020. Teleoperation of industrial robot manipulators based on augmented reality. The International Journal of Advanced Manufacturing Technology 111 (2020). https://doi.org/10.1007/s00170-020-05997-1
[30]
A. Stemmer, A. Albu-Schaffer, and G. Hirzinger. 2007. An Analytical Method for the Planning of Robust Assembly Tasks of Complex Shaped Planar Parts. In Proceedings 2007 IEEE International Conference on Robotics and Automation. 317–323. https://doi.org/10.1109/ROBOT.2007.363806
[31]
Jesus Suarez and Robin R. Murphy. 2012. Hand gesture recognition with depth images: A review. In 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication. 411–417. https://doi.org/10.1109/ROMAN.2012.6343787
[32]
Jing-Hao Sun, Ting-Ting Ji, Shu-Bin Zhang, Jia-Kui Yang, and Guang-Rong Ji. 2018. Research on the Hand Gesture Recognition Based on Deep Learning. In 2018 12th International Symposium on Antennas, Propagation and EM Theory (ISAPE). 1–4. https://doi.org/10.1109/ISAPE.2018.8634348
[33]
UltraLeap. [n. d.]. Leap Motion Controller. https://www.ultraleap.com/product/leap-motion-controller/
[34]
Bernhard Walzel, Christopher Sturm, Jürgen Fabian, and Mario Hirz. 2016. Automated robot-based charging system for electric vehicles. https://doi.org/10.1007/978-3-658-13255-2_70
[35]
Keigo Watanabe, Gilbert Tang, and Phil Webb. 2018. The Design and Evaluation of an Ergonomic Contactless Gesture Control System for Industrial Robots. Journal of Robotics 2018 (2018). https://doi.org/10.1155/2018/9791286
[36]
Automotive World. 2023. Are electric vehicles still the future of automotive? https://www.automotiveworld.com/articles/are-electric-vehicles-still-the-future-of-automotive/
[37]
Fan Zhang, Valentin Bazarevsky, Andrey Vakunov, Andrei Tkachenka, George Sung, Chuo-Ling Chang, and Matthias Grundmann. 2020. MediaPipe Hands: On-device Real-time Hand Tracking. https://doi.org/10.48550/arXiv.2006.10214
Index Terms
- OmniCharger: CNN-based Hand Gesture Interface to Operate an Electric Car Charging Robot through Teleconference
Recommendations
Humanoid robot control using depth camera
HRI '11: Proceedings of the 6th international conference on Human-robot interactionMost human interactions with the environment depend on our ability to navigate freely and to use our hands and arms to manipulate objects. Developing natural means of controlling these abilities in humanoid robots can significantly broaden the usability ...
A hand gesture interface device
This paper reports on the development of a hand to machine interface device that provides real-time gesture, position and orientation information. The key element is a glove and the device as a whole incorporates a collection of technologies. Analog ...
A hand gesture interface device
This paper reports on the development of a hand to machine interface device that provides real-time gesture, position and orientation information. The key element is a glove and the device as a whole incorporates a collection of technologies. Analog ...
Comments
Information & Contributors
Information
Published In
EISSN:2573-9522
Table of ContentsCopyright © 2024 Copyright held by the owner/author(s).
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Online AM: 27 April 2024
Accepted: 28 March 2024
Revised: 15 February 2024
Received: 20 September 2022
Check for updates
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 74Total Downloads
- Downloads (Last 12 months)74
- Downloads (Last 6 weeks)36
Reflects downloads up to
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in