research-article

Real-time Embedded Recognition of Sign Language Alphabet Fingerspelling in an IMU-Based Glove

Authors:

Chaithanya Kumar Mummadi,

Frederic Philips Peter Leo,

Keshav Deep Verma,

Shivaji Kasireddy,

Philipp Marcel Scholl,

Kristof Van LaerhovenAuthors Info & Claims

iWOAR '17: Proceedings of the 4th International Workshop on Sensor-based Activity Recognition and Interaction

Article No.: 11, Pages 1 - 6

https://doi.org/10.1145/3134230.3134236

Published: 21 September 2017 Publication History

Abstract

Data gloves have numerous applications, including enabling novel human-computer interaction and automated recognition of large sets of gestures, such as those used for sign language. For most of these applications, it is important to build mobile and self-contained applications that run without the need for frequent communication with additional services on a back-end server. We present in this paper a data glove prototype, based on multiple small Inertial Measurement Units (IMUs), with a glove-embedded classifier for the french sign language. In an extensive set of experiments with 57 participants, our system was tested by repeatedly fingerspelling the French Sign Language (LSF) alphabet. Results show that our system is capable of detecting the LSF alphabet with a mean accuracy score of 92% and an F1 score of 91%, with all detections performed on the glove within 63 milliseconds.

References

[1]

Helene Brashear, Thad Starner, Paul Lukowicz, and Holger Junker. 2003. Using multiple sensors for mobile sign language recognition. In ISWC 2003. IEEE Press.

Digital Library

[2]

Maria E Cabrera, Juan Manuel Bogado, Leonardo Fermin, Raul Acuna, and Dimitar Ralev. 2012. Glove-based gesture recognition system. In Proc. of Intl. Conf. on climbing and walking robots and the support technologies for mobile machines. 747--753.

[3]

Pengfei Gui, Liqiong Tang, and Subhas Mukhopadhyay. 2015. MEMS based IMU for tilting measurement: Comparison of complementary and kalman filter based data fusion. In ICIEA 2015. IEEE, 2004--2009.

[4]

Christopher-Eyk Hrabia, Katrin Wolf, and Mathias Wilhelm. 2013. Whole hand modeling using 8 wearable sensors: biomechanics for hand pose prediction. In Proceedings of the 4th Augmented Human International Conference. ACM, 21--28.

Digital Library

[5]

Yu Huang, Dorothy Monekosso, Hui Wang, and Juan Carlos Augusto. 2011. A concept grounding approach for glove-based gesture recognition. In Intelligent Environments (IE), 2011 7th International Conference on. IEEE, 358--361.

Digital Library

[6]

Eunseok Jeong, Jaehong Lee, and DaeEun Kim. 2011. Finger-gesture recognition glove using velostat (ICCAS 2011). In Control, Automation and Systems (ICCAS), 2011 11th International Conference on. IEEE, 206--210.

[7]

Wu Jiangqin, Gao Wen, Song Yibo, Liu Wei, and Pang Bo. 1998. A simple sign language recognition system based on data glove. In Signal Processing Proceedings, 1998. ICSP'98. 1998 Fourth International Conference on, Vol. 2. IEEE, 1257--1260.

[8]

Kunal Kadam, Rucha Ganu, Ankita Bhosekar, and SD Joshi. 2012. American Sign Language Interpreter. In Technology for Education (T4E), 2012 IEEE Fourth International Conference on. IEEE, 157--159.

Digital Library

[9]

Yousuf Khambaty, Rolando Quintana, Mehdi Shadaram, Sana Nehal, Muneeb Ali Virk, Waqar Ahmed, and Ghayas Ahmedani. 2008. Cost effective portable system for sign language gesture recognition. In System of Systems Engineering, 2008. SoSE'08. IEEE International Conference on. IEEE, 1--6.

[10]

Jonghwa Kim, Johannes Wagner, Matthias Rehm, and Elisabeth André. 2008. Bi-channel sensor fusion for automatic sign language recognition. In Automatic Face & Gesture Recognition, 2008. FG'08. 8th IEEE International Conference on. IEEE, 1--6.

[11]

T Kuroda, Y Tabata, A Goto, H Ikuta, M Murakami, and others. 2004. Consumer price data-glove for sign language recognition. In Proc. of 5th Intl Conf. Disability, Virtual Reality Assoc. Tech., Oxford, UK. 253--258.

[12]

Cemil Oz and Ming C Leu. 2007. Linguistic properties based on American Sign Language isolated word recognition with artificial neural networks using a sensory glove and motion tracker. Neurocomputing 70, 16 (2007), 2891--2901.

Digital Library

[13]

Y. Park, J. Lee, and J. Bae. 2014. Development of a finger motion measurement system using linear potentiometers. In IEEE/ASME Intl. Conf. on Advanced Intelligent Mechatronics. 125--130.

[14]

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825--2830.

Digital Library

[15]

John Kangchun Perng, Brian D. Fisher, Seth Hollar, and Kristofer S. J. Pister. 1999. Acceleration Sensing Glove (ASG). In Proc. of Third International Symposium on Wearable Computers (ISWC 1999). 178--180.

Digital Library

[16]

David J Sturman and David Zeltzer. 1994. A survey of glove-based input. Computer Graphics and Applications, IEEE 14, 1 (1994), 30--39.

Digital Library

[17]

Netchanok Tanyawiwat and Surapa Thiemjarus. 2012. Design of an assistive communication glove using combined sensory channels. In Wearable and Implantable Body Sensor Networks (BSN), 2012 Ninth International Conference on. IEEE, 34--39.

Digital Library

[18]

Noor Tubaiz, Tamer Shanableh, and Khaled Assaleh. 2015. Glove-based continuous Arabic sign language recognition in user-dependent mode. Human-Machine Systems, IEEE Transactions on 45, 4 (2015), 526--533.

[19]

Satjakarn et al. Vutinuntakasame. 2011. An assistive body sensor network glove for speech-and hearing-impaired disabilities. In Body Sensor Networks (BSN), 2011 International Conference on. IEEE, 7--12.

Digital Library

[20]

Thomas G. Zimmerman, Jaron Lanier, Chuck Blanchard, Steve Bryson, and Young Harvill. 1987. A Hand Gesture Interface Device. In Proceedings of the SIGCHI/GI Conference on Human Factors in Computing Systems and Graphics Interface (CHI '87). ACM, New York, NY, USA, 189--192.

Digital Library

Cited By

Gangakhedkar RTheil A(2024)fingARspell: A Mobile AR Tool for Learning the Deafblind Manual AlphabetProceedings of the International Conference on Mobile and Ubiquitous Multimedia10.1145/3701571.3703390(478-480)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1145/3701571.3703390
Papadimitriou KSapountzaki GVasilaki KEfthimiou EFotinea SPotamianos G(2024)A large corpus for the recognition of Greek Sign Language gesturesComputer Vision and Image Understanding10.1016/j.cviu.2024.104212249(104212)Online publication date: Dec-2024
https://doi.org/10.1016/j.cviu.2024.104212
Thapa KSeo YYang SKim K(2023)Semi-Supervised Adversarial Auto-Encoder to Expedite Human Activity RecognitionSensors10.3390/s2302068323:2(683)Online publication date: 6-Jan-2023
https://doi.org/10.3390/s23020683
Show More Cited By

Index Terms

Real-time Embedded Recognition of Sign Language Alphabet Fingerspelling in an IMU-Based Glove
1. Human-centered computing

Recommendations

Recognition of sign language with an inertial sensor-based data glove
Recent Innovations on Biomedical Engineering

Communication between people with normal hearing and hearing impairment is difficult. Recently, a variety of studies on sign language recognition have presented benefits from the development of information technology. This study presents a sign language ...
Visualization of Grasping Operations based on Hand Kinematics measured through Data Glove
AIR '17: Proceedings of the 2017 3rd International Conference on Advances in Robotics

Although a number of prosthetic hands have been reported, anthropomorphic control is still a challenge. Precise determination of human hand kinematics will certainly enhance the control for prosthetic hands. One of the ways to push the research forward ...
A glove-based gesture recognition system for Vietnamese sign language
2015 15th International Conference on Control, Automation and Systems (ICCAS)
In this paper, we address a development of a glove-based gesture recognition system for Vietnamese sign language. A sensor glove is attached ten flex sensors and one accelerometer. Here, flex sensors are used for sensing the curvature of fingers and the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

iWOAR '17: Proceedings of the 4th International Workshop on Sensor-based Activity Recognition and Interaction

September 2017

83 pages

ISBN:9781450352239

DOI:10.1145/3134230

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

Rostock: University of Rostock

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 September 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Die Österreichische Forschungsförderungsgesellschaft FFG

Conference

iWOAR '17

iWOAR '17: 4th international Workshop on Sensor-based Activity Recognition and Interaction

September 21 - 22, 2017

Rostock, Germany

Acceptance Rates

iWOAR '17 Paper Acceptance Rate 12 of 19 submissions, 63%;

Overall Acceptance Rate 46 of 73 submissions, 63%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

26
Total Citations
View Citations
341
Total Downloads

Downloads (Last 12 months)38
Downloads (Last 6 weeks)8

Reflects downloads up to 22 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gangakhedkar RTheil A(2024)fingARspell: A Mobile AR Tool for Learning the Deafblind Manual AlphabetProceedings of the International Conference on Mobile and Ubiquitous Multimedia10.1145/3701571.3703390(478-480)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1145/3701571.3703390
Papadimitriou KSapountzaki GVasilaki KEfthimiou EFotinea SPotamianos G(2024)A large corpus for the recognition of Greek Sign Language gesturesComputer Vision and Image Understanding10.1016/j.cviu.2024.104212249(104212)Online publication date: Dec-2024
https://doi.org/10.1016/j.cviu.2024.104212
Thapa KSeo YYang SKim K(2023)Semi-Supervised Adversarial Auto-Encoder to Expedite Human Activity RecognitionSensors10.3390/s2302068323:2(683)Online publication date: 6-Jan-2023
https://doi.org/10.3390/s23020683
Hu YYeo HYuan MFan HElvitigala DHu WQuigley A(2023)MicroCamProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36109217:3(1-28)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610921
Kim GYeo DJo TRus DKim S(2023)What and When to Explain?Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108867:3(1-26)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610886
Wang JWang GZhang XLiu LZeng HXiao LCao ZGu LLi T(2023)PATCHProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108857:3(1-24)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610885
Shao QWang JZhou BTran VKrishnan GNayar S(2023)N-euro PredictorProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108847:3(1-25)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610884
Wang GHan LChang YShi YQian CZhao CDing HXi WZhao CZhao J(2023)Cross-technology Communication between Visible Light and Battery-free RFIDsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108837:3(1-20)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610883
Li JHuang LShah SJones SJin YWang DRussell AChoi SGao YYuan JJin Z(2023)SignRingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108817:3(1-29)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610881
Antillon DWalker CRosset SAnderson I(2023)Glove-Based Hand Gesture Recognition for Diver CommunicationIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2022.316168234:12(9874-9886)Online publication date: Dec-2023
https://doi.org/10.1109/TNNLS.2022.3161682
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten