research-article

Public Access

SignSpeaker: A Real-time, High-Precision SmartWatch-based Sign Language Translator

Authors:

Panlong YangAuthors Info & Claims

MobiCom '19: The 25th Annual International Conference on Mobile Computing and Networking

Article No.: 24, Pages 1 - 15

https://doi.org/10.1145/3300061.3300117

Published: 05 August 2019 Publication History

All formats PDF

Abstract

Sign language is a natural and fully-formed communication method for deaf or hearing-impaired people. Unfortunately, most of the state-of-the-art sign recognition technologies are limited by either high energy consumption or expensive device costs and have a difficult time providing a real-time service in a daily-life environment. Inspired by previous works on motion detection with wearable devices, we propose Sign Speaker - a real-time, robust, and user-friendly American sign language recognition (ASLR) system with affordable and portable commodity mobile devices. SignSpeaker is deployed on a smartwatch along with a smartphone; the smartwatch collects the sign signals and the smartphone outputs translation through an inbuilt loudspeaker. We implement a prototype system and run a series of experiments that demonstrate the promising performance of our system. For example, the average translation time is approximately $1.1$ seconds for a sentence with eleven words. The average detection ratio and reliability of sign recognition are 99.2% and 99.5%, respectively. The average word error rate of continuous sentence recognition is 1.04% on average.

References

[1]

M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G.S. Corrado, A. Davis, Jeffrey Dean, Matthieu Devin, et almbox. 2016. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 (2016).

Digital Library

[2]

K. Chen, S. Patel, and S. Keller. 2016. Finexus: Tracking Precise Motions of Multiple Fingertips Using Magnetic Sensing. In ACM CHI. ACM.

Digital Library

[3]

Y. Chen and C. Shen. 2017. Performance analysis of smartphone-sensor behavior for human activity recognition. IEEE Access, Vol. 5 (2017).

[4]

H. Cooper, B. Holt, and R. Bowden. 2011. Sign language recognition. In Visual Analysis of Humans. Springer.

[5]

C. Dong, M. Leu, and Z. Yin. 2015. American sign language alphabet recognition using microsoft kinect. In CVPRW.

[6]

D. Ekiz, G. Kaya, S. Buug ur, S. Güler, B. Buz, B. Kosucu, and B. Arnrich. 2017. Sign sentence recognition with smart watches. In IEEE SIU.

[7]

Rogerio Feris, Matthew Turk, R. Raskar, K. Tan, and G. Ohashi. 2004. Exploiting depth discontinuities for vision-based fingerspelling recognition. In IEEE CVPRW.

Digital Library

[8]

Google. {n. d.} a. Profile battery usage with Batterystats and Battery Historian. https://developer.android.com/studio/profile/battery-historian

[9]

Google. {n. d.} b. Sensors Overview. https://developer.android.com/guide/topics/sensors/sensors_overview

[10]

A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In ACM ICML.

Digital Library

[11]

A. Graves and N. Jaitly. 2014. Towards End-To-End Speech Recognition with Recurrent Neural Networks. In ICML, Vol. 14.

Digital Library

[12]

F. Grosjean and H. Lane. 1977. Pauses and syntax in American sign language. Cognition, Vol. 5, 2 (1977).

[13]

HLAA. 2017. Basic Facts About Hearing Loss. http://www.hearingloss.org/content/basic-facts-about-hearing-loss.

[14]

S. Hochreiter and J. Schmidhuber. 1997. Long short-term memory. Neural computation, Vol. 9, 8 (1997).

Digital Library

[15]

C. Hsu, C. Chang, C. Lin, et almbox. 2003. A practical guide to support vector classification. (2003).

[16]

M. Kadous et almbox. 1996. Machine recognition of Auslan signs using PowerGloves: Towards large-lexicon recognition of sign language. In Proceedings of the Workshop on the Integration of Gesture in Language and Speech. Citeseer.

[17]

D. Kingma and J. Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).

[18]

T. Kuroda, Y. Tabata, A. Goto, H. Ikuta, M. Murakami, et almbox. 2004. Consumer price data-glove for sign language recognition. In Proc. of 5th Intl Conf. Disability, Virtual Reality Assoc. Tech., Oxford, UK.

[19]

J. Kwapisz, G. Weiss, and S. Moore. 2011. Activity recognition using cell phone accelerometers. ACM SigKDD, Vol. 12, 2 (2011).

Digital Library

[20]

K. Lee, O. Levy, and L. Zettlemoyer. 2017. Recurrent Additive Networks. arXiv preprint arXiv:1705.07393 (2017).

[21]

T. Lei and Y. Zhang. 2017. Training RNNs as Fast as CNNs. arXiv preprint arXiv:1709.02755 (2017).

[22]

K. Li, Z. Zhou, and C. Lee. 2016. Sign transition modeling and a scalable solution to continuous sign language recognition for real-world applications. ACM TACCESS, Vol. 8, 2 (2016).

Digital Library

[23]

S. Liddell. 2003. Grammar, gesture, and meaning in American Sign Language.

[24]

Y. Ma, G. Zhou, S. Wang, H. Zhao, and W. Jung. 2018. SignFi: Sign Language Recognition Using WiFi. ACM IMWUT, Vol. 2, 1 (2018).

Digital Library

[25]

Microsoft. 2017. Kinect for Xbox. http://www.xbox.com/en-US/xbox-one/accessories/kinect.

[26]

M. Mohandes, M. Deriche, and J. Liu. 2014. Image-based and sensor-based approaches to Arabic sign language recognition. IEEE THMS, Vol. 44, 4 (2014).

[27]

M. Mohandes. 2013. Recognition of two-handed Arabic signs using the Cyber Glove. AJSE, Vol. 38, 3 (2013),

[28]

Leap Motion. 2017. Leap Motion. http://leapmotion.com.

[29]

R. Nandakumar, V. Iyer, D. Tan, and S. Gollakota. 2016. Finger IO: Using Active Sonar for Fine-Grained Finger Tracking. In ACM CHI.

Digital Library

[30]

World Federation of the Deaf. 2016. FAQ - WFD | World Federation of the Deaf. https://wfdeaf.org/faq.

[31]

L. Potter, J. Araullo, and L. Carter. 2013. The leap motion controller: a view on sign language. In ACM OzCHI.

Digital Library

[32]

Q. Pu, S. Gupta, S. Gollakota, and S. Patel. 2013. Whole-home gesture recognition using wireless signals. In ACM MobiCom.

Digital Library

[33]

Hisatake Sato. 2001. Moving average filter. US Patent 6,304,133.

[34]

N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from over fitting. Journal of Machine Learning Research, Vol. 15, 1 (2014).

Digital Library

[35]

T. Starner, J. Weaver, and A. Pentland. 1998. Real-time american sign language recognition using desk and wearable computer based video. IEEE TPAMI, Vol. 20, 12 (1998), 1371--1375.

Digital Library

[36]

D. Stockwell and A. Peterson. 2002. Effects of sample size on accuracy of species distribution models. Ecological modelling, Vol. 148, 1 (2002), 1--13.

[37]

L. Sun, D. Zhang, B. Li, B. Guo, and S. Li. 2010. Activity recognition on an accelerometer embedded mobile phone with varying positions and orientations. In Springer ICUIC.

Digital Library

[38]

R. Tennant and M. Brown. 1998. The American sign language handshape dictionary. Gallaudet University Press.

[39]

C. Valli and C. Lucas. 2000. Linguistics of American sign language: An introduction. Gallaudet University Press.

[40]

William Vicars. 2017. Basic ASL: First 100 Signs. http://www.lifeprint.com/asl101/pages-layout/concepts.htm.

[41]

C. Vogler and D. Metaxas. 1998. ASL recognition based on a coupling between HMMs and 3D motion analysis. In IEEE ICCV.

Digital Library

[42]

C. Wang, X. Guo, Y. Wang, Y. Chen, and B. Liu. 2016. Friend or foe?: Your wearable devices reveal your personal pin. In ACM AsiaCCS.

Digital Library

[43]

H. Wang, M. Leu, and C. Oz. 2006. American Sign Language Recognition Using Multi-dimensional Hidden Markov Models. JISE, Vol. 22, 5 (2006), 1109--1123.

[44]

J. Wang, D. Vasisht, and D. Katabi. 2014. textmdRF-IDraw: virtual touch screen in the air using textmdRF signals. In ACM SIGCOMM.

Digital Library

[45]

G. Welch and G. Bishop. 1995. An introduction to the Kalman filter. (1995).

[46]

H. Wen, J. Ramos Rojas, and A. Dey. 2016. Serendipity: Finger gesture recognition using an off-the-shelf smartwatch. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 3847--3851.

Digital Library

[47]

J. Wu, Z. Tian, L. Sun, L. Estevez, and R. Jafari. 2015. Real-time American sign language recognition using wrist-worn motion and surface EMG sensors. In IEEE BSN.

[48]

W. Wu, S. Dasgupta, E. Ramirez, C. Peterson, and G. Norman. 2012. Classification accuracies of physical activities using smartphone motion sensors. JMIR, Vol. 14, 5 (2012).

[49]

Z. Zafrulla, H. Brashear, T. Starner, H. Hamilton, and P. Presti. 2011a. American sign language recognition with the kinect. In ACM ICMI.

Digital Library

[50]

Z. Zafrulla, H. Brashear, T. Starner, H. Hamilton, and P. Presti. 2011b. American Sign Language Recognition with the Kinect. In ACM ICMI

Digital Library

[51]

J. Zhang, W. Zhou, C. Xie, J. Pu, and H. Li. 2016. Chinese sign language recognition with adaptive HMM. In IEEE ICME.

[52]

T. Zhao, J. Liu, Y. Wang, H. Liu, and Y. Chen. 2018. PPG-based finger-level gesture recognition leveraging wearables. In IEEE INFOCOM.

Cited By

Liu YZhang JChen YWang WYang SNa XSun YHe Y(2025)Real-Time Continuous Activity Recognition With a Commercial mmWave RadarIEEE Transactions on Mobile Computing10.1109/TMC.2024.348381324:3(1684-1698)Online publication date: Mar-2025
https://doi.org/10.1109/TMC.2024.3483813
C. Adornado LV. Latorre DB. Serrano AK. Masukat MA. Lontoc LLontoc A(2024)The use of TensorFlow Action Recognition as the Main Component in Making a Sign Language Translator Speaker for Speech-Impaired PeopleInternational Journal of Innovative Science and Research Technology (IJISRT)10.38124/ijisrt/IJISRT24APR744(1203-1210)Online publication date: 29-Apr-2024
https://doi.org/10.38124/ijisrt/IJISRT24APR744
Cherian JRay STaele PKoh JHammond T(2024)Exploring the Impact of the NULL Class on In-the-Wild Human Activity RecognitionSensors10.3390/s2412389824:12(3898)Online publication date: 16-Jun-2024
https://doi.org/10.3390/s24123898
Show More Cited By

Recommendations

Contemporary Issues in Handheld Computing Research

Mobile phones have become ubiquitous in today's society. However, mobile users are no longer satisfied with simple phones but instead expect ever more powerful functions to be available from their mobile devices. Advanced phones known as smartphones ...
10 years mobile multimedia: from Motorola RAZR to iPhone 5
IIWAS '12: Proceedings of the 14th International Conference on Information Integration and Web-based Applications & Services

The Motorola RAZR was first developed in July 2003 and released to the market in 2004 exactly around the same time MoMM conference series was launched by Johannes Kepler University Linz. The introduction of the Motorola RAZR became the most iconic ...
Mobile Web-Based System for Remote-Controlled Electronic Devices and Smart Objects

Nowadays there are many intelligent electronic devices in the everyday environments: appliances, industrial machinery, devices for service providers in the cities, etc. These electronic devices usually communicate with other devices and people in order ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MobiCom '19: The 25th Annual International Conference on Mobile Computing and Networking

August 2019

1017 pages

ISBN:9781450361699

DOI:10.1145/3300061

General Chairs:
Sharad Agarwal
Microsoft
,
Ben Greenstein
Google
,
Aruna Balasubramanian
Stony Brook University
,
Program Chairs:
Shyam Gollakota
University of Washington
,
Xinyu Zhang
University of California, San Diego

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMOBILE: ACM Special Interest Group on Mobility of Systems, Users, Data and Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 August 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

China National Funds for Distinguished Young Scientists
National Key R&D Program of China
NSFC
NSF
Key Research Program of Frontier Sciences, CAS

Conference

MobiCom '19

Sponsor:

SIGMOBILE

MobiCom '19: The 25th Annual International Conference on Mobile Computing and Networking

October 21 - 25, 2019

Los Cabos, Mexico

Acceptance Rates

Overall Acceptance Rate 440 of 2,972 submissions, 15%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

75
Total Citations
View Citations
3,119
Total Downloads

Downloads (Last 12 months)593
Downloads (Last 6 weeks)80

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Liu YZhang JChen YWang WYang SNa XSun YHe Y(2025)Real-Time Continuous Activity Recognition With a Commercial mmWave RadarIEEE Transactions on Mobile Computing10.1109/TMC.2024.348381324:3(1684-1698)Online publication date: Mar-2025
https://doi.org/10.1109/TMC.2024.3483813
C. Adornado LV. Latorre DB. Serrano AK. Masukat MA. Lontoc LLontoc A(2024)The use of TensorFlow Action Recognition as the Main Component in Making a Sign Language Translator Speaker for Speech-Impaired PeopleInternational Journal of Innovative Science and Research Technology (IJISRT)10.38124/ijisrt/IJISRT24APR744(1203-1210)Online publication date: 29-Apr-2024
https://doi.org/10.38124/ijisrt/IJISRT24APR744
Cherian JRay STaele PKoh JHammond T(2024)Exploring the Impact of the NULL Class on In-the-Wild Human Activity RecognitionSensors10.3390/s2412389824:12(3898)Online publication date: 16-Jun-2024
https://doi.org/10.3390/s24123898
Gu YOku HTodoh M(2024)American Sign Language Recognition and Translation Using Perception Neuron Wearable Inertial Motion Capture SystemSensors10.3390/s2402045324:2(453)Online publication date: 11-Jan-2024
https://doi.org/10.3390/s24020453
Luo XLiu DKong HHuai SChen HXiong GLiu W(2024)Efficient Deep Learning Infrastructures for Embedded Computing Systems: A Comprehensive Survey and Future EnvisionACM Transactions on Embedded Computing Systems10.1145/370172824:1(1-100)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3701728
Zhang JWang QWang Q(2024)A Sign Language Recognition Framework Based on Cross-Modal Complementary Information FusionIEEE Transactions on Multimedia10.1109/TMM.2024.337709526(8131-8144)Online publication date: 2024
https://doi.org/10.1109/TMM.2024.3377095
Ouyang JWu MLi XDeng HJin ZWu D(2024)NeuroBCI: Multi-Brain to Multi-Robot Interaction Through EEG-Adaptive Neural Networks and Semantic CommunicationsIEEE Transactions on Mobile Computing10.1109/TMC.2024.344682923:12(14622-14637)Online publication date: Dec-2024
https://doi.org/10.1109/TMC.2024.3446829
Liu XLi FCao YWang Y(2024) BrailleReader: Braille Character Recognition Using Wearable Motion Sensor IEEE Transactions on Mobile Computing10.1109/TMC.2024.337956923:11(10538-10553)Online publication date: Nov-2024
https://doi.org/10.1109/TMC.2024.3379569
Wang QZheng ZWang QDeng DZhang J(2024)Generalizations of Wearable Device Placements and Sentences in Sign Language Recognition With Transformer-Based ModelIEEE Transactions on Mobile Computing10.1109/TMC.2024.337347223:10(10046-10059)Online publication date: Oct-2024
https://doi.org/10.1109/TMC.2024.3373472
Li JXu JLiu YXu WLi Z(2024)Enhancing the Applicability of Sign Language TranslationIEEE Transactions on Mobile Computing10.1109/TMC.2024.335011123:9(8634-8648)Online publication date: Sep-2024
https://doi.org/10.1109/TMC.2024.3350111
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

EPUB

View this article in ePub.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten