research-article

An integrated RGB-D system for looking up the meaning of signs

Authors:

Christopher Conly,

Vassilis AthitsosAuthors Info & Claims

PETRA '15: Proceedings of the 8th ACM International Conference on PErvasive Technologies Related to Assistive Environments

Article No.: 24, Pages 1 - 8

https://doi.org/10.1145/2769493.2769534

Published: 01 July 2015 Publication History

Abstract

Users of written languages have the ability to quickly and easily look up the meaning of an unknown word. Those who use sign languages, however, lack this advantage, and it can be a challenge to find the meaning of an unknown sign. While some sign-to-written language dictionaries do exist, they are cumbersome and slow to use. We present an improved American Sign Language video dictionary system that allows a user to perform an unknown sign in front of a sensor and quickly retrieve a ranked list of similar signs with a video example of each. Earlier variants of the system required the use of a separate piece of software to record the query sign, as well as user intervention to provide bounding boxes for the hands and face in the first frame of the sign. The system presented here integrates all functionality into one piece of software and automates head and hand detection with the use of an RGB-D sensor, eliminating some of the shortcomings of the previous system, while improving match accuracy and shortening the time required to perform a query.

References

[1]

V. Athitsos, C. Neidle, S. Sclaroff, J. Nash, A. Stefan, Q. Yuan, and A. Thangali. The american sign language lexicon video dataset. In Computer Vision and Pattern Recognition Workshops, 2008. CVPRW'08. IEEE Computer Society Conference on, pages 1--8. IEEE, 2008.

[2]

B. Bauer, H. Hienz, and K.-F. Kraiss. Video-based continuous sign language recognition using statistical methods. In Pattern Recognition, 2000. Proceedings. 15th International Conference on, volume 2, pages 463--466. IEEE, 2000.

[3]

D. Bragg, K. Rector, and R. E. Ladner. A user-powered american sign language dictionary. 2015.

Digital Library

[4]

Y. Cui and J. Weng. Appearance-based hand sign recognition from intensity image sequences. Computer Vision and Image Understanding, 78(2):157--176, 2000.

Digital Library

[5]

P. Dreuw, T. Deselaers, D. Keysers, and H. Ney. Modeling image variability in appearance-based gesture recognition. In ECCV Workshop on Statistical Methods in Multi-Image and Video Processing, pages 7--18, 2006.

[6]

R. Elliott, H. Cooper, E.-J. Ong, J. Glauert, R. Bowden, and F. Lefebvre-Albaret. Search-by-example in multilingual sign language databases. In Proc. Sign Language Translation and Avatar Technologies Workshops, 2011.

[7]

S. Escalera, J. González, X. Baró, M. Reyes, O. Lopes, I. Guyon, V. Athitsos, and H. Escalante. Multi-modal gesture recognition challenge 2013: Dataset and results. In Proceedings of the 15th ACM on International conference on multimodal interaction, pages 445--452. ACM, 2013.

Digital Library

[8]

T. Kadir, R. Bowden, E.-J. Ong, and A. Zisserman. Minimal training, large lexicon, unconstrained sign language recognition. In BMVC, pages 1--10, 2004.

[9]

J. B. Kruskal and M. Liberman. The symmetric time warping algorithm: From continuous to discrete. In Time Warps. Addison-Wesley, 1983.

[10]

R.-H. Liang and M. Ouhyoung. A real-time continuous gesture recognition system for sign language. In Automatic Face and Gesture Recognition, 1998. Proceedings. Third IEEE International Conference on, pages 558--567. IEEE, 1998.

Digital Library

[11]

Microsoft.com. Developing with kinect, 2015.

[12]

Microsoft.com. Kinect for windows, 2015.

[13]

Opencv.org. Opencv, 2015.

[14]

G. Pavlakos, S. Theodorakis, V. Pitsikalis, S. Katsamanis, and P. Maragos. Kinect-based multimodal gesture recognition using a two-pass fusion scheme. In Proc. IntâĂ&Zacute;l Conf. on Image Processing, 2014.

[15]

T. Pfister, J. Charles, and A. Zisserman. Domain-adaptive discriminative one-shot learning of gestures. In Computer Vision--ECCV 2014, pages 814--829. Springer, 2014.

[16]

Qt-project.org. Qt project, 2015.

[17]

A. Roussos, S. Theodorakis, V. Pitsikalis, and P. Maragos. Dynamic affine-invariant shape-appearance handshape features and classification in sign language videos. The Journal of Machine Learning Research, 14(1):1627--1663, 2013.

Digital Library

[18]

J. Shotton, R. Girshick, A. Fitzgibbon, T. Sharp, M. Cook, M. Finocchio, R. Moore, P. Kohli, A. Criminisi, A. Kipman, and A. Blake. Efficient human pose estimation from single depth images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(12):2821--2840, 2013.

Digital Library

[19]

A. Stefan, H. Wang, and V. Athitsos. Towards automated large vocabulary gesture search. In Proceedings of the 2nd International Conference on PErvasive Technologies Related to Assistive Environments, pages 16:1--16:8. ACM, 2009.

Digital Library

[20]

R. Tennant. American Sign Language handshape dictionary. Gallaudet University Press, Washington, D.C, 2010.

[21]

C. Vogler and D. Metaxas. Parallel hidden markov models for american sign language recognition. In Computer Vision, 1999. The Proceedings of the Seventh IEEE International Conference on, volume 1, pages 116--122. IEEE, 1999.

[22]

H. Wang, A. Stefan, S. Moradi, V. Athitsos, C. Neidle, and F. Kamangar. A system for large vocabulary sign search. In Proceedings of the 11th European conference on Trends and Topics in Computer Vision - Volume Part I, ECCV'10, pages 342--353, Berlin, Heidelberg, 2012. Springer-Verlag.

Digital Library

[23]

R. Y. Wang and J. Popović. Real-time hand-tracking with a color glove. ACM Transactions on Graphics (TOG), 28(3):63:1--63:8, 2009.

Digital Library

[24]

M.-H. Yang and N. Ahuja. Recognizing hand gestures using motion trajectories. In Face Detection and Gesture Recognition for Human-Computer Interaction, pages 53--81. Springer, 2001.

[25]

G. Yao, H. Yao, X. Liu, and F. Jiang. Real time large vocabulary continuous sign language recognition based on op/viterbi algorithm. In Pattern Recognition, 2006. ICPR 2006. 18th International Conference on, volume 3, pages 312--315. IEEE, 2006.

Digital Library

[26]

Z. Zafrulla, H. Brashear, T. Starner, H. Hamilton, and P. Presti. American sign language recognition with the kinect. In Proceedings of the 13th international conference on multimodal interfaces, pages 279--286. ACM, 2011.

Digital Library

[27]

J. Zieren and K.-F. Kraiss. Robust person-independent visual sign language recognition. In Pattern recognition and image analysis, pages 520--528. Springer, 2005.

Digital Library

Cited By

Hassan Sde Lacerda Pataca CAmin ANourian LNavarro DLee SGordon AWatkins MTigwell GHuenerfauth M(2024)Exploring the Benefits and Applications of Video-Span Selection and Search for Real-Time Support in Sign Language Video Comprehension among ASL LearnersACM Transactions on Accessible Computing10.1145/369064717:3(1-35)Online publication date: 4-Oct-2024
https://dl.acm.org/doi/10.1145/3690647
Hassan SAmin Ade Lacerda Pataca CNavarro DGordon ALee SHuenerfauth M(2022)Support in the Moment: Benefits and use of video-span selection and search for sign-language video comprehension among ASL learnersProceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3517428.3544883(1-14)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1145/3517428.3544883
Hassan SAmin AGordon ALee SHuenerfauth M(2022)Design and Evaluation of Hybrid Search for American Sign Language to English Dictionaries: Making the Most of Imperfect Sign RecognitionProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501986(1-13)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3501986
Show More Cited By

Index Terms

An integrated RGB-D system for looking up the meaning of signs
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Computer graphics
    1. Animation
      1. Motion capture
      2. Motion processing

Recommendations

An evaluation of RGB-D skeleton tracking for use in large vocabulary complex gesture recognition
PETRA '14: Proceedings of the 7th International Conference on PErvasive Technologies Related to Assistive Environments

An essential component of any hand gesture recognition system is the hand detector and tracker. While a system with a small vocabulary of sufficiently dissimilar gestures may work well with approximate estimations of hand locations, more accurate hand ...
Toward a 3D body part detection video dataset and hand tracking benchmark
PETRA '13: Proceedings of the 6th International Conference on PErvasive Technologies Related to Assistive Environments

The purpose of this paper is twofold. First, we introduce our Microsoft Kinect--based video dataset of American Sign Language (ASL) signs designed for body part detection and tracking research. This dataset allows researchers to experiment with using ...
Kinect-based Taiwanese sign-language recognition system

Gesture-recognition is an important component for many intelligent human---computer interaction applications. For example, a realtime sign-language recognition system would detect and interpret hand gestures. Many vision-based sign-language recognition ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

PETRA '15: Proceedings of the 8th ACM International Conference on PErvasive Technologies Related to Assistive Environments

July 2015

526 pages

ISBN:9781450334525

DOI:10.1145/2769493

Conference Chair:
Fillia Makedon
University of Texas Arlington
,
Program Chairs:
Gian-Luca Mariottini,
Oliver Korn,
Illias Maglogiannis,
Vangelis Metsis

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

NSF: National Science Foundation
University of Texas at Austin: University of Texas at Austin
Univ. of Piraeus: University of Piraeus
NCRS: Demokritos National Center for Scientific Research
Ionian: Ionian University, GREECE

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 July 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

PETRA '15

Sponsor:

NSF
University of Texas at Austin
Univ. of Piraeus
NCRS
Ionian

PETRA '15: 8th PErvasive Technologies Related to Assistive Environments

July 1 - 3, 2015

Corfu, Greece

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
132
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Hassan Sde Lacerda Pataca CAmin ANourian LNavarro DLee SGordon AWatkins MTigwell GHuenerfauth M(2024)Exploring the Benefits and Applications of Video-Span Selection and Search for Real-Time Support in Sign Language Video Comprehension among ASL LearnersACM Transactions on Accessible Computing10.1145/369064717:3(1-35)Online publication date: 4-Oct-2024
https://dl.acm.org/doi/10.1145/3690647
Hassan SAmin Ade Lacerda Pataca CNavarro DGordon ALee SHuenerfauth M(2022)Support in the Moment: Benefits and use of video-span selection and search for sign-language video comprehension among ASL learnersProceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3517428.3544883(1-14)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1145/3517428.3544883
Hassan SAmin AGordon ALee SHuenerfauth M(2022)Design and Evaluation of Hybrid Search for American Sign Language to English Dictionaries: Making the Most of Imperfect Sign RecognitionProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501986(1-13)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3501986
Hassan SAlonzo OGlasser AHuenerfauth M(2021)Effect of Sign-recognition Performance on the Usability of Sign-language Dictionary SearchACM Transactions on Accessible Computing10.1145/347065014:4(1-33)Online publication date: 31-Dec-2021
https://dl.acm.org/doi/10.1145/3470650
Alonzo OGlasser AHuenerfauth MBigham JAzenkot SKane S(2019)Effect of Automatic Sign Recognition Performance on the Usability of Video-Based Search Interfaces for Sign Language DictionariesProceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3308561.3353791(56-67)Online publication date: 24-Oct-2019
https://dl.acm.org/doi/10.1145/3308561.3353791
Escalera SAthitsos VGuyon I(2017)Challenges in Multi-modal Gesture RecognitionGesture Recognition10.1007/978-3-319-57021-1_1(1-60)Online publication date: 20-Jul-2017
https://doi.org/10.1007/978-3-319-57021-1_1
Yauri Vidalón JDe Martino J(2016)Brazilian Sign Language Recognition Using KinectComputer Vision – ECCV 2016 Workshops10.1007/978-3-319-48881-3_27(391-402)Online publication date: 3-Nov-2016
https://doi.org/10.1007/978-3-319-48881-3_27

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten