research-article

Feedback Strategies for Embodied Agents to Enhance Sign Language Vocabulary Learning

Authors:

Patrick OlivierAuthors Info & Claims

IVA '20: Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents

Article No.: 47, Pages 1 - 8

https://doi.org/10.1145/3383652.3423871

Published: 19 October 2020 Publication History

Abstract

When learning sign language, feedback on accuracy is critical to vocabulary acquisition. When designing technologies to provide such visual feedback, we need to research effective ways to identify errors and present meaningful and effective feedback to learners. Motion capture technologies provide new opportunities to enhance sign language learning experiences through the presentation of visual feedback that indicates the accuracy of the signs made by learners. We designed, developed, and evaluated an embodied agent-based system for learning the location and gross motor movements of sign language vocabulary. The system presents a sign, tracks the learner's attempts at a sign, and provides visual feedback to the learner on their errors. We compared five different types of visual feedback, and in a study involving 51 participants we established that learners preferred visual feedback where their attempts at a sign were shown concurrently with the movements of the instructor with or without explicit corrections.

References

[1]

Waqar Ahmad, Aliya Darr, Lesley Jones, and Gohar Nisar. 1998. Deafness and ethnicity: Services, policy and politics. Cambridge, England: Cambridge University Press.

[2]

David Anderson, Dennis Sweeney, Thomas Williams, Jeffrey Camm, and James Cochran. 2013. Statistics for Business & Economics. Cengage Learning.

[3]

Oya Aran, Ismail Ari, Lale Akarun, Bülent Sankur, Alexandre Benoit, Alice Caplier, Pavel Campr, Ana Huerta Carrillo, et al. 2009. Signtutor: An interactive system for sign language tutoring. IEEE MultiMedia 1 (2009), 81--93.

Digital Library

[4]

John Bitchener. 2008. Evidence in support of written corrective feedback. Journal of Second Language Writing 17, 2 (2008), 102--118.

[5]

John D Bonvillian et al. 1993. Hand Preference in Young Children's Early Signing. (1993).

[6]

Marilyn Daniels. 2001. Dancing with words: Signing for hearing children's literacy. Bergin & Garvey.

[7]

Kirsten Ellis and Jan Carlo Barca. 2012. Exploring Sensor Gloves for Teaching Children Sign Language. Adv. in Hum.-Comp. Int. 2012, Article 12 (Jan. 2012), 8 pages. https://doi.org/10.1155/2012/210507

[8]

Kirsten Ellis and Kathy Blashki. 2004. Children, Australian sign language and the web; The possibilities. In AusWeb04: The 10th Australian World Wide Web Conference Proceedings. Southern Cross University.

[9]

Kirsten Ellis, Julie Fisher, Louisa Willoughby, and Jan Carlo Barca. 2015. A design science exploration of a visual-spatial learning system with feedback. Australasian Conference on Information Systems (2015).

[10]

Rod Ellis, Shawn Loewen, and Rosemary Erlam. 2006. Implicit and explicit corrective feedback and the acquisition of L2 grammar. Studies in Second Language Acquisition 28, 02 (2006), 339--368.

[11]

Julie Fisher, Kirsten Ellis, Louisa Willoughby, and Jan Carlo Barca. 2014. Taking a User Centred Design Approach for Designing a System to Teach Sign Language. ACIS, Auckland, New Zealand.

[12]

Donald A Grushkin. 1998. Lexidactylophobia: The (irrational) fear of finger-spelling. American Annals of the Deaf 143, 5 (1998), 404--415.

[13]

Rose Hatala, David A Cook, Benjamin Zendejas, Stanley J Hamstra, and Ryan Brydges. 2014. Feedback for simulation-based procedural skills training: a metaanalysis and critical narrative synthesis. Advances in Health Sciences Education 19, 2 (2014), 251--272.

[14]

Gertraud Havranek and Hermann Cesnik. 2001. Factors affecting the success of corrective feedback. EUROSLA yearbook 1, 1 (2001), 99--122.

[15]

Valerie Henderson, Seungyon Lee, Helene Brashear, Harley Hamilton, Thad Starner, and Steven Hamilton. 2005. Development of an American Sign Language game for deaf children. In Proceedings of the 2005 conference on Interaction Design and Children. ACM, 70--79.

Digital Library

[16]

Valerie Henderson-Summet, Kimberly Weaver, Tracy L Westeyn, and Thad E Starner. 2008. American sign language vocabulary: computer aided instruction for non-signers. In Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, 281--282.

Digital Library

[17]

Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics (1979), 65--70.

[18]

He Huang, Todd Ingalls, Loren Olson, Kathleen Ganley, Thanassis Rikakis, and Jiping He. 2005. Interactive multimodal biofeedback for task-oriented neural rehabilitation. In 27th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Vol. 3. 2547--2550.

[19]

Peter J Huber. 2011. Robust statistics. In International Encyclopedia of Statistical Science. Springer, 1248--1251.

[20]

Matt Huenerfauth, Elaine Gale, Brian Penly, Sree Pillutla, Mackenzie Willard, and Dhananjai Hariharan. 2017. Evaluation of Language Feedback Methods for Student Videos of American Sign Language. ACM Trans. Access. Comput. 10, 1, Article 2 (April 2017), 30 pages. https://doi.org/10.1145/3046788

Digital Library

[21]

Fumitada Itakura. 1975. Minimum prediction residual principle applied to speech recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing 23, 1 (1975), 67--72.

[22]

Noriko Iwashita. 2003. Negative feedback and positive evidence in task-based interaction. Studies in Second Language Acquisition 25, 01 (2003), 1--36.

[23]

Trevor Johnston and Adam Schembri. 2007. Australian Sign Language (Auslan): An introduction to sign language linguistics. Cambridge University Press, USA.

[24]

Kostas Karpouzis, George Caridakis, S-E Fotinea, and Eleni Efthimiou. 2007. Educational resources and implementation of a Greek sign language synthesis architecture. Computers & Education 49, 1 (2007), 54--74.

Digital Library

[25]

Tomaz Koritnik, Tadej Bajd, and Marko Munih. 2008. Virtual environment for lower-extremities training. Gait & posture 27, 2 (2008), 323--330.

[26]

Jeroen F Lichtenauer, Emile A Hendriks, and Marcel JT Reinders. 2008. Sign language recognition by combining statistical DTW and independent classification. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 11 (2008), 2040--2046.

Digital Library

[27]

Paul A Lyddon. 2011. The efficacy of corrective feedback and textual enhancement in promoting the acquisition of grammatical redundancies. The Modern Language Journal 95, s1 (2011), 104--129.

[28]

Laura Marchal-Crespo, Mark van Raai, Georg Rauter, Peter Wolf, and Robert Riener. 2013. The effect of haptic guidance and visual feedback on learning a complex tennis task. Experimental brain research 231, 3 (2013), 277--291.

[29]

Harold Pashler, Nicholas J Cepeda, John T Wixted, and Doug Rohrer. 2005. When does feedback facilitate learning of words? Journal of Experimental Psychology: Learning, Memory, and Cognition 31, 1 (2005), 3--8.

[30]

Han Duy Phan, Kirsten Ellis, and Alan Dorin. 2018. MIC, an interactive sign language teaching system. In Proceedings of the 30th Australian Conference on Computer-Human Interaction. ACM, 544--547.

Digital Library

[31]

Roland Sigrist, Georg Rauter, Robert Riener, and Peter Wolf. 2013. Augmented visual, auditory, haptic, and multimodal feedback in motor learning: a review. Psychonomic Bulletin & Review 20, 1 (2013), 21--53.

[32]

Bill Vicars. 2013. "100 Basic Signs" (American Sign Language) (www.lifeprint.com). https://www.youtube.com/watch?v=ianCxd71xIo. Accessed: 2019-01-09.

[33]

Louisa Willoughby, Stephanie Linder, Kirsten Ellis, and Julie Fisher. 2015. Errors and Feedback in the Beginner Auslan Classroom. Sign Language Studies 15, 3 (2015), 322--347.

[34]

Zahoor Zafrulla, Helene Brashear, Thad Starner, Harley Hamilton, and Peter Presti. 2011. American sign language recognition with the kinect. In Proceedings of the 13th International Conference on Multimodal Interfaces. ACM, 279--286.

Digital Library

[35]

Jihai Zhang, Wengang Zhou, and Houqiang Li. 2014. A threshold-based hmm-dtw approach for continuous sign language recognition. In Proceedings of International Conference on Internet Multimedia Computing and Service. ACM, 237.

Digital Library

Cited By

Gangakhedkar RTheil A(2024)fingARspell: A Mobile AR Tool for Learning the Deafblind Manual AlphabetProceedings of the International Conference on Mobile and Ubiquitous Multimedia10.1145/3701571.3703390(478-480)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1145/3701571.3703390
Wang JIvrissimtzis ILi ZZhou YShi L(2023)Exploring the Potential of Immersive Virtual Environments for Learning American Sign LanguageResponsive and Sustainable Educational Futures10.1007/978-3-031-42682-7_31(459-474)Online publication date: 28-Aug-2023
https://doi.org/10.1007/978-3-031-42682-7_31
Wang JIvrissimtzis ILi ZZhou YShi L(2023)User-Defined Hand Gesture Interface to Improve User Experience of Learning American Sign LanguageAugmented Intelligence and Intelligent Tutoring Systems10.1007/978-3-031-32883-1_43(479-490)Online publication date: 22-May-2023
https://doi.org/10.1007/978-3-031-32883-1_43
Show More Cited By

Index Terms

Feedback Strategies for Embodied Agents to Enhance Sign Language Vocabulary Learning
1. Human-centered computing
  1. Accessibility
    1. Empirical studies in accessibility
  2. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. Usability testing

Recommendations

MIC, an interactive sign language teaching system
OzCHI '18: Proceedings of the 30th Australian Conference on Computer-Human Interaction

We propose to demonstrate a novel sign language teaching system, My Interactive Coach (MIC). This system can teach learners Australian signs, allow them to practice, record their practiced performance, and provide different forms of visual feedback on ...
A teaching system of Japanese sign language using sign language recognition and generation
MULTIMEDIA '02: Proceedings of the tenth ACM international conference on Multimedia

In recent years, the number of sign language learners is increasing in Japan. And there are many teaching materials of sign language such as textbooks, videotapes and software for PCs. However, these teaching materials have several problems that ...
Integrating MultiWordNet with Italian Sign Language lexical resources

A novel Italian Sign Language MultiWordNet (LMWN), which integrates the MultiWordNet (MWN) lexical database with the Italian Sign Language (LIS), is presented in this paper. The approach relies on LIS lexical resources which support and help to search ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

IVA '20: Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents

October 2020

394 pages

ISBN:9781450375863

DOI:10.1145/3383652

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAI: ACM Special Interest Group on Artificial Intelligence

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

IVA '20

Sponsor:

SIGAI

IVA '20: ACM International Conference on Intelligent Virtual Agents

October 20 - 22, 2020

Scotland, Virtual Event, UK

Acceptance Rates

Overall Acceptance Rate 53 of 196 submissions, 27%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
155
Total Downloads

Downloads (Last 12 months)20
Downloads (Last 6 weeks)2

Reflects downloads up to 30 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gangakhedkar RTheil A(2024)fingARspell: A Mobile AR Tool for Learning the Deafblind Manual AlphabetProceedings of the International Conference on Mobile and Ubiquitous Multimedia10.1145/3701571.3703390(478-480)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1145/3701571.3703390
Wang JIvrissimtzis ILi ZZhou YShi L(2023)Exploring the Potential of Immersive Virtual Environments for Learning American Sign LanguageResponsive and Sustainable Educational Futures10.1007/978-3-031-42682-7_31(459-474)Online publication date: 28-Aug-2023
https://doi.org/10.1007/978-3-031-42682-7_31
Wang JIvrissimtzis ILi ZZhou YShi L(2023)User-Defined Hand Gesture Interface to Improve User Experience of Learning American Sign LanguageAugmented Intelligence and Intelligent Tutoring Systems10.1007/978-3-031-32883-1_43(479-490)Online publication date: 22-May-2023
https://doi.org/10.1007/978-3-031-32883-1_43
Prietch SSánchez JGuerrero J(2022)A Systematic Review of User Studies as a Basis for the Design of Systems for Automatic Sign Language ProcessingACM Transactions on Accessible Computing10.1145/356339515:4(1-33)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3563395
Hassan SLee SMetaxas DNeidle CHuenerfauth M(2022)Understanding ASL Learners’ Preferences for a Sign Language Recording and Automatic Feedback System to Support Self-StudyProceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3517428.3550367(1-5)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1145/3517428.3550367
Hossain SKamzin AAmperayani VPaudyal PBanerjee AGupta S(2021)Engendering Trust in Automated Feedback: A Two Step Comparison of Feedbacks in Gesture Based LearningArtificial Intelligence in Education10.1007/978-3-030-78292-4_16(190-202)Online publication date: 14-Jun-2021
https://dl.acm.org/doi/10.1007/978-3-030-78292-4_16

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten