research-article

AAC with Automated Vocabulary from Photographs: Insights from School and Speech-Language Therapy Settings

Authors:

Mauricio Fontana de Vargas,

Karyn MoffattAuthors Info & Claims

ASSETS '22: Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility

Article No.: 23, Pages 1 - 18

https://doi.org/10.1145/3517428.3544805

Published: 22 October 2022 Publication History

Abstract

Traditional symbol-based AAC devices impose meta-linguistic and memory demands on individuals with complex communication needs and hinder conversation partners from stimulating symbolic language in meaningful moments. This work presents a prototype application that generates situation-specific communication boards formed by a combination of descriptive, narrative, and semantic related words and phrases inferred automatically from photographs. Through semi-structured interviews with AAC professionals, we investigate how this prototype was used to support communication and language learning in naturalistic school and therapy settings. We find that the immediacy of vocabulary reduces conversation partners’ workload, opens up opportunities for AAC stimulation, and facilitates symbolic understanding and sentence construction. We contribute a nuanced understanding of how vocabularies generated automatically from photographs can support individuals with complex communication needs in using and learning symbolic AAC, offering insights into the design of automatic vocabulary generation methods and interfaces to better support various scenarios of use and goals.

References

[1]

Lawrence W Barsalou. 1999. Language comprehension: Archival memory or preparation for situated action?(1999).

[2]

David Beukelman, Jackie McGinnis, and Deanna Morrow. 1991. Vocabulary selection in augmentative and alternative communication. Augmentative and alternative communication 7, 3 (1991), 171–185.

[3]

David R Beukelman, Susan Fager, Laura Ball, and Aimee Dietz. 2007. AAC for adults with acquired neurological conditions: A review. Augmentative and alternative communication 23, 3 (2007), 230–242.

[4]

David R Beukelman, Karen Hux, Aimee Dietz, Miechelle McKelvey, and Kristy Weissling. 2015. Using visual scene displays as communication support options for people with chronic, severe aphasia: A summary of AAC research and future research directions. Augmentative and Alternative Communication 31, 3 (2015), 234–245.

[5]

David R Beukelman, Pat Mirenda, 1998. Augmentative and alternative communication. Paul H. Brookes Baltimore.

[6]

Filip Bircanin, Bernd Ploderer, Laurianne Sitbon, Andrew A Bayor, and Margot Brereton. 2019. Challenges and opportunities in using Augmentative and Alternative Communication (AAC) technologies: Design considerations for adults with severe disabilities. In Proceedings of the 31st Australian Conference on Human-Computer-Interaction. 184–196.

Digital Library

[7]

Rolf Black, Joseph Reddington, Ehud Reiter, Nava Tintarev, and Annalu Waller. 2010. Using NLG and sensors to support personal narrative for children with complex communication needs. In Proceedings of the NAACL HLT 2010 Workshop on Speech and Language Processing for Assistive Technologies. 1–9.

Digital Library

[8]

Sarah Blackstone, J Light, D Beukelman, and H Shane. 2004. Visual scene displays. Augmentative Communication News 16, 2 (2004), 1–16.

[9]

Nancy C Brady, Kandace Fleming, Kathy Thiemann-Bourque, Lesley Olswang, Patricia Dowden, Muriel D Saunders, and Janet Marquis. 2012. Development of the communication complexity scale. (2012).

[10]

Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77–101.

[11]

Virginia Braun and Victoria Clarke. 2019. Reflecting on reflexive thematic analysis. Qualitative research in sport, exercise and health 11, 4 (2019), 589–597.

[12]

Kris Brock, Rajinder Koul, Melinda Corwin, and Ralf Schlosser. 2017. A comparison of visual scene and grid displays for people with chronic aphasia: A pilot study to improve communication using AAC. Aphasiology 31, 11 (2017), 1282–1306.

[13]

John Seely Brown, Allan Collins, and Paul Duguid. 1989. Situated cognition and the culture of learning. Educational researcher 18, 1 (1989), 32–42.

[14]

Joan Bruno and David Trembath. 2006. Use of aided language stimulation to improve syntactic performance during a weeklong intervention program. Augmentative and Alternative Communication 22, 4 (2006), 300–313.

[15]

Shakila Dada and Erna Alant. 2009. The effect of aided language stimulation on vocabulary acquisition in children with little or no functional speech. (2009).

[16]

Jiamin Dai and Karyn Moffatt. 2020. Making Space for Social Sharing: Insights from a Community-Based Social Group for People with Dementia. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, 1–13.

Digital Library

[17]

Simon De Deyne, Steven Verheyen, Amy Perfors, and Daniel J Navarro. 2015. Evidence for widespread thematic structure in the mental lexicon. In CogSci.

[18]

Mauricio Fontana de Vargas and Karyn Moffatt. 2021. Automated Generation of Storytelling Vocabulary from Photographs for use in AAC. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 1353–1364.

[19]

Carrie Demmans Epp, Justin Djordjevic, Shimu Wu, Karyn Moffatt, and Ronald M Baecker. 2012. Towards providing just-in-time vocabulary support for assistive and augmentative communication. In Proceedings of the 2012 ACM international conference on Intelligent User Interfaces. 33–36.

Digital Library

[20]

Emma Dixon and Amanda Lazar. 2020. Approach Matters: Linking Practitioner Approaches to Technology Design for People with Dementia. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, 1–15.

Digital Library

[21]

Kathryn DR Drager, Janice Light, Jessica Currall, Nimisha Muttiah, Vanessa Smith, Danielle Kreis, Alyssa Nilam-Hall, Daniel Parratt, Kaitlin Schuessler, Kaitlin Shermetta, 2019. AAC technologies with visual scene displays and “just in time” programming and symbolic communication turns expressed by students with severe disability. Journal of intellectual & developmental disability 44, 3(2019), 321–336.

[22]

Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh K Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, 2015. From captions to visual concepts and back. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1473–1482.

[23]

Brenda Fossett and Pat Mirenda. 2007. Augmentative and alternative communication. Handbook on developmental disabilities(2007), 330–348.

[24]

Carol Goossens’. 1989. Aided communication intervention before assessment: A case study of a child with cerebral palsy. Augmentative and Alternative Communication 5, 1 (1989), 14–26.

[25]

Jessica Gosnell, John Costello, and Howard Shane. 2011. Using a clinical approach to answer “What communication apps should we use?”. Perspectives on augmentative and alternative communication 20, 3(2011), 87–96.

[26]

Audrey L Holland. 1975. Language therapy for children: Some thoughts on context and content. Journal of Speech and Hearing Disorders 40, 4 (1975), 514–523.

[27]

Ting-Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, 2016. Visual storytelling. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 1233–1239.

[28]

Eunsook Hyun and J Dan Marshall. 2003. Teachable-moment-oriented curriculum practice in early childhood education. Journal of Curriculum Studies 35, 1 (2003), 111–127.

[29]

Shaun K Kane, Barbara Linam-Church, Kyle Althoff, and Denise McCall. 2012. What we talk about: Designing a context-aware communication tool for people with aphasia. In Proceedings of the 14th international ACM SIGACCESS conference on Computers and accessibility. 49–56.

Digital Library

[30]

Per Ola Kristensson, James Lilley, Rolf Black, and Annalu Waller. 2020. A Design Engineering Approach for Quantitatively Exploring Context-Aware Sentence Retrieval for Nonspeaking Individuals with Motor Disabilities. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–11.

Digital Library

[31]

Amanda Lazar, Caroline Edasis, and Anne Marie Piper. 2017. Supporting people with dementia in digital social sharing. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 2149–2162.

Digital Library

[32]

Amanda Lazar, Jessica L. Feuston, Caroline Edasis, and Anne Marie Piper. 2018. Making as Expression: Informing Design with People with Complex Communication Needs through Art Therapy. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 1–16.

Digital Library

[33]

Janice Light. 1988. Interaction involving individuals using augmentative and alternative communication systems: State of the art and future directions. Augmentative and alternative communication 4, 2 (1988), 66–82.

[34]

Janice Light. 1989. Toward a definition of communicative competence for individuals using augmentative and alternative communication systems. Augmentative and alternative communication 5, 2 (1989), 137–144.

[35]

Janice Light, Kathryn Drager, John McCarthy, Suzanne Mellott, Diane Millar, Craig Parrish, Arielle Parsons, Stacy Rhoads, Maricka Ward, and Michelle Welliver. 2004. Performance of typically developing four-and five-year-old children with AAC systems using different language organization techniques. Augmentative and Alternative Communication 20, 2 (2004), 63–88.

[36]

Janice Light, David McNaughton, and Jessica Caron. 2019. New and emerging AAC technology supports for children with complex communication needs and their communication partners: State of the science and future research directions. Augmentative and Alternative Communication 35, 1 (2019), 26–41.

[37]

Janice Light, Krista M Wilkinson, Amber Thiessen, David R Beukelman, and Susan Koch Fager. 2019. Designing effective AAC displays for individuals with developmental or acquired disabilities: State of the science and future research directions. Augmentative and Alternative Communication 35, 1 (2019), 42–55.

[38]

Eugene T McDonald and Adeline R Schultz. 1973. Communication boards for cerebral-palsied children. Journal of Speech and Hearing Disorders 38, 1 (1973), 73–88.

[39]

Aimee Mooney, Steven Bedrick, Glory Noethe, Scott Spaulding, and Melanie Fried-Oken. 2018. Mobile technology to support lexical retrieval during activity retell in primary progressive aphasia. Aphasiology 32, 6 (2018), 666–692.

[40]

Timothy Neate, Abi Roper, Stephanie Wilson, and Jane Marshall. 2019. Empowering expression for users with aphasia through constrained creativity. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–12.

Digital Library

[41]

Nhu-Van Nguyen, Christophe Rigaud, and Jean-Christophe Burie. 2017. Comic characters detection using deep learning. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), Vol. 3. IEEE, 41–46.

[42]

Christopher S Norrie, Annalu Waller, and Elizabeth FS Hannah. 2021. Establishing Context: AAC Device Adoption and Support in a Special-Education Setting. ACM Transactions on Computer-Human Interaction (TOCHI) 28, 2(2021), 1–30.

Digital Library

[43]

Mmachi God’sglory Obiorah, Anne Marie Marie Piper, and Michael Horn. 2021. Designing AACs for People with Aphasia Dining in Restaurants. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–14.

Digital Library

[44]

Rupal Patel and Rajiv Radhakrishnan. 2007. Enhancing Access to Situational Vocabulary by Leveraging Geographic Context.Assistive Technology Outcomes and Benefits 4, 1 (2007), 99–114.

[45]

Ehud Reiter. 2007. An architecture for data-to-text systems. In Proceedings of the Eleventh European Workshop on Natural Language Generation (ENLG 07). 97–104.

Digital Library

[46]

Ralf W Schlosser, Howard C Shane, Anna A Allen, Jennifer Abramson, Emily Laubscher, and Katherine Dimery. 2016. Just-in-time supports in augmentative and alternative communication. Journal of Developmental and Physical Disabilities 28, 1(2016), 177–193.

[47]

Samuel C Sennott, Janice C Light, and David McNaughton. 2016. AAC modeling intervention research review. Research and Practice for Persons with Severe Disabilities 41, 2(2016), 101–115.

[48]

H Shane and J Costello. 1994. Augmentative communication assessment and the feature matching process. In Mini-seminar presented at the annual convention of the American Speech-Language-Hearing Association, New Orleans, LA.

[49]

Martine M Smith. 2015. Language development of individuals who require aided communication: Reflections on state of the science and future research directions. Augmentative and Alternative Communication 31, 3 (2015), 215–233.

[50]

Carla Tamburro, Timothy Neate, Abi Roper, and Stephanie Wilson. 2020. Accessible Creativity with a Comic Spin. ASSETS 2020 - 22nd International ACM SIGACCESS Conference on Computers and Accessibility, 1–11.

Digital Library

[51]

Jennifer J Thistle and Krista M Wilkinson. 2013. Working memory demands of aided augmentative and alternative communication for individuals with developmental disabilities. Augmentative and Alternative Communication 29, 3 (2013), 235–245.

[52]

Nava Tintarev, Ehud Reiter, Rolf Black, and Annalu Waller. 2014. Natural language generation for augmentative and assistive technologies. In Natural Language Generation in Interactive Systems. Cambridge University Press, 252–277.

[53]

Nava Tintarev, Ehud Reiter, Rolf Black, Annalu Waller, and Joe Reddington. 2016. Personal storytelling: Using Natural Language Generation for children with complex communication needs, in the wild…. International Journal of Human-Computer Studies 92 (2016), 1–16.

Digital Library

[54]

Sarah E Wallace and Karen Hux. 2014. Effect of two layouts on high technology AAC navigation and content location by people with aphasia. Disability and Rehabilitation: Assistive Technology 9, 2 (2014), 173–182.

[55]

Bruce Wisenburn and D Jeffery Higginbotham. 2008. An AAC application using speaking partner speech recognition to automatically produce contextually relevant utterances: Objective results. Augmentative and alternative communication 24, 2 (2008), 100–109.

[56]

Bruce Wisenburn and D Jeffery Higginbotham. 2009. Participant evaluations of rate and communication efficacy of an AAC application using natural language processing. Augmentative and Alternative Communication 25, 2 (2009), 78–89.

[57]

Di Wu, Si-Jia Zheng, Xiao-Ping Zhang, Chang-An Yuan, Fei Cheng, Yang Zhao, Yong-Jun Lin, Zhong-Qiu Zhao, Yong-Li Jiang, and De-Shuang Huang. 2019. Deep learning-based methods for person re-identification: A comprehensive review. Neurocomputing 337(2019), 354–371.

Digital Library

[58]

Yi Zheng, Yifan Zhao, Mengyuan Ren, He Yan, Xiangju Lu, Junhui Liu, and Jia Li. 2020. Cartoon face recognition: A benchmark dataset. In Proceedings of the 28th ACM international conference on multimedia. 2264–2272.

Digital Library

Cited By

Curtis HLau YNeate T(2024)Breaking Badge: Augmenting Communication with Wearable AAC Smartbadges and DisplaysProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642327(1-25)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642327
Curtis HNeate T(2024)Beyond Repairing with Electronic Speech: Towards Embodied Communication and Assistive TechnologyProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642274(1-12)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642274
Fontana De Vargas MYu CShane HMoffatt K(2024)Co-Designing QuickPic: Automated Topic-Specific Communication Boards from Photographs for AAC-Based Language InstructionProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642080(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642080
Show More Cited By

Index Terms

AAC with Automated Vocabulary from Photographs: Insights from School and Speech-Language Therapy Settings
1. Applied computing
2. Human-centered computing
  1. Human computer interaction (HCI)

Index terms have been assigned to the content through auto-classification.

Recommendations

Co-Designing QuickPic: Automated Topic-Specific Communication Boards from Photographs for AAC-Based Language Instruction
CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems

Traditional topic-specific communication boards for Augmentative and Alternative Communication (AAC) require manual programming of relevant symbolic vocabulary, which is time-consuming and often impractical even for experienced Speech-Language ...
Using Technology-Based Augmentative and Alternative Communication to Develop Adaptive Behavior in a Child with Autism Spectrum Disorders – An Intervention Program
Universal Access in Human-Computer Interaction
Abstract
Individuals with autism face lots of difficulties in developing their communication skills, which affect their socialization while also increase their challenged behavior. Consequently, many individuals with autism are candidates for Augmentative ...
Assistive Technologies, Digital Literacy and Didactic for Inclusion

This article aims to investigate the relationship between technologies and disabilities in the field of special education. In particular, this article discusses the role of aiding technologies, such as augmentative and alternative communication (AAC), ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ASSETS '22: Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility

October 2022

902 pages

ISBN:9781450392587

DOI:10.1145/3517428

Editors:
Jon Froehlich
University of Washington, USA
,
Kristen Shinohara
Rochester Institute of Technology, USA
,
Stephanie Ludi
University of North Texas, USA

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGACCESS: ACM Special Interest Group on Accessible Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 October 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Best Student Paper

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Conference

ASSETS '22

Sponsor:

SIGACCESS

ASSETS '22: The 24th International ACM SIGACCESS Conference on Computers and Accessibility

October 23 - 26, 2022

Athens, Greece

Acceptance Rates

ASSETS '22 Paper Acceptance Rate 35 of 132 submissions, 27%;

Overall Acceptance Rate 436 of 1,556 submissions, 28%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
470
Total Downloads

Downloads (Last 12 months)209
Downloads (Last 6 weeks)10

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Curtis HLau YNeate T(2024)Breaking Badge: Augmenting Communication with Wearable AAC Smartbadges and DisplaysProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642327(1-25)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642327
Curtis HNeate T(2024)Beyond Repairing with Electronic Speech: Towards Embodied Communication and Assistive TechnologyProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642274(1-12)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642274
Fontana De Vargas MYu CShane HMoffatt K(2024)Co-Designing QuickPic: Automated Topic-Specific Communication Boards from Photographs for AAC-Based Language InstructionProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642080(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642080
Ludi SShinohara K(2023)Overview of Assets 2022ACM SIGACCESS Accessibility and Computing10.1145/3629723.3629724(1-1)Online publication date: 20-Oct-2023
https://dl.acm.org/doi/10.1145/3629723.3629724
Valencia SCave RKallarackal KSeaver KTerry MKane S(2023)“The less I type, the better”: How AI Language Models can Enhance or Impede Communication for AAC UsersProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581560(1-14)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581560

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents