research-article

Open access

Designing a Customizable Picture-Based Augmented Reality Application For Therapists and Educational Professionals Working in Autistic Contexts

Authors:

Amanda O'Brien,

Ralf W Schlosser,

Howard C. Shane,

Dylan Oesch-Emmel,

Eileen T. Crehan,

Fahad DogarAuthors Info & Claims

ASSETS '22: Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility

Article No.: 21, Pages 1 - 16

https://doi.org/10.1145/3517428.3544884

Published: 22 October 2022 Publication History

All formats PDF

Abstract

This paper presents the design and evaluation of CustomAR – a customizable Augmented Reality (AR) application, designed in collaboration with therapists, that allows them to create and customize picture-based AR experiences for use in an autistic context. Using a 2-week diary study, we gauge whether the application’s customization options and features are sufficient to allow therapists and educational professionals to create AR experiences for the various learning activities they conduct with autistic children, and what challenges they face in this regard. We find that participants think the application would be suitable for creating AR experiences for a wide range of learning activities, such as choice-making and teaching daily living skills, and think that the application’s freeze feature can be helpful when working with children with limited attention. Towards the end of the paper we discuss the challenges users face when trying to incorporate picture-based AR in practical therapy exercises, and how they can be addressed.

References

[1]

2017. Daily living skills: Strategies to help sequence & achieve personal hygiene tasks. https://www.toolstogrowot.com/blog/2017/08/09/daily-living-skills-strategies-to-help-sequence-achieve-personal-hygiene-tasks

[2]

2021. AugmentedClass! augmented reality for education - apps on Google Play. https://play.google.com/store/apps/details?id=com.AugmentedClass.AClass

[3]

2021. The benefits of visual supports for children with autism. https://www.autismparentingmagazine.com/benefits-of-autism-visual-supports/

[4]

2021. Qualtrics XM - experience management software. https://www.qualtrics.com/

[5]

2021. What is Visual Scheduling?https://www.appliedbehavioranalysisprograms.com/faq/what-is-visual-scheduling/#:~:text=Visual scheduling is a systematic,and achieve success in life.

[6]

2022. About AR Foundation: AR FOUNDATION: 4.1.7. https://docs.unity3d.com/Packages/[email protected]/manual/index.html

[7]

2022. Apple. https://testflight.apple.com/

[8]

2022. Build new augmented reality experiences that seamlessly blend the digital and physical worlds. https://developers.google.com/ar

[9]

2022. Video conferencing, Cloud Phone, WEBINARS, Chat, Virtual EVENTS: ZOOM. https://zoom.us/

[10]

2022. What is Amazon S3? - amazon simple storage service. https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html

[11]

Mohammed Alzyoudi, AbedAlziz Sartawi, and Osha Almuhiri. 2015. The impact of video modelling on improving social skills in children with autism. British Journal of Special Education 42, 1, 53–68.

[12]

Zhen Bai, Alan F Blackwell, and George Coulouris. 2013. Through the looking glass: Pretend play for children with autism. In 2013 IEEE International Symposium on Mixed and augmented reality (ISMAR). IEEE, 49–58.

[13]

Zhen Bai, Alan F Blackwell, and George Coulouris. 2015. Exploring expressive augmented reality: The FingAR puppet system for social pretend play. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 1035–1044.

Digital Library

[14]

Andy Bondy and Lori Frost. 2011. A picture’s worth: PECS and other visual communication strategies in autism. Woodbine House.

[15]

Monique Botha, Jacqueline Hanlon, and Gemma Louise Williams. 2021. Does language matter? Identity-first versus person-first language use in autism research: A response to Vivanti. Journal of Autism and Developmental Disorders, 1–9.

[16]

Brian A Boyd, Maureen A Conroy, G Richmond Mancil, Taketo Nakao, and Peter J Alter. 2007. Effects of circumscribed interests on the social behaviors of children with autism spectrum disorders. Journal of autism and developmental disorders 37, 8, 1550–1561.

[17]

Jorge Brandão, Pedro Cunha, José Vasconcelos, Vítor Carvalho, and Filomena Soares. 2015. An augmented reality gamebook for children with autism spectrum disorders. In The International Conference on E-Learning in the Workplace. 1–6.

[18]

Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2, 77–101.

[19]

Chien-Hsu Chen, I-Jui Lee, and Ling-Yi Lin. 2016. Augmented reality-based video-modeling storybook of nonverbal facial cues for children with autism spectrum disorder to improve their perceptions and judgments of facial expressions and emotions. Computers in Human Behavior 55, 477–485.

Digital Library

[20]

Chi-Hsuan Chung and Chien-Hsu Chen. 2017. Augmented reality based social stories training system for promoting the social skills of children with autism. In Advances in ergonomics modeling, usability & special populations. Springer, 495–505.

[21]

Seungwon Chung and Jung-Woo Son. 2020. Visual perception in autism spectrum disorder: A review of neuroimaging studies. Journal of the Korean Academy of Child and Adolescent Psychiatry 31, 3, 105.

[22]

David F Cihak, Eric J Moore, Rachel E Wright, Don D McMahon, Melinda M Gibbons, and Cate Smith. 2016. Evaluating augmented reality to complete a chain task for elementary students with autism. Journal of Special Education Technology 31, 2, 99–108.

[23]

Blythe A Corbett and Maryam Abdullah. 2005. Video modeling: Why does it work for children with autism?Journal of Early and Intensive Behavior Intervention 2, 1, 2.

[24]

Camilla Almeida da Silva, António Ramires Fernandes, and Ana Paula Grohmann. 2014. STAR: speech therapy with augmented reality for children with autism spectrum disorders. In International Conference on Enterprise Information Systems. Springer, 379–396.

[25]

Sarah Dettmer, Richard L Simpson, Brenda Smith Myles, and Jennifer B Ganz. 2000. The use of visual supports to facilitate transitions of students with autism. Focus on autism and other developmental disabilities 15, 3, 163–169.

[26]

Mihaela Dragomir, Andrew Manches, Sue Fletcher-Watson, and Helen Pain. 2018. Facilitating pretend play in autistic children: results from an augmented reality app evaluation. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility. 407–409.

Digital Library

[27]

Yao Du, LouAnne Boyd, and Seray Ibrahim. 2018. From Behavioral and Communication Intervention to Interaction Design: User Perspectives from Clinicians. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility. 198–202.

Digital Library

[28]

Carl J Dunst, Carol M Trivette, and Tracy Masiello. 2011. Exploratory investigation of the effects of interest-based learning on the development of young children with autism. Autism 15, 3, 295–305.

[29]

Lizbeth Escobedo, David H Nguyen, LouAnne Boyd, Sen Hirano, Alejandro Rangel, Daniel Garcia-Rosas, Monica Tentori, and Gillian Hayes. 2012. MOSOCO: a mobile assistive tool to support children with autism practicing social skills in real-life situations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2589–2598.

Digital Library

[30]

Lizbeth Escobedo, Monica Tentori, Eduardo Quintana, Jesus Favela, and Daniel Garcia-Rosas. 2014. Using augmented reality to help children with autism stay focused. IEEE Pervasive Computing 13, 1, 38–46.

Digital Library

[31]

Lori Frost and Andy Bondy. 2002. The picture exchange communication system training manual. Pyramid Educational Products.

[32]

Jennifer B Ganz, Theresa L Earles-Vollrath, Amy K Heath, Richard I Parker, Mandy J Rispoli, and Jaime B Duran. 2012. A meta-analysis of single case research studies on aided augmentative and alternative communication systems with individuals with autism spectrum disorders. Journal of autism and developmental disorders 42, 1, 60–74.

[33]

Barbara C Gartin and Nikki L Murdick. 2005. Idea 2004: The IEP. Remedial and Special Education 26, 6, 327–331.

[34]

Martin Guha. 2014. Diagnostic and statistical manual of mental disorders: DSM-5. Reference Reviews.

[35]

Individual Education Plan IEP. 1978. Special educational needs.

[36]

Apple Inc.2022. ARKit - augmented reality. https://developer.apple.com/augmented-reality/arkit

[37]

Chloe Jennifer Jordan and Catherine L Caldwell-Harris. 2012. Understanding differences in neurotypical and autism spectrum special interests through internet forums. Intellectual and developmental disabilities 50, 5, 391–402.

[38]

Kamran Khowaja, Dena Al-Thani, Bilikis Banire, Siti Salwah Salim, and Asadullah Shah. 2019. Use of augmented reality for social communication skills in children and adolescents with autism spectrum disorder (ASD): A systematic review. In 2019 IEEE 6th International Conference on Engineering Technologies and Applied Sciences (ICETAS). IEEE, 1–7.

[39]

Kamran Khowaja, Bilikis Banire, Dena Al-Thani, Mohammed Tahri Sqalli, Aboubakr Aqle, Asadullah Shah, and Siti Salwah Salim. 2020. Augmented reality for learning of children and adolescents with autism spectrum disorder (ASD): A systematic review. IEEE Access 8, 78779–78807.

[40]

I Kurniawan 2018. The improvement of autism spectrum disorders on children communication ability with PECS method Multimedia Augmented Reality-Based. In Journal of Physics: Conference Series, Vol. 947. IOP Publishing, 012009.

[41]

Aaron Lanou, Lauren Hough, and Elizabeth Powell. 2012. Case studies on using strengths and interests to address the needs of students with autism spectrum disorders. Intervention in School and Clinic 47, 3, 175–182.

[42]

Gun A Lee, Ungyeon Yang, Yongwan Kim, Dongsik Jo, Ki-Hong Kim, Jae Ha Kim, and Jin Sung Choi. 2009. Freeze-Set-Go interaction method for handheld mobile augmented reality environments. In Proceedings of the 16th ACM Symposium on Virtual Reality Software and Technology. 143–146.

Digital Library

[43]

I-Jui Lee, Chien-Hsu Chen, Chuan-Po Wang, and Chi-Hsuan Chung. 2018. Augmented reality plus concept map technique to teach children with ASD to use social cues when meeting and greeting. The Asia-Pacific Education Researcher 27, 3, 227–243.

[44]

Roberto E Lopez-Herrejon, Oishi Poddar, Gerardo Herrera, and Javier Sevilla. 2020. Customization support in computer-based technologies for autism: A systematic mapping study. International Journal of Human–Computer Interaction 36, 13, 1273–1290.

[45]

Anabela Marto, Henrique A Almeida, and Alexandrino Gonçalves. 2019. Using augmented reality in patients with autism: A systematic review. In ECCOMAS Thematic Conference on Computational Vision and Medical Image Processing. Springer, 454–463.

[46]

MegEvans. 2022. Identity-first language. https://autisticadvocacy.org/about-asan/identity-first-language/

[47]

Esteban Menéndez and MD Lopez De Luise. 2018. Augmented reality as visual communication for people with ASD. In Proc. 9th Int. Conf. Soc. Inf. Technol.(ICSIT). 28–32.

[48]

Abhishek Mishra. 2019. Machine learning in the AWS cloud: Add intelligence to applications with Amazon Sagemaker and Amazon Rekognition. https://aws.amazon.com/rekognition/

[49]

Debora Nunes and Mary Frances Hanline. 2007. Enhancing the alternative and augmentative communication use of a child with autism through a parent-implemented naturalistic intervention. International Journal of Disability, Development and Education 54, 2, 177–197.

[50]

Shaila M Rao and Brenda Gagie. 2006. Learning through seeing and doing: Visual supports for children with autism. Teaching exceptional children 38, 6, 26–33.

[51]

Christopher Stephen Rayner. 2010. Video-modelling to improve task completion in a child with autism. Developmental Neurorehabilitation 13, 3, 225–230.

[52]

Mikle South, Sally Ozonoff, and William M McMahon. 2005. Repetitive behavior profiles in Asperger syndrome and high-functioning autism. Journal of autism and developmental disorders 35, 2, 145–158.

[53]

Elizabeth A Steed and Nancy Leech. 2021. Shifting to Remote Learning During COVID-19: Differences for Early Childhood and Early Childhood Special Education Teachers. Early childhood education journal, 1–10.

[54]

Issey Takahashi, Mika Oki, Baptiste Bourreau, Itaru Kitahara, and Kenji Suzuki. 2018. FUTUREGYM: A gymnasium with interactive floor projection for children with special needs. International Journal of Child-Computer Interaction 15, 37–47.

[55]

[55] Unity Technologies.2022. https://unity.com/

[56]

Catherine Tissot and Roy Evans. 2003. Visual teaching strategies for children with autism. Early Child Development and Care 173, 4, 425–433.

[57]

Laurie A Vismara and Gregory L Lyons. 2007. Using perseverative interests to elicit joint attention behaviors in young children with autism: Theoretical and clinical implications for understanding motivation. Journal of Positive Behavior Interventions 9, 4, 214–228.

[58]

Cheng Zheng, Caowei Zhang, Xuan Li, Xin Liu, Chuqi Tang, Guanyun Wang, Cheng Yao, Fan Zhang, Wenjie Xu, and Fangtian Ying. 2017. Toon-chat: a cartoon-masked chat system for children with autism. In ACM SIGGRAPH 2017 Posters. 1–2.

Digital Library

[59]

Cheng Zheng, Caowei Zhang, Xuan Li, Fan Zhang, Bing Li, Chuqi Tang, Cheng Yao, Ting Zhang, and Fangtian Ying. 2017. KinToon: a kinect facial projector for communication enhancement for ASD children. In Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology. 201–203.

Digital Library

Cited By

Liu JHuang DSong YLuo SHu B(2024)Foldiverse: Augmenting Paper Folding Physiotherapy for Children with Autism via Family-Centered Mixed Reality DesignCompanion Proceedings of the 2024 Annual Symposium on Computer-Human Interaction in Play10.1145/3665463.3678823(378-383)Online publication date: 14-Oct-2024
https://dl.acm.org/doi/10.1145/3665463.3678823
Haroon RDogar F(2024)TwIPS: A Large Language Model Powered Texting Application to Simplify Conversational Nuances for Autistic UsersProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675633(1-18)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3675633
South LYildirim CPavel ABorkin M(2024)Barriers to Photosensitive Accessibility in Virtual RealityProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642635(1-13)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642635

Index Terms

Designing a Customizable Picture-Based Augmented Reality Application For Therapists and Educational Professionals Working in Autistic Contexts
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. User studies
    2. Interaction paradigms
      1. Mixed / augmented reality
2. Social and professional topics
  1. User characteristics
    1. People with disabilities

Recommendations

Effectiveness of a Mixed Reality system in terms of social interaction behaviors in children with and without Autism Spectrum Condition
Interacción '21: Proceedings of the XXI International Conference on Human Computer Interaction

The present study analyses the effectiveness of a mixed reality (MR) face-to-face full-body system to foster social interaction behaviors (SIBs) in a child with ASC and a play partner, a child without ASC (non-ASC). An MR system allows the development ...
Virtual reality and augmented reality for education: panel
SA '16: SIGGRAPH ASIA 2016 Symposium on Education: Talks

Since the last few decades, virtual reality (VR) and augmented-reality (AR) interfaces have shown the potential to enhance teaching and learning, by combining physical and virtual worlds and leveraging the advantages of both [Yannier et al. 2015]. ...
Using Augmented Reality to Treat Phobias

Exposure in vivo and VR therapies have proven to be effective in the treatment of different psychological disorders such as acrophobia, claustrophobia, or phobia to small animals; however until now augmented reality (AR) has not been used in these ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ASSETS '22: Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility

October 2022

902 pages

ISBN:9781450392587

DOI:10.1145/3517428

Editors:
Jon Froehlich
University of Washington, USA
,
Kristen Shinohara
Rochester Institute of Technology, USA
,
Stephanie Ludi
University of North Texas, USA

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGACCESS: ACM Special Interest Group on Accessible Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 October 2022

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ASSETS '22

Sponsor:

SIGACCESS

ASSETS '22: The 24th International ACM SIGACCESS Conference on Computers and Accessibility

October 23 - 26, 2022

Athens, Greece

Acceptance Rates

ASSETS '22 Paper Acceptance Rate 35 of 132 submissions, 27%;

Overall Acceptance Rate 436 of 1,556 submissions, 28%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
803
Total Downloads

Downloads (Last 12 months)399
Downloads (Last 6 weeks)49

Reflects downloads up to 03 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Liu JHuang DSong YLuo SHu B(2024)Foldiverse: Augmenting Paper Folding Physiotherapy for Children with Autism via Family-Centered Mixed Reality DesignCompanion Proceedings of the 2024 Annual Symposium on Computer-Human Interaction in Play10.1145/3665463.3678823(378-383)Online publication date: 14-Oct-2024
https://dl.acm.org/doi/10.1145/3665463.3678823
Haroon RDogar F(2024)TwIPS: A Large Language Model Powered Texting Application to Simplify Conversational Nuances for Autistic UsersProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675633(1-18)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3675633
South LYildirim CPavel ABorkin M(2024)Barriers to Photosensitive Accessibility in Virtual RealityProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642635(1-13)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642635

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten