Softly Non-Spoken: Exploring the Potential of Soft Robotics to Support Non-Verbal Communication
Authors: 
Alison Oldfield, Katherine Broomfield, Hemma Philamore, Richard Sewell, and Emma PowellAuthors Info & Claims
Alison Oldfield, Katherine Broomfield, Hemma Philamore, Richard Sewell, and Emma PowellAuthors Info & ClaimsMay 2024
Article No.: 296, Pages 1 - 7
Abstract
Existing Augmentative and Alternative Communication (AAC) technologies aim to support people with communication disability participate more fully in life and society, but these often prioritize linguistic and spoken modes of expression. Embracing a broader view of communication that includes non-verbal and embodied expressions, this interdisciplinary project explored the potential of soft robotics for developing AAC technologies that create opportunities for more nuanced forms of communication. To do this, the Softly Non-Spoken project facilitated interdisciplinary workshops and developed soft robotic objects which were then used as probes to explore how such technologies might support expressive, affective non-verbal communication. Results suggest that soft robotics materials and functions offer potential in this design space, including compelling possibilities for object-enabled human interaction that could offer more individual control around communication. The findings also propose a conceptualization of communication that encompasses embodied expression and could cultivate ambiguity and intrigue in future AAC design.
Supplemental Material
MP4 File
Talk Video
- Download
- 52.25 MB
- Transcript
References
[1]
James Law, Anna van der Gaag, Bill Hardcastle, Janet Beck, Andrew MacGregor, Charlene Plunkett. 2007. Communication Support Needs: A review of the literature. Scottish Executive Social Research, Edinburgh.
[2]
World Health Organization (WHO). 2007. WHO International classification of functioning, disability and health— Version for children and youth: ICF-CY. WHO, Geneva.
[3]
James Law, Robert Rush, Ingrid Schoon, Samantha Parsons. 2009. Modeling developmental language difficulties from school entry into adulthood: literacy, mental health, and employment outcomes. J Speech Lang Hear Res. 52, 6 (December 2009), 1401-16. https://doi.org/10.1044/1092- 4388(2009/09-0142)
[4]
Susie Parr. 2007. Living with severe aphasia: Tracking social exclusion. Aphasiology. 21, 1, 98–123. https://doi.org/10.1080/02687030600798337
[5]
David R. Beukelman and Janice Catherine Light. 2020. Augmentative & alternative communication: Supporting children and adults with complex communication needs. Brookes Publishing Co. Baltimore, USA.
[6]
Nerissa Hall, Jenifer Juengling-Sudkamp, Michelle L. Gutmann, and Ellen R. Cohn. 2023. Fundamentals of AAC: a case-based approach to enhancing communication. Plural Publishing, Inc. San Diego, USA.
[7]
Simon Judge and Gillian Townend. 2013. Perceptions of the design of voice output communication aids: Perceptions of the design of voice output communication aids. Int J of Lang & Comm Dis. 48, 4 (July-August 2013), 366–381. https://doi.org/10.1111/1460-6984.12012
[8]
Susan Baxter, Pam Enderby, Phillipa Evans and Simon Judge. 2012. Barriers and facilitators to the use of high-technology augmentative and alternative communication devices: a systematic review and qualitative synthesis: AAC barriers and facilitators review. Int J of Lang & Comm Dis. 47, 2 (March-April 2012), 115–129. https://doi.org/10.1111/j.1460-6984.2011.00090.x
[9]
D Jeffrey Higginbotham, Howard Shane, Susanne Russell and Kevin Caves. 2007. Access to AAC: Present, past, and future, Augmentative and Alternative Communication, 23, 3 (September 2007), 243–257. https://doi.org/10.1080/07434610701571058
[10]
Humphrey Curtis, Timothy Neate, Carlota Vazquez Gonzalez. 2022. State of the Art in AAC: A Systematic Review and Taxonomy. In: 24th International Proceedings of the ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '22), Association for Computing Machinery, New York, 1-22.
[11]
Katherine Broomfield, Karen Sage, Georgina Jones, Deborah Harrop, and Simon Judge. 2022. A qualitative evidence synthesis of the experiences and perspectives of communicating using augmentative and alternative communication (AAC)’, Dis and Rehab: AssistTech, 1–15. https://doi.org/10.1080/17483107.2022.2105961
[12]
Katherine Broomfield, Karen Sage, Georgina L Jones, Simon Judge, Deborah James. 2022. The Unspoken Voice: Applying John Shotter's Dialogic Lens to Qualitative Data from People Who have Communication Difficulties. Qualitative Health Research 33, 1-2 (January 2023), 1-3. https://doi.org/10.1177/10497323221139803
[13]
Meryl Alper. 2017. Giving Voice: Mobile Communication, Disability and Inequality. MIT Press, Cambridge, USA.
[14]
Graham Pullin, Jutta Treviranus, Rupal Patel, and Jeff Higginbotham. 2017. Designing interaction, voice, and inclusion in AAC research. Augmentative and Alternative Communication. 33, 3 (September 2017), 139-148. https://doi.org/10.1080/07434618.2017.1342690
[15]
John Shotter. 2011. Getting it: withness-thinking and the dialogical– in practice. Hampton. New York.
[16]
Sophya Yumakulov, Dean Yergens and Gregor Wolbring. 2012. Imagery of Disabled People within Social Robotics Research. In Social Robotics. ICSR Lecture Notes in Computer Science, vol. 7621. Springer, Berlin, Heidelberg.
[17]
Madeline Cruice. 2008. The contribution and impact of the International Classification of Functioning, Disability and Health on quality of life in communication disorders. Int J Speech-Lang Path. 10, 1–2, 38–49. https://doi.org/10.1080/17549500701790520
[18]
Stephanie Valencia, Mark Steidl, Michael L. Rivera, Cynthia L. Bennett, Jeffrey P. Bigham, and Henny Admoni. 2021. Aided Nonverbal Communication through Physical Expressive Objects. In The 23rd International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’21), October 18–22, 2021, Virtual Event, USA. ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/3441852.3471228
[19]
Rúbia E. O. Schultz Ascari, Roberto Pereira, and Luciano Silva. 2018. Towards a Methodology to Support Augmentative and Alternative Communication by means of Personalized Gestural Interaction. In Proceedings of the 17th Brazilian Symposium on Human Factors in Computing Systems (IHC 2018). Association for Computing Machinery, New York, NY, USA, Article 38, 1–12. https://doi.org/10.1145/3274192.3274230
[20]
Timothy Neate, Abi Roper, Stephanie Wilson, and Jane Marshall. 2019. Empowering Expression for Users with Aphasia through Constrained Creativity. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). Association for Computing Machinery, New York, NY, USA, Paper 385, 1-12. https://doi.org/10.1145/3290605.3300615
[21]
Panagiotis Polygerinos, Nikolaus Correll, Stephan A. Morin, Bobak Mosadegh, Cagdas D. Onal, Kirstin Petersen, Matteo Cianchetti, Michael T. Tolley and Robert F. Shepherd. 2017. Soft robotics: Review of fluid‐driven intrinsically soft devices; manufacturing, sensing, control, and applications in human‐robot interaction. Advanced Engineering Materials, 19, 12, https://doi.org/10.1002/adem.201700016
[22]
Takashi Minato, Yuichiro Yoshikawa, Tomoyuki Noda, Shuhei Ikemoto, Hiroshi Ishiguro, and Minoru Asada. 2007. CB2: A child robot with biomimetic body for cognitive developmental robotics. In 7th IEEE-RAS International Conference on Humanoid Robots, Pittsburgh, PA, USA, 557-562. https://doi.org/10.1109/ICHR.2007.4813926
[23]
Ipke Wachsmuth, Manuela Lenzen and Gunther Knoblich. 2008. Embodied communication in humans and machines. Oxford University Press, UK.
[24]
Jonas Jørgensen, Kirsten Borup Bojesen and Elizabeth Jochum. 2022. Is a soft robot more “natural”? Exploring the perception of soft robotics in human-robot interaction. International Journal of Social Robotics. 14, 95-113. https://doi.org/10.1007/s12369-021-00761-1
[25]
Anna Dagmar Bille Milthers, Anne Bjerre Hammer, Jonathan Jung Johansen, Lasse Goul Jensen, Elizabeth Ann Jochum, and Markus Löchtefeld. 2019. The Helpless Soft Robot - Stimulating Human Collaboration through Robotic Movement. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA '19). Association for Computing Machinery, New York, NY, USA, Paper LBW2421, 1–6. https://doi.org/10.1145/3290607.3312807
[26]
Thomas Arnold and Matthias Scheutz. 2017. The tactile ethics of soft robotics: Designing wisely for human–robot interaction. Soft robotics. 4, 2 (June 2017), 81-87. https://doi.org/10.1089/soro.2017.0032
[27]
Ulrika M. Ferm, Britt K. Claesson, Cajsa Ottesjo and Stina Ericsson. 2015. Participation and Enjoyment in Play with a Robot between Children with Cerebral Palsy who use AAC and their Peers. Augmentative and Alternative Communication. 31, 2 (June 2015), 108-123. https://doi.org/ 10.3109/07434618.2015.1029141
[28]
Kyung Hea Jeon, Seok Jeong Yeon, Young Tae Kim, Seokwoo Song, and John Kim. 2014. Robot-based augmentative and alternative communication for nonverbal children with communication disorders. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '14). Association for Computing Machinery, New York, NY, USA, 853–859. https://doi.org/10.1145/2632048.2636078
[29]
Andrea Bonarini, Francesco Clasadonte, Franca Garzotto, Mirko Gelsomini, and Maximiliano Romero. 2016. Playful interaction with Teo, a Mobile Robot for Children with Neurodevelopmental Disorders. In Proceedings of the 7th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion (DSAI '16). Association for Computing Machinery, New York, NY, USA, 223–231. https://doi.org/10.1145.3019943.3019976
[30]
Walter Dan Stiehl, Jeff Liberman, Cynthia Breazeal, Louis Basel, Roshni Cooper, Heather Knight, Levi Lalla, Allan Maymin, and Scott Purchase. 2006. The huggable: a therapeutic robotic companion for relational, affective touch. In 3rd IEEE Consumer Communications and Networking Conference, (IEEE CCNC 2006), Las Vegas, NV, USA, 1290-1291. https://doi.org/ 10.1109/CCNC.2006.1593253
[31]
Jayne Wallace, John McCarthy, Peter C. Wright, and Patrick Olivier. 2013. Making design probes work. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). Association for Computing Machinery, New York, NY, USA, 3441–3450. https://doi.org/10.1145/2470654.2466473
[32]
Bill Gaver, Tony Dunne, and Elena Pacenti. 1999. Design: Cultural probes. interactions 6, 1 (Jan./Feb. 1999), 21–29. https://doi.org/10.1145/291224.291235
[33]
Kirsten Boehner, Janet Vertesi, Phoebe Sengers, and Paul Dourish. 2007. How HCI interprets the probes. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '07). Association for Computing Machinery, New York, NY, USA, 1077–1086. https://doi.org/10.1145/1240624.1240789
[34]
Vetere, F., Gibbs, M. R., Kjeldskov, J., Howard, S., Mueller, F., Pedell, S., Mecoles, K., and Bunyan, M. 2005. Mediating intimacy. In Proc CHI '05. NY: ACM Press, 471-480.
[35]
Dickerson, S.S., Stone, V., Panchura, C., Usiak, D.J.: The meaning of communication: Experiences with augmentative communication devices, Rehab Nursing; Chicago, 27(6), pp. 215–20.(2002)
[36]
Wickenden, M. 2011. ‘Whose Voice is That?: Issues of Identity, Voice and Representation Arising in an Ethnographic Study of the Lives of Disabled Teenagers who use Augmentative and Alternative Communication (AAC)’, Dis Studies Quarterly, 31(4). Available at: https://doi.org/10.18061/dsq.v31i4.1724
[37]
Teachman, G., McDonough, Pl. Macarthur, C. and Gibson, B.E. 2018. A Critical Dialogical Methodology for Conducting Research With Disabled Youth Who Use Augmentative and Alternative Communication. Qualitative Inquiry, 24, 1, 35– 44. https://doi.org/10.1177/1077800417727763
Recommendations
Salience modeling based on non-verbal modalities for spoken language understanding
ICMI '06: Proceedings of the 8th international conference on Multimodal interfacesPrevious studies have shown that, in multimodal conversational systems, fusing information from multiple modalities together can improve the overall input interpretation through mutual disambiguation. Inspired by these findings, this paper investigates ...
Comments
Information & Contributors
Information
Published In
May 2024
4761 pages
ISBN:9798400703317
DOI:10.1145/3613905
Copyright © 2024 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 May 2024
Check for updates
Qualifiers
- Work in progress
- Research
- Refereed limited
Data Availability
Conference
Acceptance Rates
Overall Acceptance Rate 6,164 of 23,696 submissions, 26%
Upcoming Conference
CHI PLAY '24
- Sponsor:
- sigchi
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 127Total Downloads
- Downloads (Last 12 months)127
- Downloads (Last 6 weeks)42
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format