EmoSparkle: Tangible Prototype to Convey Visual Expressions for Visually Impaired Individuals in Real-time Conversations
Authors: 
Zhe Sun, Robin Ananda, Xinyi FuAuthors Info & Claims
Zhe Sun, Robin Ananda, Xinyi FuAuthors Info & ClaimsPages 38 - 49
Abstract
Non-verbal expression plays an important role in social communication. Visually impaired people may feel embarrassed because they have limited access to such information. In this article, we proposed EmoSparkle, a prototype system that converts facial expressions into tangible feedback in real time. Our system included a real-time facial expression recognition algorithm as well as a haptic user interface that conveyed recognized expressions via tactile feedback. We proposed a strategy for mapping expressions to tactile feedback. A pilot study was conducted with 6 visually impaired participants to derive our design requirements before we designed EmoSparkle. To evaluate the prototype system, a user study with 16 visually impaired participants was conducted, which included a pre-study interview, a usability test, and a semi-structured interview. Our system performed well in terms of learnability and usability according to the results. Results also revealed the potential of our system to elicit emotional resonance as well as the possible design space of emotional auxiliary devices.
References
[1]
Asad Abbas and Stephan K Chalup. 2017. Group emotion recognition in the wild by combining deep neural networks for facial expression classification and scene-context analysis. In Proceedings of the 19th ACM International Conference on Multimodal Interaction. ACM, 561–568.
[2]
Moses Akazue, Martin Halvey, and Lynne Baillie. 2017. Using thermal stimuli to enhance photo-sharing in social media. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 2 (2017), 4.
[3]
Octavio Arriaga, Matias Valdenegro-Toro, and Paul Plöger. 2017. Real-time convolutional neural networks for emotion and gender classification. arXiv preprint arXiv:1710.07557(2017).
[4]
Aaron Bangor, Philip Kortum, and James Miller. 2009. Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of usability studies 4, 3 (2009), 114–123.
[5]
Emad Barsoum, Cha Zhang, Cristian Canton Ferrer, and Zhengyou Zhang. 2016. Training deep networks for facial expression recognition with crowd-sourced label distribution. In Proceedings of the 18th ACM International Conference on Multimodal Interaction. 279–283.
[6]
Hendrik P Buimer, Marian Bittner, Tjerk Kostelijk, Thea M van der Geest, Abdellatif Nemri, Richard JA van Wezel, and Yan Zhao. 2018. Conveying facial expressions to blind and visually impaired persons through a wearable vibrotactile device. PloS one 13, 3 (2018), e0194737.
[7]
Young-Hyen Byeon and Keun-Chang Kwak. 2014. Facial expression recognition using 3d convolutional neural network. International journal of advanced computer science and applications 5, 12(2014).
[8]
François Chollet. 2016. Xception: Deep Learning with Depthwise Separable Convolutions. arxiv:1610.02357 [cs.CV]
[9]
Sidney K D’mello and Jacqueline Kory. 2015. A review and meta-analysis of multimodal affect detection systems. ACM computing surveys (CSUR) 47, 3 (2015), 1–36.
[10]
Paul Ekman and Wallace V. Friesen. 1971. Constants across cultures in the face and emotion.Journal of Personality & Social Psychology 17, 2 (1971), 124–129.
[11]
Shuang He, Yanhong Jia, Zhe Sun, Chenxin Yu, Xin Yi, Yuanchun Shi, and Yingqing Xu. 2019. AR assistive system in domestic environment using HMDs: Comparing visual and aural instructions. In International Conference on Human-Computer Interaction. Springer, 71–83.
[12]
Ómar I Jóhannesson, Rebekka Hoffmann, Vigdís Vala Valgeirsdóttir, Rúnar Unnþórsson, Alin Moldoveanu, and Árni Kristjánsson. 2017. Relative vibrotactile spatial acuity of the torso. Experimental brain research 235, 11 (2017), 3505–3515.
[13]
Mark L Knapp, Judith A Hall, and Terrence G Horgan. 2013. Nonverbal communication in human interaction. Cengage Learning.
[14]
Sreekar Krishna, Shantanu Bala, Troy McDaniel, Stephen McGuire, and Sethuraman Panchanathan. 2010. VibroGlove: an assistive technology aid for conveying facial expressions. In CHI’10 Extended Abstracts on Human Factors in Computing Systems. ACM, 3637–3642.
[15]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097–1105.
[16]
Miriam Madsen, Rana El Kaliouby, Matthew Goodwin, and Rosalind Picard. 2008. Technology for just-in-time in-situ learning of facial affect for persons diagnosed with an autism spectrum disorder. In Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility. 19–26.
[17]
Troy McDaniel, Diep Tran, Samjhana Devkota, Kaitlyn DiLorenzo, Bijan Fakhri, and Sethuraman Panchanathan. 2018. Tactile Facial Expressions and Associated Emotions toward Accessible Social Interactions for Individuals Who Are Blind. In Proceedings of the 2018 Workshop on Multimedia for Accessible Human Computer Interface. ACM, 25–32.
[18]
Daniel McDuff, Kael Rowan, Piali Choudhury, Jessica Wolk, ThuVan Pham, and Mary Czerwinski. 2019. A multimodal emotion sensing platform for building emotion-aware applications. arXiv preprint arXiv:1903.12133(2019).
[19]
AKMMahbubur Rahman, ASM Iftekhar Anam, and Mohammed Yeasin. 2017. EmoAssist: emotion enabled assistive tool to enhance dyadic conversation for the blind. Multimedia Tools and Applications 76, 6 (2017), 7699–7730.
[20]
James Russell. 1980. A Circumplex Model of Affect. Journal of Personality and Social Psychology 39 (12 1980), 1161–1178. https://doi.org/10.1037/h0077714
[21]
Andreas Sackl, Raimund Schatz, Bruno Gardlo, and Manfred Tscheligi. 2021. Enhancing video communication experience for low vision users. In Proceedings of the 18th International Web for All Conference. 1–5.
[22]
Katri Salminen, Veikko Surakka, Jukka Raisamo, Jani Lylykangas, Roope Raisamo, Kalle Mäkelä, and Teemu Ahmaniemi. 2013. Cold or hot? how thermal stimuli are related to human emotional system?. In International workshop on haptic and audio interaction design. Springer, 20–29.
[23]
Lei Shi, Brianna J Tomlinson, John Tang, Edward Cutrell, Daniel McDuff, Gina Venolia, Paul Johns, and Kael Rowan. 2019. Accessible video calling: Enabling nonvisual perception of visual conversation cues. Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019), 1–22.
[24]
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556(2014).
[25]
Mohammad Soleymani, David Garcia, Brendan Jou, Björn Schuller, Shih-Fu Chang, and Maja Pantic. 2017. A survey of multimodal sentiment analysis. Image and Vision Computing 65 (2017), 3–14.
[26]
Kenji Suzuki. 2015. Social imaging technology to identify and represent social behaviors. In Adjunct Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers. 907–908.
[27]
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1–9.
[28]
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2818–2826.
[29]
M Iftekhar Tanveer, ASM Anam, Mohammed Yeasin, and Majid Khan. 2013. Do you see what I see?: designing a sensory substitution device to access non-verbal modes of communication. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, 10.
[30]
Jordan Tewell, Jon Bird, and George R Buchanan. 2017. The heat is on: a temperature display for conveying affective feedback. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 1756–1767.
[31]
Shangfei Wang, Zhilei Liu, Zhaoyu Wang, Guobing Wu, Peijia Shen, Shan He, and Xufa Wang. 2012. Analyses of a multimodal spontaneous facial expression database. IEEE Transactions on Affective Computing 4, 1 (2012), 34–46.
[32]
Graham Wilson and Stephen A Brewster. 2017. Multi-moji: Combining thermal, vibrotactile & visual stimuli to expand the affective range of feedback. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 1743–1755.
[33]
Graham Wilson, Gavin Davidson, and Stephen A Brewster. 2015. In the heat of the moment: subjective interpretations of thermal feedback during interaction. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 2063–2072.
[34]
Graham Wilson, Dobromir Dobrev, and Stephen A Brewster. 2016. Hot under the collar: mapping thermal feedback to dimensional models of emotion. In proceedings of the 2016 CHI conference on human factors in computing systems. ACM, 4838–4849.
[35]
Graham Wilson, Martin Halvey, Stephen A Brewster, and Stephen A Hughes. 2011. Some like it hot: thermal feedback for mobile devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2555–2564.
[36]
Shamima Yasmin and Sethuraman Panchanathan. 2016. Haptic mirror: a platform for active exploration of facial expressions and social interaction by individuals who are blind. In Proceedings of the 15th ACM SIGGRAPH Conference on Virtual-Reality Continuum and Its Applications in Industry-Volume 1. ACM, 319–329.
[37]
Yongjae Yoo, Taekbeom Yoo, Jihyun Kong, and Seungmoon Choi. 2015. Emotional responses of tactile icons: Effects of amplitude, frequency, duration, and envelope. In 2015 IEEE World Haptics Conference (WHC). IEEE, 235–240.
[38]
Annuska Zolyomi, Andrew Begel, Jennifer Frances Waldern, John Tang, Michael Barnett, Edward Cutrell, Daniel McDuff, Sean Andrist, and Meredith Ringel Morris. 2019. Managing stress: The needs of autistic adults in video calling. Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019), 1–29.
Index Terms
- EmoSparkle: Tangible Prototype to Convey Visual Expressions for Visually Impaired Individuals in Real-time Conversations
Recommendations
Making Facial Expressions of Emotions Accessible for Visually Impaired Persons
ASSETS '17: Proceedings of the 19th International ACM SIGACCESS Conference on Computers and AccessibilityOne of the big problems visually impaired persons experience in their daily lives, is the inability to see non-verbal cues of conversation partners. In this study, a wearable assistive technology is presented and evaluated which supports visually ...
Skim-reading Strategies in Sighted and Visually-Impaired Individuals: A Comparative Study
PETRA '18: Proceedings of the 11th PErvasive Technologies Related to Assistive Environments ConferenceAs the amount of readily available reading material increases, the ability to read selectively, or to skim-read, is becoming an important skill. In this paper, we report a comparative study of existing skim-reading strategies in sighted and visually ...
Emoji Accessibility for Visually Impaired People
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing SystemsEmoji are graphical symbols that appear in many aspects of our lives. Worldwide, around 36 million people are blind and 217 million have a moderate to severe visual impairment. This portion of the population may use and encounter emoji, yet it is unclear ...
Comments
Information & Contributors
Information
Published In
October 2022
342 pages
ISBN:9781450398695
DOI:10.1145/3565698
Copyright © 2022 Owner/Author.
This work is licensed under a Creative Commons Attribution-NonCommercial International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 12 February 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
Conference
Chinese CHI 2022
Chinese CHI 2022: The Tenth International Symposium of Chinese CHI
October 22 - 23, 2022
Guangzhou, China and Online, China
Acceptance Rates
Overall Acceptance Rate 17 of 40 submissions, 43%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 242Total Downloads
- Downloads (Last 12 months)242
- Downloads (Last 6 weeks)27
Reflects downloads up to 02 Feb 2025
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in