Party Face Congratulations! Exploring Design Ideas to Help Sighted Users with Emoji Accessibility when Messaging with Screen Reader Users
Article No.: 175, Pages 1 - 31
Abstract
Emoji are a popular, expressive form of non-verbal communication. However, people often use emoji in ways that result in confusing or cumbersome screen reader output. We created two accessibility support designs: (1) Preview, which displays a basic text transcript of a message with emoji that a screen reader would narrate, and (2) Alert, which summarises potential accessibility issues caused by emoji within a message. We explored our designs using an online survey and provided participants with the option to edit messages that contained emoji, should they choose to do so. We collected 1508 modified messages from 116 sighted participants and conducted a qualitative analysis of the data to identify the strategies participants used when asked to edit a message for accessibility issues and their appreciation of each design. We found that participants preferred the Preview design over Alert since it allows for subjective interpretations of what constitutes an accessible message. We report sighted users' rewriting strategies (e.g., editing the message to move the emoji to the end) and incorrect assumptions about screen readers that would lead to using textual markers that are incompatible with screen readers. We discuss the design implications for future systems for accessible messaging.
References
[1]
Wei Ai, Xuan Lu, Xuanzhe Liu, Ning Wang, Gang Huang, and Qiaozhu Mei. 2017. Untangling emoji popularity through semantic embeddings. In Proceedings of the international AAAI conference on web and social media, Vol. 11. 2--11. https://doi.org/10.1609/icwsm.v11i1.14903
[2]
Cynthia L. Bennett, Cole Gleason, Morgan Klaus Scheuerman, Jeffrey P. Bigham, Anhong Guo, and Alexandra To. 2021. ?It's Complicated": Negotiating Accessibility and (Mis)Representation in Image Descriptions of Race, Gender, and Disability. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 375, bibinfonumpages19 pages. https://doi.org/10.1145/3411764.3445498
[3]
Keith Broni. 2021. Top Emoji Trends of 2021. https://blog.emojipedia.org/top-emoji-trends-of-2021/ Accessed: 2023-07--11.
[4]
Keith Broni. 2022. New Emojis In 2022--2023. https://blog.emojipedia.org/new-emojis-in-2022--2023/ Accessed: 2023-07--11.
[5]
Stanley J. Cantrell, R. Michael Winters, Prakriti Kaini, and Bruce N. Walker. 2022. Sonification of Emotion in Social Media: Affect and Accessibility in Facebook Reactions. Proc. ACM Hum.-Comput. Interact., Vol. 6, CSCW1, Article 119 (apr 2022), bibinfonumpages26 pages. https://doi.org/10.1145/3512966
[6]
Victoria Clarke and Virginia Braun. 2013. Successful qualitative research: A practical guide for beginners. Successful qualitative research (2013), 1--400.
[7]
Henriette Cramer, Paloma de Juan, and Joel Tetreault. 2016. Sender-Intended Functions of Emojis in US Messaging. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services (Florence, Italy) (MobileHCI '16). Association for Computing Machinery, New York, NY, USA, 504--509. https://doi.org/10.1145/2935334.2935370
[8]
Maitraye Das, Darren Gergle, and Anne Marie Piper. 2019. "It Doesn't Win You Friends": Understanding Accessibility in Collaborative Writing for People with Vision Impairments. Proc. ACM Hum.-Comput. Interact., Vol. 3, CSCW, Article 191 (nov 2019), bibinfonumpages26 pages. https://doi.org/10.1145/3359293
[9]
Maitraye Das, Thomas Barlow McHugh, Anne Marie Piper, and Darren Gergle. 2022a. Co11ab: Augmenting Accessibility in Synchronous Collaborative Writing for People with Vision Impairments. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 196, bibinfonumpages18 pages. https://doi.org/10.1145/3491102.3501918
[10]
Maitraye Das, Anne Marie Piper, and Darren Gergle. 2022b. Design and Evaluation of Accessible Collaborative Writing Techniques for People with Vision Impairments. ACM Trans. Comput.-Hum. Interact., Vol. 29, 2, Article 9 (jan 2022), bibinfonumpages42 pages. https://doi.org/10.1145/3480169
[11]
Emojipedia. [n.,d.]. Emoji Shortcodes. https://emojipedia.org/shortcodes/ Accessed: 2023-07--18.
[12]
Richard Fox. 2001. Constructivism examined. Oxford review of education, Vol. 27, 1 (2001), 23--35.
[13]
Cole Gleason, Patrick Carrington, Cameron Cassidy, Meredith Ringel Morris, Kris M. Kitani, and Jeffrey P. Bigham. 2019a. ?It's Almost like They're Trying to Hide It": How User-Provided Image Descriptions Have Failed to Make Twitter Accessible. In The World Wide Web Conference (San Francisco, CA, USA) (WWW '19). Association for Computing Machinery, New York, NY, USA, 549--559. https://doi.org/10.1145/3308558.3313605
[14]
Cole Gleason, Patrick Carrington, Lydia B. Chilton, Benjamin Gorman, Hernisa Kacorri, Andrés Monroy-Hernández, Meredith Ringel Morris, Garreth Tigwell, and Shaomei Wu. 2020a. Future Research Directions for Accessible Social Media. SIGACCESS Access. Comput. 127, Article 3 (jul 2020), bibinfonumpages12 pages. https://doi.org/10.1145/3412836.3412839
[15]
Cole Gleason, Patrick Carrington, Lydia B. Chilton, Benjamin M. Gorman, Hernisa Kacorri, Andrés Monroy-Hernández, Meredith Ringel Morris, Garreth W. Tigwell, and Shaomei Wu. 2019b. Addressing the Accessibility of Social Media. In Conference Companion Publication of the 2019 on Computer Supported Cooperative Work and Social Computing (Austin, TX, USA) (CSCW '19). Association for Computing Machinery, New York, NY, USA, 474--479. https://doi.org/10.1145/3311957.3359439
[16]
Cole Gleason, Amy Pavel, Himalini Gururaj, Kris Kitani, and Jeffrey Bigham. 2020b. Making GIFs Accessible. In Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility (Virtual Event, Greece) (ASSETS '20). Association for Computing Machinery, New York, NY, USA, Article 24, bibinfonumpages10 pages. https://doi.org/10.1145/3373625.3417027
[17]
Cole Gleason, Amy Pavel, Xingyu Liu, Patrick Carrington, Lydia B. Chilton, and Jeffrey P. Bigham. 2019c. Making Memes Accessible. In Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility (Pittsburgh, PA, USA) (ASSETS '19). Association for Computing Machinery, New York, NY, USA, 367--376. https://doi.org/10.1145/3308561.3353792
[18]
Carla F. Griggio, Joanna McGrenere, and Wendy E. Mackay. 2019. Customizations and Expression Breakdowns in Ecosystems of Communication Apps. Proc. ACM Hum.-Comput. Interact., Vol. 3, CSCW, Article 26 (nov 2019), bibinfonumpages26 pages. https://doi.org/10.1145/3359128
[19]
Carla F. Griggio, Arissa J. Sato, Wendy E. Mackay, and Koji Yatani. 2021. Mediating Intimacy with DearBoard: A Co-Customizable Keyboard for Everyday Messaging. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 342, bibinfonumpages16 pages. https://doi.org/10.1145/3411764.3445757
[20]
Emory James Edwards, Kyle Lewis Polster, Isabel Tuason, Emily Blank, Michael Gilbert, and Stacy Branham. 2021. "That's in the Eye of the Beholder": Layers of Interpretation in Image Descriptions for Fictional Representations of People with Disabilities. In Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility (Virtual Event, USA) (ASSETS '21). Association for Computing Machinery, New York, NY, USA, Article 19, bibinfonumpages14 pages. https://doi.org/10.1145/3441852.3471222
[21]
Ryan Kelly and Leon Watts. 2015. Characterising the inventive appropriation of emoji as relationally meaningful in mediated close personal relationships. Experiences of technology appropriation: Unanticipated users, usage, circumstances, and design, Vol. 2 (2015).
[22]
Philippe Kimura-Thollander and Neha Kumar. 2019. Examining the "Global" Language of Emojis: Designing for Cultural Representation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--14. https://doi.org/10.1145/3290605.3300725
[23]
Jonathan Lazar, Aaron Allen, Jason Kleinman, and Chris Malarkey. 2007. What frustrates screen reader users on the web: A study of 100 blind users. International Journal of human-computer interaction, Vol. 22, 3 (2007), 247--269. https://doi.org/10.1080/10447310709336964
[24]
Cheuk Yin Phipson Lee, Zhuohao Zhang, Jaylin Herskovitz, JooYoung Seo, and Anhong Guo. 2022. CollabAlly: Accessible Collaboration Awareness in Document Editing. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 596, bibinfonumpages17 pages. https://doi.org/10.1145/3491102.3517635
[25]
Hae-Na Lee and Vikas Ashok. 2022. Impact of Out-of-Vocabulary Words on the Twitter Experience of Blind Users. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 608, bibinfonumpages20 pages. https://doi.org/10.1145/3491102.3501958
[26]
Junchen Li, Garreth W. Tigwell, and Kristen Shinohara. 2021. Accessibility of High-Fidelity Prototyping Tools. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 493, bibinfonumpages17 pages. https://doi.org/10.1145/3411764.3445520
[27]
Weijian Li, Yuxiao Chen, Tianran Hu, and Jiebo Luo. 2018. Mining the relationship between emoji usage patterns and personality. In Proceedings of the international AAAI conference on web and social media, Vol. 12. https://doi.org/10.1609/icwsm.v12i1.15054
[28]
Content Design London. 2020. Emojis. https://readabilityguidelines.co.uk/images/emojis/ Accessed: 2023-07--18.
[29]
Danielle Lottridge, Chris Yoon, Darren Burton, Chester Wang, and Jofish Kaye. 2022. Ally: Understanding Text Messaging to Build a Better Onscreen Keyboard for Blind People. ACM Trans. Access. Comput., Vol. 15, 4, Article 30 (oct 2022), bibinfonumpages22 pages. https://doi.org/10.1145/3533707
[30]
Xuan Lu, Wei Ai, Xuanzhe Liu, Qian Li, Ning Wang, Gang Huang, and Qiaozhu Mei. 2016. Learning from the Ubiquitous Language: An Empirical Analysis of Emoji Usage of Smartphone Users. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (Heidelberg, Germany) (UbiComp '16). Association for Computing Machinery, New York, NY, USA, 770--780. https://doi.org/10.1145/2971648.2971724
[31]
Kelly Mack, Rai Ching Ling Hsu, Andrés Monroy-Hernández, Brian A. Smith, and Fannie Liu. 2023. Towards Inclusive Avatars: Disability Representation in Avatar Platforms. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 607, bibinfonumpages13 pages. https://doi.org/10.1145/3544548.3581481
[32]
Nora McDonald, Sarita Schoenebeck, and Andrea Forte. 2019. Reliability and Inter-Rater Reliability in Qualitative Research: Norms and Guidelines for CSCW and HCI Practice. Proc. ACM Hum.-Comput. Interact., Vol. 3, CSCW, Article 72 (Nov. 2019), bibinfonumpages23 pages. https://doi.org/10.1145/3359174
[33]
Lindsay McShane, Ethan Pancer, Maxwell Poole, and Qi Deng. 2021. Emoji, playfulness, and brand engagement on twitter. Journal of Interactive Marketing, Vol. 53, 1 (2021), 96--110. https://doi.org/10.1016/j.intmar.2020.06.002
[34]
Hannah Miller, Daniel Kluver, Jacob Thebault-Spieker, Loren Terveen, and Brent Hecht. 2017. Understanding emoji ambiguity in context: The role of text in emoji-related miscommunication. In Eleventh International AAAI Conference on Web and Social Media.
[35]
Hannah Jean Miller, Jacob Thebault-Spieker, Shuo Chang, Isaac Johnson, Loren Terveen, and Brent Hecht. 2016. “Blissfully Happy” or “Ready to Fight”: Varying Interpretations of Emoji. In Tenth International AAAI Conference on Web and Social Media.
[36]
Hannah Miller Hillberg, Zachary Levonian, Daniel Kluver, Loren Terveen, and Brent Hecht. 2018. What I See is What You Don't Get: The Effects of (Not) Seeing Emoji Rendering Differences across Platforms. Proc. ACM Hum.-Comput. Interact., Vol. 2, CSCW, Article 124 (nov 2018), bibinfonumpages24 pages. https://doi.org/10.1145/3274393
[37]
Meredith Ringel Morris, Annuska Zolyomi, Catherine Yao, Sina Bahram, Jeffrey P. Bigham, and Shaun K. Kane. 2016. "With Most of It Being Pictures Now, I Rarely Use It": Understanding Twitter's Evolving Accessibility to Blind Users. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI '16). Association for Computing Machinery, New York, NY, USA, 5506--5516. https://doi.org/10.1145/2858036.2858116
[38]
Maria Teresa Paratore, Maria Claudia Buzzi, Marina Buzzi, and Barbara Leporini. 2021. An enriched emoji picker to improve accessibility in mobile communications. In Human-Computer Interaction--INTERACT 2021: 18th IFIP TC 13 International Conference, Bari, Italy, August 30--September 3, 2021, Proceedings, Part I 18. Springer, 418--433. https://doi.org/10.1007/978--3-030--85623--6_25
[39]
Athira Pillai, Kristen Shinohara, and Garreth W. Tigwell. 2022. Website Builders Still Contribute To Inaccessible Web Design. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility (Athens, Greece) (ASSETS '22). Association for Computing Machinery, New York, NY, USA, Article 71, bibinfonumpages4 pages. https://doi.org/10.1145/3517428.3550368
[40]
Henning Pohl, Dennis Stanke, and Michael Rohs. 2016. EmojiZoom: Emoji Entry via Large Overview Maps. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services (Florence, Italy) (MobileHCI '16). Association for Computing Machinery, New York, NY, USA, 510--517. https://doi.org/10.1145/2935334.2935382
[41]
Monica A Riordan and Ella Glikson. 2020. On the hazards of the technology age: How using emojis affects perceptions of leaders. International Journal of Business Communication (2020), 2329488420971690. https://doi.org/10.1177/2329488420971690
[42]
Alexander Robertson, Walid Magdy, and Sharon Goldwater. 2018. Self-representation on twitter using emoji skin color modifiers. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 12. https://doi.org/10.1609/icwsm.v12i1.15055
[43]
Alexander Robertson, Walid Magdy, and Sharon Goldwater. 2021. Black or White but Never Neutral: How Readers Perceive Identity from Yellow or Skin-Toned Emoji. Proc. ACM Hum.-Comput. Interact., Vol. 5, CSCW2, Article 350 (oct 2021), bibinfonumpages23 pages. https://doi.org/10.1145/3476091
[44]
David Rodrigues, Diniz Lopes, Marlia Prada, Dominic Thompson, and Margarida V. Garrido. 2017. A Frown Emoji Can Be Worth a Thousand Words. Telemat. Inf., Vol. 34, 8 (dec 2017), 1532--1543. https://doi.org/10.1016/j.tele.2017.07.001
[45]
Morgan Klaus Scheuerman, Katta Spiel, Oliver L. Haimson, Foad Hamidi, and Stacy M. Branham. 2020. HCI Guidelines for Gender Equality and Inclusivity; version 1.1 updated 5/21/2020. https://www.morgan-klaus.com/gender-guidelines.html Accessed: 2023-07--12.
[46]
Esha Shandilya, Mingming Fan, and Garreth W. Tigwell. 2022. ?I Need to Be Professional until My New Team Uses Emoji, GIFs, or Memes First'': New Collaborators' Perspectives on Using Non-Textual Communication in Virtual Workspaces. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 528, bibinfonumpages13 pages. https://doi.org/10.1145/3491102.3517514
[47]
Accessible Social. [n.,d.]. Emojis. https://www.accessible-social.com/copy-and-formatting/emoji Accessed: 2023-07--18.
[48]
Statista. 2021. Number of people with blindness worldwide in 2020, by age and gender. https://www.statista.com/statistics/1237876/number-blindness-by-age-gender/ Accessed: 2023-07--11.
[49]
Satomi Sugiyama. 2015. Kawaii meiru and Maroyaka neko: Mobile emoji for relationship maintenance and aesthetic expressions among Japanese teens. First Monday (2015). https://doi.org/10.5210/fm.v20i10.5826
[50]
Garreth W. Tigwell and David R. Flatla. 2016. Oh That's What You Meant! Reducing Emoji Misunderstanding. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct (Florence, Italy) (MobileHCI '16). Association for Computing Machinery, New York, NY, USA, 859--866. https://doi.org/10.1145/2957265.2961844
[51]
Garreth W. Tigwell, Benjamin M. Gorman, and Rachel Menzies. 2020. Emoji Accessibility for Visually Impaired People. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '20). Association for Computing Machinery, New York, NY, USA, 1--14. https://doi.org/10.1145/3313831.3376267
[52]
Unicode. 2019. Members - Unicode. hrefhttps://home.unicode.org/membership/members/https://home.unicode.org/membership/members/. Accessed: 2019-09--18.
[53]
Benjamin Weissman and Darren Tanner. 2018. A strong wink between verbal and emoji-based irony: How the brain processes ironic emojis during language comprehension. PLOS ONE, Vol. 13, 8 (08 2018), 1--26. https://doi.org/10.1371/journal.pone.0201727
[54]
WHO. 2022. Blindness and vision impairment. https://www.who.int/news-room/fact-sheets/detail/blindness-and-visual-impairment Accessed: 2023-07--11.
[55]
Sarah Wiseman and Sandy J. J. Gould. 2018. Repurposing Emoji for Personalised Communication: Why Pizza Means ?I Love You". In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI '18). Association for Computing Machinery, New York, NY, USA, 1--10. https://doi.org/10.1145/3173574.3173726
[56]
Mingrui Ray Zhang, Ruolin Wang, Xuhai Xu, Qisheng Li, Ather Sharif, and Jacob O. Wobbrock. 2021b. Voicemoji: Emoji Entry Using Voice for Visually Impaired People. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 37, bibinfonumpages18 pages. https://doi.org/10.1145/3411764.3445338
[57]
Yiqiong Zhang, Min Wang, and Ying Li. 2021a. More than playfulness: Emojis in the comments of a WeChat official account. Internet Pragmatics, Vol. 4, 2 (2021), 247--271. https://doi.org/10.1075/ip.00048.zha
[58]
Rui Zhou, Jasmine Hentschel, and Neha Kumar. 2017. Goodbye Text, Hello Emoji: Mobile Communication on WeChat in China. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI '17). Association for Computing Machinery, New York, NY, USA, 748--759. https://doi.org/10.1145/3025453.3025800
[59]
Ben Zimmer. 2013. The cyberpragmatics of bounding asterisks. https://languagelog.ldc.upenn.edu/nll/?p=4466
Index Terms
- Party Face Congratulations! Exploring Design Ideas to Help Sighted Users with Emoji Accessibility when Messaging with Screen Reader Users
Recommendations
Emoji Accessibility for Visually Impaired People
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing SystemsEmoji are graphical symbols that appear in many aspects of our lives. Worldwide, around 36 million people are blind and 217 million have a moderate to severe visual impairment. This portion of the population may use and encounter emoji, yet it is unclear ...
Web accessibility for older users: successes and opportunities (keynote)
W4A '09: Proceedings of the 2009 International Cross-Disciplinary Conference on Web Accessibililty (W4A)Web accessibility for older users -- are we there yet? We understand many of the needs, however there are some gaps in our knowledge and much integration and adoption still required.
The Web Accessibility Initiative: Ageing Education and Harmonisation (...
Rich Representations of Visual Content for Screen Reader Users
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing SystemsAlt text (short for "alternative text") is descriptive text associated with an image in HTML and other document formats. Screen reader technologies speak the alt text aloud to people who are visually impaired. Introduced with HTML 2.0 in 1995, the alt ...
Comments
Information & Contributors
Information
Published In
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 26 April 2024
Published in PACMHCI Volume 8, Issue CSCW1
Check for updates
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 102Total Downloads
- Downloads (Last 12 months)102
- Downloads (Last 6 weeks)29
Reflects downloads up to 10 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in