research-article

Empath-D: VR-based Empathetic App Design for Accessibility

Authors:

Kenny Tsu Wei Choo,

Rajesh Krishna BalanAuthors Info & Claims

MobiSys '18: Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services

Pages 123 - 135

https://doi.org/10.1145/3210240.3210331

Published: 10 June 2018 Publication History

Abstract

With app-based interaction increasingly permeating all aspects of daily living, it is essential to ensure that apps are designed to be inclusive and are usable by a wider audience such as the elderly, with various impairments (e.g., visual, audio and motor). We propose Empath-D, a system that fosters empathetic design, by allowing app designers, in-situ, to rapidly evaluate the usability of their apps, from the perspective of impaired users. To provide a truly authentic experience, Empath-D carefully orchestrates the interaction between a smartphone and a VR device, allowing the user to experience simulated impairments in a virtual world while interacting naturally with the app, using a real smartphone. By carefully orchestrating the VR-smartphone interaction, Empath-D tackles challenges such as preserving low-latency app interaction, accurate visualization of hand movement and low-overhead perturbation of I/O streams. Experimental results show that user interaction with Empath-D is comparable (both in accuracy and user perception) to real-world app usage, and that it can simulate impairment effects as effectively as a custom hardware simulator.

References

[1]

{n. d.}. AGNES (Age Gain Now Empathy Systems. ({n. d.}). Retrieved 2018-04-13 from http://agelab.mit.edu/agnes-age-gain-now-empathy-system

[2]

{n. d.}. US Section 508 Standards. ({n. d.}). Retrieved 2018-04-13 from https://www.section508.gov/

[3]

2008. Web Content Accessibility Guidelines (WCAG) 2.0. (11 December 2008). Retrieved 2018-04-13 from https://www.w3.org/TR/WCAG20/

[4]

2016. Samsung Galaxy S7 Specifications. (2016). Retrieved 2018-04-13 from http://www.samsung.com/global/galaxy/galaxy-s7/#!/spec

[5]

2017. Samsung Gear VR Specifications. (2017). Retrieved 2018-04-13 from http://www.samsung.com/global/galaxy/gear-vr/specs/

[6]

2018. SolvePnP, Camera Calibration and 3D Reconstruction, OpenCV. (2018). Retrieved 2018-04-13 from https://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html

[7]

Omid Abari, Dinesh Bharadia, Austin Duffield, and Dina Katabi. 2017. Enabling High-Quality Untethered Virtual Reality. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). USENIX Association, Boston, MA, 531--544. https://www.usenix.org/conference/nsdi17/technical-sessions/presentation/abari

Digital Library

[8]

Ardalan Amiri Sani, Kevin Boos, Min Hong Yun, and Lin Zhong. 2014. Rio: A System Solution for Sharing I/O Between Mobile Systems. In Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '14). ACM, New York, NY, USA, 259--272.

Digital Library

[9]

Halim Cagri Ates, Alexander Fiannaca, and Eelke Folmer. 2015. Immersive Simulation of Visual Impairments Using a Wearable See-through Display. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (TEI '15). ACM, New York, NY, USA, 225--228.

Digital Library

[10]

Kevin Boos, David Chu, and Eduardo Cuervo. 2016. FlashBack: Immersive Virtual Reality on Mobile Devices via Rendering Memoization. In Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '16). ACM, New York, NY, USA, 291--304.

Digital Library

[11]

Kenny Tsu Wei Choo, Rajesh Krishna Balan, Tan Kiat Wee, Jagmohan Chauhan, Archan Misra, and Youngki Lee. 2017. Empath-D: Empathetic Design for Accessibility. In Proceedings of the 18th International Workshop on Mobile Computing Systems and Applications (HotMobile '17). ACM, New York, NY, USA, 55--60.

Digital Library

[12]

Eduardo Cuervo, Alec Wolman, Landon P. Cox, Kiron Lebeck, Ali Razeen, Stefan Saroiu, and Madanlal Musuvathi. 2015. Kahawai: High-Quality Mobile Gaming Using GPU Offload. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '15). ACM, New York, NY, USA, 121--135.

Digital Library

[13]

Marshall Flax. 2018. Low Vision Simulators. (2018). Retrieved 2018-04-13 from https://www.lowvisionsimulators.com/

[14]

S. Garrido-Jurado, R. Mu noz Salinas, F.J. Madrid-Cuevas, and M.J. Marín-Jiménez. 2014. Automatic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recognition 47, 6 (2014), 2280--2292.

Digital Library

[15]

Greg Gay and Cindy Qi Li. 2010. AChecker: Open, Interactive, Customizable, Web Accessibility Checking. In Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A) (W4A '10). ACM, New York, NY, USA, Article 23, 2 pages.

Digital Library

[16]

Genymotion. {n. d.}. Genymotion Android Emulator. ({n. d.}). Retrieved 2018-04-13 from https://www.genymotion.com/

[17]

Shuai Hao, Bin Liu, Suman Nath, William G.J. Halfond, and Ramesh Govindan. 2014. PUMA: Programmable UI-automation for Large-scale Dynamic Analysis of Mobile Apps. In Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '14). ACM, New York, NY, USA, 204--217.

Digital Library

[18]

Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology. Vol. 52. Elsevier, 139--183.

[19]

Kazunori Higuchi, Yasuo Sakaguchi, Kazuhiko Sugiyama, and Tomoaki Nakano. 1999. Simulating the human vision of elderly for designing control panels. In Systems, Man, and Cybernetics, 1999. IEEE SMC'99 Conference Proceedings. 1999 IEEE International Conference on, Vol. 5. IEEE, 703--708.

[20]

Intel. 2016. Intel®RealSense™ Camera SR300 Product Specifications. (2016). Retrieved 2018-04-13 from https://ark.intel.com/products/92329/Intel-RealSense-Camera-SR300

[21]

Zeqi Lai, Y. Charlie Hu, Yong Cui, Linhui Sun, and Ningwei Dai. 2017. Furion: Engineering High-Quality Immersive Virtual Reality on Today's Mobile Devices. In Proceedings of the 23rd Annual International Conference on Mobile Computing and Networking (MobiCom '17). ACM, New York, NY, USA, 409--421.

Digital Library

[22]

Kyungmin Lee, Jason Flinn, T.J. Giuli, Brian Noble, and Christopher Peplin. 2013. AMC: Verifying User Interface Properties for Vehicular Applications. In Proceeding of the 11th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '13). ACM, New York, NY, USA, 1--12.

Digital Library

[23]

Gordon E Legge, Sing-Hang Cheung, Deyue Yu, Susana TL Chung, Hye-Won Lee, and Daniel P Owens. 2007. The case for the visual span as a sensory bottleneck in reading. Journal of Vision 7, 2 (2007), 9--9.

[24]

Bin Liu, Suman Nath, Ramesh Govindan, and Jie Liu. 2014. DECAF: Detecting and Characterizing Ad Fraud in Mobile Apps. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14). USENIX Association, Seattle, WA, 57--70. https://www.usenix.org/conference/nsdi14/technical-sessions/presentation/liu_bin

Digital Library

[25]

Aravind Machiry, Rohan Tahiliani, and Mayur Naik. 2013. Dynodroid: An Input Generation System for Android Apps. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2013). ACM, New York, NY, USA, 224--234.

Digital Library

[26]

Jennifer Mankoff, Holly Fait, and Ray Juang. 2005. Evaluating accessibility by simulating the experiences of users with vision or motor impairments. IBM Systems Journal 44, 3 (2005), 505--517.

Digital Library

[27]

Chulhong Min, Seungchul Lee, Changhun Lee, Youngki Lee, Seungwoo Kang, Seungpyo Choi, Wonjung Kim, and Junehwa Song. 2016. PADA: Power-aware Development Assistant for Mobile Sensing Applications. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '16). ACM, New York, NY, USA, 946--957.

Digital Library

[28]

Chulhong Min, Youngki Lee, Chungkuk Yoo, Seungwoo Kang, Sangwon Choi, Pillsoon Park, Inseok Hwang, Younghyun Ju, Seungpyo Choi, and Junehwa Song. 2015. Power Forecaster: Predicting Smartphone Power Impact of Continuous Sensing Applications at Pre-installation Time. In Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems (SenSys '15). ACM, New York, NY, USA, 31--44.

Digital Library

[29]

Produkt + Projekt Wolfgang Moll. {n. d.}. Age simulation suit GERT - the GERontolic Test suit. ({n. d.}). Retrieved 2018-04-13 from http://www.age-simulation-suit.com/

[30]

Alan Newell and Peter Gregor. 1988. Human computer interaction for people with disabilities. (1988).

[31]

Alan F Newell, Peter Gregor, Maggie Morgan, Graham Pullin, and Catriona Macaulay. 2011. User-sensitive inclusive design. Universal Access in the Information Society 10, 3 (2011), 235--243.

Digital Library

[32]

Nvidia. 2016. GeForce GTX 1080 Specifications. (2016). Retrieved 2018-04-13 from https://www.geforce.com/hardware/desktop-gpus/geforce-gtx-1080/specifications

[33]

National Institute on Aging. 2016. World's older population grows dramatically. (28 March 2016). Retrieved 2018-04-13 from https://www.nih.gov/news-events/news-releases/worlds-older-population-grows-dramatically

[34]

DG Pelli, JG Robson, et al. 1988. The design of a new letter chart for measuring contrast sensitivity. In Clinical Vision Sciences. Citeseer.

[35]

Android Open Source Project. 2017. SurfaceFlinger and Hardware Composer. (March 2017). Retrieved 2018-04-13 from https://source.android.com/devices/graphics/arch-sf-hwc

[36]

Vaibhav Rastogi, Yan Chen, and William Enck. 2013. AppsPlayground: Automatic Security Analysis of Smartphone Applications. In Proceedings of the Third ACM Conference on Data and Application Security and Privacy (CODASPY '13). ACM, New York, NY, USA, 209--220.

Digital Library

[37]

Lenin Ravindranath, Suman Nath, Jitendra Padhye, and Hari Balakrishnan. 2014. Automatic and Scalable Fault Detection for Mobile Applications. In Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '14). ACM, New York, NY, USA, 190--203.

Digital Library

[38]

IBM Accessibility Research. 2017. IBM Accessibility Checklist 7.0. (18 July 2017). Retrieved 2018-04-13 from http://www-03.ibm.com/able/guidelines/ci162/accessibility_checklist.html

[39]

Justin B. Rousek, Sonja Koneczny, and M. Susan Hallbeck. 2009. Simulating Visual Impairment to Detect Hospital Wayfinding Difficulties. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 53, 8 (Oct. 2009), 531--535.

[40]

Samsung. 2014. Samsung Galaxy S5 Specifications. (2014). Retrieved 2018-04-13 from http://www.samsung.com/uk/smartphones/galaxy-s5-g900f/SM-G900FZKABTU/

[41]

Alvy Ray Smith and James F. Blinn. 1996. Blue Screen Matting. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '96). ACM, New York, NY, USA, 259--268.

Digital Library

[42]

Herman Snellen. 1873. Probebuchstaben zur bestimmung der sehschärfe. Vol. 1. H. Peters.

[43]

G.J. Sullivan, J. R. Ohm, W. J. Han, and T. Wiegand. 2012. Overview of the High Efficiency Video Coding (HEVC) Standard. IEEE Transactions on Circuits and Systems for Video Technology 22, 12 (Dec 2012), 1649--1668.

Digital Library

[44]

Unity Technologies. {n. d.}. Unity. ({n. d.}). Retrieved 2018-04-13 from https://unity3d.com/

[45]

Sam Tregillus and Eelke Folmer. 2016. VR-STEP: Walking-in-Place Using Inertial Sensing for Hands Free Navigation in Mobile VR Environments. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1250--1255.

Digital Library

[46]

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600--612.

Digital Library

[47]

Fabian Werfel, Roman Wiche, Jochen Feitsch, and Christian Geiger. 2016. Empathizing Audiovisual Sense Impairments: Interactive Real-Time Illustration of Diminished Sense Perception. In Proc. of AH.

Digital Library

[48]

Yu Zhong, Astrid Weber, Casey Burkhardt, Phil Weaver, and Jeffrey P. Bigham. 2015. Enhancing Android Accessibility for Users with Hand Tremor by Reducing Fine Pointing and Steady Tapping. In Proceedings of the 12th Web for All Conference (W4A '15). ACM, New York, NY, USA, Article 29, 10 pages.

Digital Library

[49]

George J. Zimmerman. 1979. Zimmerman Low Vision Simulation Kit. (1979). Retrieved 2018-04-13 from http://www.lowvisionsimulationkit.com/

Cited By

Huang LChen B(2023)A scientometric analysis of research on people with visual impairments in the field of HCI design: mapping the intellectual structure and evolutionUniversal Access in the Information Society10.1007/s10209-023-01067-xOnline publication date: 15-Dec-2023
https://doi.org/10.1007/s10209-023-01067-x
Dudley JYin LGaraj VKristensson P(2023)Inclusive Immersion: a review of efforts to improve accessibility in virtual reality, augmented reality and the metaverseVirtual Reality10.1007/s10055-023-00850-827:4(2989-3020)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1007/s10055-023-00850-8
Zhang QBarbareschi GHuang YLi JPai YWard JKunze K(2022)Seeing our Blind Spots: Smart Glasses-based Simulation to Increase Design Students’ Awareness of Visual ImpairmentProceedings of the 35th Annual ACM Symposium on User Interface Software and Technology10.1145/3526113.3545687(1-14)Online publication date: 29-Oct-2022
https://dl.acm.org/doi/10.1145/3526113.3545687
Show More Cited By

Index Terms

Empath-D: VR-based Empathetic App Design for Accessibility
1. Human-centered computing

Recommendations

Examining Augmented Virtuality Impairment Simulation for Mobile App Accessibility Design
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

With mobile apps rapidly permeating all aspects of daily living with use by all segments of the population, it is crucial to support the evaluation of app usability for specific impaired users to improve app accessibility. In this work, we examine the ...
Empath-D: Empathetic Design for Accessibility
HotMobile '17: Proceedings of the 18th International Workshop on Mobile Computing Systems and Applications

We describe our vision for Empath-D, our system to enable Empathetic User Interface Design. Our key idea is to leverage Virtual and Augmented Reality (VR / AR) displays to provide an Immersive Reality environment, where developers/designers can emulate ...
Towards Sound Accessibility in Virtual Reality
ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction

Virtual reality (VR) leverages sight, hearing, and touch senses to convey virtual experiences. For d/Deaf and hard of hearing (DHH) people, however, information conveyed through sound may not be accessible. While prior work has explored making every day ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MobiSys '18: Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services

June 2018

560 pages

ISBN:9781450357203

DOI:10.1145/3210240

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMOBILE: ACM Special Interest Group on Mobility of Systems, Users, Data and Computing

In-Cooperation

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 June 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

MobiSys '18

Sponsor:

SIGMOBILE

MobiSys '18: The 16th Annual International Conference on Mobile Systems, Applications, and Services

June 10 - 15, 2018

Munich, Germany

Acceptance Rates

Overall Acceptance Rate 274 of 1,679 submissions, 16%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
570
Total Downloads

Downloads (Last 12 months)68
Downloads (Last 6 weeks)3

Reflects downloads up to 03 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Huang LChen B(2023)A scientometric analysis of research on people with visual impairments in the field of HCI design: mapping the intellectual structure and evolutionUniversal Access in the Information Society10.1007/s10209-023-01067-xOnline publication date: 15-Dec-2023
https://doi.org/10.1007/s10209-023-01067-x
Dudley JYin LGaraj VKristensson P(2023)Inclusive Immersion: a review of efforts to improve accessibility in virtual reality, augmented reality and the metaverseVirtual Reality10.1007/s10055-023-00850-827:4(2989-3020)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1007/s10055-023-00850-8
Zhang QBarbareschi GHuang YLi JPai YWard JKunze K(2022)Seeing our Blind Spots: Smart Glasses-based Simulation to Increase Design Students’ Awareness of Visual ImpairmentProceedings of the 35th Annual ACM Symposium on User Interface Software and Technology10.1145/3526113.3545687(1-14)Online publication date: 29-Oct-2022
https://dl.acm.org/doi/10.1145/3526113.3545687
Zhu FLyu ZSousa MGrossman T(2022)Touching The Droid: Understanding and Improving Touch Precision With Mobile Devices in Virtual Reality2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR55827.2022.00099(807-816)Online publication date: Oct-2022
https://doi.org/10.1109/ISMAR55827.2022.00099
Lehman SLing HTan C(2020)ARCHIE: A User-Focused Framework for Testing Augmented Reality Applications in the Wild2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)10.1109/VR46266.2020.1581006269928(903-912)Online publication date: Mar-2020
https://doi.org/10.1109/VR46266.2020.1581006269928
Lehman SLing HTan C(2020)ARCHIE: A User-Focused Framework for Testing Augmented Reality Applications in the Wild2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)10.1109/VR46266.2020.00013(903-912)Online publication date: Mar-2020
https://doi.org/10.1109/VR46266.2020.00013
Choo KBalan RLee YBrewster SFitzpatrick GCox AKostakos V(2019)Examining Augmented Virtuality Impairment Simulation for Mobile App Accessibility DesignProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300605(1-11)Online publication date: 2-May-2019
https://dl.acm.org/doi/10.1145/3290605.3300605
Roy QZakaria CPerrault SNancel MKim WMisra ACockburn A(2019)A Comparative Study of Pointing Techniques for Eyewear Using a Simulated Pedestrian EnvironmentHuman-Computer Interaction – INTERACT 201910.1007/978-3-030-29387-1_36(625-646)Online publication date: 25-Aug-2019
https://doi.org/10.1007/978-3-030-29387-1_36
Kim WChoo KLee YMisra ABalan R(2018)Empath-DProceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services10.1145/3210240.3211108(534-534)Online publication date: 10-Jun-2018
https://dl.acm.org/doi/10.1145/3210240.3211108

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten