Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3025453.3025846acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article
Public Access

Interaction Proxies for Runtime Repair and Enhancement of Mobile Application Accessibility

Published: 02 May 2017 Publication History

Abstract

We introduce interaction proxies as a strategy for runtime repair and enhancement of the accessibility of mobile applications. Conceptually, interaction proxies are inserted between an application's original interface and the manifest interface that a person uses to perceive and manipulate the application. This strategy allows third-party developers and researchers to modify an interaction without an application's source code, without rooting the phone, without otherwise modifying an application, while retaining all capabilities of the system (e.g., Android's full implementation of the TalkBack screen reader). This paper introduces interaction proxies, defines a design space of interaction re-mappings, identifies necessary implementation abstractions, presents details of implementing those abstractions in Android, and demonstrates a set of Android implementations of interaction proxies from throughout our design space. We then present a set of interviews with blind and low-vision people interacting with our prototype interaction proxies, using these interviews to explore the seamlessness of interaction, the perceived usefulness and potential of interaction proxies, and visions of how such enhancements could gain broad usage. By allowing third-party developers and researchers to improve an interaction, interaction proxies offer a new approach to personalizing mobile application accessibility and a new approach to catalyzing development, deployment, and evaluation of mobile accessibility enhancements.

Supplementary Material

suppl.mov (pn2960-file3.mp4)
Supplemental video
suppl.mov (pn2960p.mp4)
Supplemental video

References

[1]
Android Open Source Project. Accessibility Developer Checklist. http://developer.android.com/guide/topics/ui/accessibili ty/checklist.html#requirements
[2]
Android Open Source Project. Dashboards. http://developer.android.com/about/dashboards/index.html
[3]
Apple Inc. Use Guided Access with iPhone, iPad, and iPod touch. https://support.apple.com/en-us/HT202612
[4]
Jeffrey P. Bigham. (2014). Making the Web Easier to See with Opportunistic Accessibility Improvement. Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2014), 117--122.
[5]
Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Aubrey Tatarowicz, Brandyn White, Samuel White, and Tom Yeh. (2010). VizWiz: Nearly Real-time Answers to Visual Questions. Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2010), 333--342.
[6]
Jeffrey P. Bigham, Ryan S. Kaminsky, Richard E. Ladner, Oscar M. Danielsson, and Gordon L. Hempton. (2006). WebInSight: Making Web Images Accessible. Proceedings of the ACM Conference on Computers and Accessibility (ASSETS 2006), 181--188.
[7]
Jeffrey P. Bigham and Richard E. Ladner. (2007). Accessmonkey: a Collaborative Scripting Framework for Web Users and Developers. Proceedings of the International Conference on Web accessibility (W4A 2007), 25--34.
[8]
Jeffrey P. Bigham, Craig M. Prince, and Richard E. Ladner. (2008). WebAnywhere: a Screen Reader Onthe-Go. Proceedings of the International Workshop on Web Accessibility (W4A 2008), 73--82.
[9]
Erin L. Brady, Yu Zhong, Meredith Ringel Morris, and Jeffrey P. Bigham. (2013). Investigating the Appropriateness of Social Network Question Asking as a Resource for Blind Users. Proceedings of the Conference on Computer Supported Cooperative Work (CSCW 2013), 1225--1236.
[10]
Erin Brady, Meredith Ringel Morris, and Jeffrey P. Bigham. (2015). Gauging Receptiveness to Social Microvolunteering. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2015), 1055--1064.
[11]
Erin Brady, Meredith Ringel Morris, Yu Zhong, Samuel White, and Jeffrey P. Bigham. (2013). Visual Challenges in the Everyday Lives of Blind People. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2013), 2117--2126.
[12]
Jim A. Carter and David W. Fourne. (2007). Techniques to Assist in Developing Accessibility Engineers. Proceedings of the ACM Conference on Computers and Accessibility (ASSETS 2007), 123--130.
[13]
Tsung-Hsiang Chang, Tom Yeh, and Robert C. Miller. (2011). Associating The Visual Representation of User Interfaces with their Internal Structures and Metadata. Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2011), 245--256.
[14]
Alan Cooper. (1995). About Face: The Essentials of User Interface Design. John Wiley & Sons, Inc., New York, NY.
[15]
Morgan Dixon and James Fogarty. (2010). Prefab: Implementing Advanced Behaviors Using Pixel-Based Reverse Engineering of Interface Structure. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2010), 1525--1534.
[16]
Morgan Dixon, James Fogarty, and Jacob O. Wobbrock. (2012). A General-Purpose Target-Aware Pointing Enhancement using Pixel-Level Analysis of Graphical Interfaces. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2012), 3167--3176.
[17]
Morgan Dixon, Gierad Laput, and James Fogarty. (2014). Pixel-Based Methods for Widget State and Style in a Runtime Implementation of Sliding Widgets. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2014), 2231--2240.
[18]
Morgan Dixon, Daniel Leventhal, and James Fogarty. (2011). Content and Hierarchy in Pixel-Based Methods for Reverse Engineering Interface Structure. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2011), 969--978.
[19]
Morgan Dixon, A. Conrad Nied, and James Fogarty. (2014). Prefab Layers and Prefab Annotations: Extensible Pixel-Based Interpretation of Graphical Interfaces. Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2014), 221--230.
[20]
James R. Eagan, Michel Beaudouin-Lafon, and Wendy E. Mackay. (2011). Cracking the Cocoa Nut: User Interface Programming at Runtime. Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2011), 225--234.
[21]
W. Keith Edwards, Scott E. Hudson, Joshua Marinacci, Roy Rodenstein, Thomas Rodriguez, and Ian E. Smith. (1997). Systematic Output Modification in a 2D User Interface Toolkit. Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 1997), 151--158.
[22]
Leah Findlater, Alex Jansen, Kristen Shinohara, Morgan Dixon, Peter Kamb, Joshua Rakita, and Jacob O. Wobbrock. (2010). Enhanced Area Cursors: Reducing Fine-Pointing Demands for People with Motor Impairments. Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2010), 153--162.
[23]
Krzysztof Z. Gajos, Daniel S. Weld, and Jacob O. Wobbrock. (2010). Automatically Generating Personalized User Interfaces with SUPPLE. Artificial Intelligence, 174(12--13), 910--950.
[24]
Google Inc. Accessibility Scanner. https://play.google.com/store/apps/details?id=com.goo gle.android.apps.accessibility.auditor
[25]
Vicki L. Hanson and John T. Richards. (2013). Progress on Website Accessibility? ACM Transactions on the Web, 7(1), 2:1--2:30.
[26]
Kyle J. Harms, Jordana H. Kerr, and Caitlin L. Kelleher. (2011). Improving Learning Transfer from Stencils-Based Tutorials. Proceedings of the International Conference on Interaction Design and Children (IDC 2011), 157--160.
[27]
Yun Huang, Brian Dobreski, Bijay Bhaskar Deo, Jiahang Xin, Natã Miccael Barbosa, Yang Wang, and Jeffrey P. Bigham. (2015). CAN: Composable Accessibility Infrastructure via Data-Driven Crowdsourcing. Proceedings of the Web for All Conference (W4A 2015), 1--10.
[28]
Amy Hurst, Scott E. Hudson, and Jennifer Mankoff. (2010). Automatically Identifying Targets Users Interact With During Real World Tasks. Proceedings of the International Conference on Intelligent User Interfaces (IUI 2010), 11--20.
[29]
Amy Hurst, Jennifer Mankoff, Anind K. Dey, and Scott E. Hudson. (2007). Dirty Desktops: Using a Patina of Magnetic Mouse Dust to Make Common Interactor Targets Easier to Select. Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2007), 183--186.
[30]
Edwin L. Hutchins, James D. Hollan, and Donald A. Norman. (2009). Direct Manipulation Interfaces. Human--Computer Interaction, 1(4), 311--338.
[31]
IDI Web Accessibility Checker?: Web Accessibility Checker. http://achecker.ca/checker/index.php
[32]
Yeongjin Jang, Chengyu Song, Simon P. Chung, Tielei Wang, and Wenke Lee. (2014). A11y Attacks: Exploiting Accessibility in Operating Systems. Proceedings of the ACM Conference on Computer and Communications Security (CCS 2014), 103--115.
[33]
Alex Jansen, Leah Findlater, and Jacob O. Wobbrock. (2011). From The Lab to The World: Lessons from Extending a Pointing Technique for Real-World Use. Extended Abstracts of the ACM Conference on Human Factors in Computing Systems (CHI 2011), 1867--1872.
[34]
Shaun K. Kane, Jeffrey P. Bigham, and Jacob O. Wobbrock. (2008). Slide Rule: Making Mobile Touch Screens Accessible to Blind People Using Multi-Touch Interaction Techniques. Proceedings of the ACM Conference on Computers and Accessibility (ASSETS 2008), 73--80.
[35]
Shaun K. Kane, Chandrika Jayant, Jacob O. Wobbrock, and Richard E. Ladner. (2009). Freedom to Roam: A Study of Mobile Device Adoption and Accessibility for People with Visual and Motor Disabilities. Proceedings of the ACM Conference on Computers and Accessibility (ASSETS 2009), 115--122.
[36]
Shaun K. Kane, Meredith Ringel Morris, Annuska Z. Perkins, Daniel J. Wigdor, Richard E. Ladner, and Jacob O. Wobbrock. (2011). Access Overlays: Improving Non-Visual Access to Large Touch Screens for Blind Users. Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2011), 273--282.
[37]
Shinya Kawanaka, Yevgen Borodin, Jeffrey P. Bigham, Darren Lunn, Hironobu Takagi, and Chieko Asakawa. (2008). Accessibility Commons: A Metadata Infrastructure for Web Accessibility. Proceedings of the ACM Conference on Computers and Accessibility (ASSETS 2008), 153--160.
[38]
Caitlin Kelleher and Randy Pausch. (2005). StencilsBased Tutorials: Design and Evaluation. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2005), 541--550.
[39]
Richard E. Ladner. (2015). Design for User Empowerment. Interactions, 22(2), 24--29.
[40]
Walter Lasecki, Christopher Miller, Adam Sadilek, Andrew Abumoussa, Donato Borrello, Raja Kushalnagar, and Jeffrey Bigham. (2012). Real-Time Captioning by Groups of Non-Experts. Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2012), 23--34.
[41]
Walter S. Lasecki, Phyo Thiha, Yu Zhong, Erin Brady, and Jeffrey P. Bigham. (2013). Answering Visual Questions with Conversational Crowd Assistants. Proceedings of the ACM Conference on Computers and Accessibility (ASSETS 2013), 18:1--18:8.
[42]
Jonathan Lazar, Daniel F. Goldstein, and Anne Taylor. (2015). Ensuring Digital Accessibility through Process and Policy. http://www.elsevier.com/books/ensuring-digitalaccessibility-through-process-and-policy/lazar/978-012--800646--7
[43]
Lauren R. Milne, Cynthia L. Bennett, and Richard E. Ladner. (2014). The Accessibility of Mobile Health Sensors for Blind Users. International Technology and Persons with Disabilities Conference Scientific/Research Proceedings (CSUN 2014), 166--175.
[44]
J. Morris and J. Mueller. (2014). Blind and Deaf Consumer Preferences for Android and iOS Smartphones. In Inclusive Designing. Springer International Publishing, Cham, 69--79.
[45]
Brad Myers, Scott E. Hudson, and Randy Pausch. (2000). Past, Present, and Future of User Interface Software Tools. ACM Transactions on ComputerHuman Interaction 7, 3--28.
[46]
Dan R. Olsen, Jr., Scott E. Hudson, Thom Verratti, Jeremy M. Heiner, and Matt Phelps. (1999). Implementing Interface Attachments Based on Surface Representations. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 1999), 191--198.
[47]
Elaine Pearson, Chrstopher Bailey, and Steve Green. (2011). A Tool to Support the Web Accessibility Evaluation Process for Novices. Proceedings of the Conference on Innovation and Technology in Computer Science Education (ITiCSE 2011), 28--32.
[48]
André Rodrigues. (2015). Breaking Barriers with Assistive Macros. Proceedings of the ACM Conference on Computers and Accessibility (ASSETS 2015), 351--352.
[49]
André Rodrigues and Tiago Guerreiro. (2014). SWAT: Mobile System-Wide Assistive Technologies. Proceedings of the International BCS Human Computer Interaction Conference (British HCI 2016), 341--346. https://dl.acm.org/citation.cfm?id=2742991
[50]
Franziska Roesner, James Fogarty, and Tadayoshi Kohno. (2012). User Interface Toolkit Mechanisms for Securing Interface Elements. Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2012), 239--250.
[51]
Daisuke Sato, Hironobu Takagi, Masatomo Kobayashi, Shinya Kawanaka, Chieko Asakawa, and Asakawa Chieko. (2010). Exploratory Analysis of Collaborative Web Accessibility Improvement. ACM Transactions on Accessible Computing (TACCESS), 3(2), 5:1--5:30.
[52]
Wolfgang Stuerzlinger, Olivier Chapuis, Dusty Phillips, and Nicolas Roussel. (2006). User Interface Façades: Towards Fully Adaptable User Interfaces. Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2006), 309--318.
[53]
Hironobu Takagi and Chieko Asakawa. (2000). Transcoding Proxy for Nonvisual Web Access. Proceedings of the ACM Conference on Assistive Technologies (ASSETS 2000), 164--171.
[54]
Hironobu Takagi, Shinya Kawanaka, Masatomo Kobayashi, Takashi Itoh, and Chieko Asakawa. (2008). Social Accessibility: Achieving Accessibility Through Collaborative Metadata Authoring. Proceedings of the ACM Conference on Computers and Accessibility (ASSETS 2008), 193--200.
[55]
Hironobu Takagi, Shinya Kawanaka, Masatomo Kobayashi, Daisuke Sato, and Chieko Asakawa. (2009). Collaborative Web Accessibility Improvement: Challenges and Possibilities. Proceedings of the ACM Conference on Computers and Accessibility (ASSETS 2009), 195--202.
[56]
Desney S. Tan, Brian Meyers, and Mary Czerwinski. (2004). WinCuts: Manipulating Arbitrary Window Regions for More Effective Use of Screen Space. Extended Abstracts of the ACM Conference on Human Factors in Computing Systems (CHI 2004), 1525--1528.
[57]
Jacob O. Wobbrock, James Fogarty, Shih-Yen (Sean) Liu, Shunichi Kimuro, and Susumu Harada. (2009). The Angle Mouse: Target-Agnostic Dynamic Gain Adjustment Based on Angular Deviation. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2009), 1401--1410.
[58]
Jacob O. Wobbrock, Shaun K. Kane, Krzysztof Z. Gajos, Susumu Harada, and Jon E. Froehlich. (2011). Ability-Based Design: Concept, Principles and Examples. ACM Transactions on Accessible Computing (TACCESS), 3(3), 1--27.
[59]
Aileen Worden, Nef Walker, Krishna Bharat, and Scott E. Hudson. (1997). Making Computers Easier for Older Adults to Use: Area Cursors and Sticky Icons. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 1997), 266--271.
[60]
World Health Organization. (2011). World Report on Disability. http://www.who.int/disabilities/world_report/2011/report/en/
[61]
Tom Yeh, Tsung-Hsiang Chang, and Robert C. Miller. (2009). Sikuli: Using GUI Screenshots for Search and Automation. Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2009), 183--192.
[62]
Jason Chen Zhao, Richard C. Davis, Pin Sym Foong, and Shengdong Zhao. (2015). CoFaçade: A Customizable Assistive Approach for Elders and Their Helpers. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2015), 1583--1592.
[63]
Yu Zhong, Astrid Weber, Casey Burkhardt, Phil Weaver, and Jeffrey P. Bigham. (2015). Enhancing Android Accessibility for Users with Hand Tremor by Reducing Fine Pointing and Steady Tapping. Proceedings of the Web for All Conference on (W4A 2015), 29:1--29:10.

Cited By

View all
  • (2024)InteractOut: Leveraging Interaction Proxies as Input Manipulation Strategies for Reducing Smartphone OveruseProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642317(1-19)Online publication date: 11-May-2024
  • (2024)AccessFixer: Enhancing GUI Accessibility for Low Vision Users With R-GCN ModelIEEE Transactions on Software Engineering10.1109/TSE.2023.333742150:2(173-189)Online publication date: Feb-2024
  • (2024)Understanding mobile GUI: From pixel-words to screen-sentencesNeurocomputing10.1016/j.neucom.2024.128200601(128200)Online publication date: Oct-2024
  • Show More Cited By

Index Terms

  1. Interaction Proxies for Runtime Repair and Enhancement of Mobile Application Accessibility

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
    May 2017
    7138 pages
    ISBN:9781450346559
    DOI:10.1145/3025453
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 02 May 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. accessibility
    2. interaction proxies
    3. runtime modification

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    CHI '17
    Sponsor:

    Acceptance Rates

    CHI '17 Paper Acceptance Rate 600 of 2,400 submissions, 25%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)149
    • Downloads (Last 6 weeks)13
    Reflects downloads up to 03 Oct 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)InteractOut: Leveraging Interaction Proxies as Input Manipulation Strategies for Reducing Smartphone OveruseProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642317(1-19)Online publication date: 11-May-2024
    • (2024)AccessFixer: Enhancing GUI Accessibility for Low Vision Users With R-GCN ModelIEEE Transactions on Software Engineering10.1109/TSE.2023.333742150:2(173-189)Online publication date: Feb-2024
    • (2024)Understanding mobile GUI: From pixel-words to screen-sentencesNeurocomputing10.1016/j.neucom.2024.128200601(128200)Online publication date: Oct-2024
    • (2024)Are your apps accessible? A GCN-based accessibility checker for low vision usersInformation and Software Technology10.1016/j.infsof.2024.107518174(107518)Online publication date: Oct-2024
    • (2023)Interaction Proxy ManagerProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36109297:3(1-39)Online publication date: 27-Sep-2023
    • (2023)BrushLens: Hardware Interaction Proxies for Accessible Touchscreen Interface ActuationProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606730(1-17)Online publication date: 29-Oct-2023
    • (2023)ScaleFix: An Automated Repair of UI Scaling Accessibility Issues in Android Applications2023 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58846.2023.00025(147-159)Online publication date: 1-Oct-2023
    • (2023)Screen Recognition: Creating Accessibility Metadata for Mobile Applications using View Type Detection2023 9th International Conference on Computer and Communications (ICCC)10.1109/ICCC59590.2023.10507590(1787-1793)Online publication date: 8-Dec-2023
    • (2022)A Systematic Survey on Android API Usage for Data-driven Analytics with SmartphonesACM Computing Surveys10.1145/353081455:5(1-38)Online publication date: 3-Dec-2022
    • (2022)A Large-Scale Longitudinal Analysis of Missing Label Accessibility Failures in Android AppsProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3502143(1-16)Online publication date: 29-Apr-2022
    • Show More Cited By

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Get Access

    Login options

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media