research-article

A survey on the tool support for the automatic evaluation of mobile accessibility

Authors:

Marcelo Medeiros Eler,

Gordon FraserAuthors Info & Claims

DSAI '18: Proceedings of the 8th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion

Pages 286 - 293

https://doi.org/10.1145/3218585.3218673

Published: 20 June 2018 Publication History

Abstract

To make a mobile app or website accessible, developers must incorporate accessibility practices into the development activities and perform accessibility evaluations. Accessibility evaluation is usually a manual approach, which can be laborious and difficult to scale or reproduce. Often, the implementation and evaluation of accessibility requirements are overlooked by developers due to high demand and time to market pressure. Automating mobile accessibility evaluation can help, but few approaches and tools are currently available. In addition, few studies discuss for which accessibility guidelines automatic evaluation is supported. Therefore, the aim of this paper is to survey automated accessibility evaluation tools for mobile applications to identify which accessibility properties can be automatically evaluated given the available tools. We expect our findings will provide practitioners with an overview of which tools are available for automatic accessibility evaluation and what are their capabilities. By exploring the limitations of current approaches, we also discuss where developers have little support, thus providing guidance for future research, innovation and development in this field.

References

[1]

Domenico Amalfitano, Nicola Amatucci, Atif M. Memon, Porfirio Tramontana, and Anna Rita Fasolino. 2017. A general framework for comparing automatic testing techniques of Android mobile apps. Journal of Systems and Software 125 (2017), 322--343.

[2]

Domenico Amalfitano, Anna Rita Fasolino, Porfirio Tramontana, Salvatore De Carmine, and Atif M. Memon. 2012. Using GUI Ripping for Automated Testing of Android Applications. 258--261.

Digital Library

[3]

Domenico Amalfitano, Anna Rita Fasolino, Porfirio Tramontana, and Bryan Robbins. 2013. Chapter 1 - Testing Android Mobile Applications: Challenges, Strategies, and Approaches. In Advances in Computers. Vol. 89. Elsevier, 1--52.

[4]

Tanzirul Azim and Iulian Neamtiu. 2013. Targeted and depth-first exploration for systematic testing of android apps. ACM SIGPLAN Notices 48, 10 (2013), 641--660.

Digital Library

[5]

Young-Min Baek and Doo-Hwan Bae. 2016. Automated Model-based Android GUI Testing Using Multi-level GUI Comparison Criteria. ACM, 238--249.

Digital Library

[6]

Abhijeet Banerjee, Lee Kee Chong, Sudipta Chattopadhyay, and Abhik Roy-choudhury. 2014. Detecting Energy Bugs and Hotspots in Mobile Apps. 588--598.

Digital Library

[7]

BBC. 2018. BBC Mobile Accessibility Guidelines. (March 2018). http://www.bbc.co.uk/guidelines/futuremedia/accessibility/mobile

[8]

Giorgio Brajnik. 2004. Comparing accessibility evaluation tools: a method for tool effectiveness. Universal Access in the Information Society 3, 3 (01 Oct 2004), 252--263.

Digital Library

[9]

Wontae Choi, George Necula, and Koushik Sen. 2013. Guided GUI testing of android apps with minimal restart and approximate learning. ACM SIGPLAN Notices 48, 10 (2013), 623--640.

Digital Library

[10]

Shauvik Roy Choudhary, Alessandra Gorla, and Alessandro Orso. 2015. Automated Test Input Generation for Android: Are We There Yet? (E). 429--440.

[11]

J. de Sousa e Silva, A. Pereira, R. Gonçalves, and S. Gomes. 2014. State of the art of accessible development for smart devices: From a disable and not impaired point of view. In 2014 9th Iberian Conference on Information Systems and Technologies (CISTI). 1--5.

[12]

Marcelo Medeiros Eler, José Miguel Rojas, Yan Ge, and Gordon Fraser. 2018. Automated Accessibility Testing of Mobile Apps. In Proceedings 11th IEEE Conference on Software Testing, Validation and Verification. IEEE, 1--11.

[13]

forApp. 2018. forApp Mobile Accessibility Inspection Solution. (April 2018). http://www.forapp.org.

[14]

Clint Gibler, Jonathan Crussell, Jeremy Erickson, and Hao Chen. 2012. AndroidLeaks: Automatically Detecting Potential Privacy Leaks in Android Applications on a Large Scale. In Int. Conference on Trust and Trustworthy Computing (TRUST), Stefan Katzenbeisser, Edgar Weippl, L. Jean Camp, Melanie Volkamer, Mike Reiter, and Xinwen Zhang (Eds.). Springer Berlin Heidelberg, 291--307.

Digital Library

[15]

Google. 2016. Get started with Accessibility Scanner. (Nov. 2016). https://support.google.com/accessibility/android/answer/6376570.

[16]

Google. 2018. Android Lint. https://developer.android.com/studio/write/lint.html. (March 2018).

[17]

Google. 2018. EarlGrey: iOS UI Automation Test Framework. https://github.com/google/EarlGrey. (March 2018).

[18]

Google. 2018. Espresso. (March 2018). https://google.github.io/android-testing-support-library/docs/espresso/index.html.

[19]

Google. 2018. Robolectric. (March 2018). http://robolectric.org/.

[20]

Shuai Hao, Bin Liu, Suman Nath, William GJ Halfond, and Ramesh Govindan. 2014. PUMA: programmable Ul-automation for large-scale dynamic analysis of mobile apps. In Proceedings of the 12th annual international conference on Mobile systems, applications, and services. ACM, 204--217.

Digital Library

[21]

IBM. 2018. IBM AbilityLab Mobile Accessibility Checker. (April 2018). https://www-03.ibm.com/able/mobile-accessibility-checker.html

[22]

KIF. 2018. KIF: Keep It Functional - An iOS Functional Testing Framework. https://github.com/kif-framework/KIF. (March 2018).

[23]

Pavneet Singh Kochhar, Ferdian Thung, Nachiappan Nagappan, Thomas Zimmermann, and David Lo. 2015. Understanding the test automation culture of app developers. In 2015 IEEE 8th Int. Conf. on Software Testing, Verification and Validation (ICST). IEEE, 1--10.

[24]

Yepang Liu, Chang Xu, and Shing-Chi Cheung. 2014. Characterizing and Detecting Performance Bugs for Smartphone Applications. 1013--1024.

Digital Library

[25]

Aravind Machiry, Rohan Tahiliani, and Mayur Naik. 2013. Dynodroid An input generation system for android apps. ACM, 224--234.

Digital Library

[26]

Riyadh Mahmood, Nariman Mirzaei, and Sam Malek. 2014. Evodroid: Segmented evolutionary testing of android apps. ACM, 599--609.

Digital Library

[27]

Ke Mao, Mark Harman, and Yue Jia. 2016. Sapienz: Multi-objective Automated Testing for Android Applications. 94--105.

Digital Library

[28]

Microsoft. 2018. AccChecker. (March 2018). https://msdn.microsoft.com/pt-br/library/windows/desktop/hh920985(v=vs.85).aspx.

[29]

Microsoft. 2018. AccScope. (March 2018). https://msdn.microsoft.com/en-us/library/windows/desktop/dn433239(v=vs.85).aspx.

[30]

Microsoft. 2018. Inspect. (March 2018). https://msdn.microsoft.com/library/windows/desktop/Dd318521.

[31]

Dudekula Mohammad Rafi, Katam Reddy Kiran Moses, Kai Petersen, and Mika Mäntylä. 2012. Benefits and limitations of automated software testing: Systematic literature review and practitioner survey. In Proceedings of the 7th International Workshop on Automation of Software Test, AST 2012. 36--42.

Digital Library

[32]

Henry Muccini, Antonio Di Francesco, and Patrizio Esposito. 2012. Software Testing of Mobile Applications: Challenges and Future Research Directions. In Int. Workshop on Automation of Software Test (AST). IEEE Press, 29--35. http://dl.acm.org/citation.cfm?id=2663608.2663615

Digital Library

[33]

MWAC. 2018. Mobile Web Accessibility Checker. (April 2018). http://www.userlight.com/apps/mwac/.

[34]

Alberto Dumont Alves Oliveira and Marcelo Medeiros Eler. 2017. Strategies and Challenges on the Accessibility and Interoperability of e-Government Web Portals: A Case Study on Brazilian Federal Universities. In Computer Software and Applications Conference (COMPSAC), 2017 IEEE 41st Annual, Vol. 1. IEEE, 737--742.

[35]

Freddy Paz and Jose Antonio Pow-Sang. 2016. A systematic mapping review of usability evaluation methods for software development process. International Journal of Software Engineering and Its Applications 10 (2016), 165--178.

[36]

Colin Shanley. 2016. Cracking Accessibility on Mobile Devices: The definitive field guide to accessibility and digital inclusion for business managers and project teams. RS Books, 258.

[37]

D. Bernardo Silva, A. T. Endo, M. M. Eler, and V H. S. Durelli. 2016. An analysis of automated tests for mobile Android applications. In 2016 XLII Latin American Computing Conference (CLEI). 1--9.

[38]

Markel Vigo, Justin Brown, and Vivienne Conway. 2013. Benchmarking Web Accessibility Evaluation Tools: Measuring the Harm of Sole Reliance on Automated Tests. In Proceedings of the 10th International Cross-Disciplinary Conferenceon Web Accessibility (W4A '13). ACM, New York, NY, USA, Article 1, 10 pages.

Digital Library

[39]

W3C. 2015. Mobile Accessibility: How WCAG 2.0 and Other W3C/WAI Guidelines Apply to Mobile. (Feb. 2015). https://www.w3.org/TR/mobile-accessibility-mapping/

[40]

World Bank. 2011. Main Report - World Report on Disability. (Jan. 2011). http://documents.worldbank.org/curated/en/665131468331271288/Main-report

[41]

Samer Zein, Norsaremah Salleh, and John Grundy. 2016. A systematic mapping study of mobile application testing techniques. 117 (2016), 334--356.

Digital Library

[42]

Samer Zein, Norsaremah Salleh, and John Grundy. 2016. A systematic mapping study of mobile application testing techniques. Journal of Systems and Software 117 (2016), 334--356.

Digital Library

Cited By

Swearngin AWu JZhang XGomez ECoughenour JStukenborg RGarg BHughes GHilliard ABigham JNichols J(2024)Towards Automated Accessibility Report Generation for Mobile AppsACM Transactions on Computer-Human Interaction10.1145/367496731:4(1-44)Online publication date: 19-Sep-2024
https://dl.acm.org/doi/10.1145/3674967
Lee JLee BChoi SSeo JChoe E(2024)Identify, Adapt, PersistProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36595858:2(1-21)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659585
Taeb MSwearngin ASchoop ECheng RJiang YNichols J(2024)AXNav: Replaying Accessibility Tests from Natural LanguageProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642777(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642777
Show More Cited By

Index Terms

A survey on the tool support for the automatic evaluation of mobile accessibility
1. Human-centered computing
  1. Accessibility
    1. Accessibility systems and tools
  2. Ubiquitous and mobile computing
    1. Ubiquitous and mobile devices
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation

Recommendations

Evaluation of the Accessibility of Mobile Applications: Current Approaches and Challenges
HCI International 2023 – Late Breaking Papers
Abstract
The World Health Organization estimates that around 1.3 billion people worldwide are affected by some form of impairment. Ensuring accessibility of mobile applications has become imperative to meet the requirements of this group. Mobile ...
An Exploratory Study about Accessibility, Usability and User Experience with the Visually Impaired using Mobile Applications
SBQS '20: Proceedings of the XIX Brazilian Symposium on Software Quality

Accessibility, Usability and User eXperience (UX) are relevant software quality criteria. However, Visually Impaired People (VIP) still face barriers in their interaction with applications on mobile devices due to the lack of quality in the criteria ...
AET tool for hybrid accessibility evaluation
Abstract
Accessible websites are a necessity for everyone, especially for people with disabilities. However, this is an objective that has not been achieved yet and requires initiatives that continue guiding the way towards an increasingly inclusive web ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

DSAI '18: Proceedings of the 8th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion

June 2018

365 pages

ISBN:9781450364676

DOI:10.1145/3218585

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 June 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

DSAI 2018

DSAI 2018: 8th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion

June 20 - 22, 2018

Thessaloniki, Greece

Acceptance Rates

DSAI '18 Paper Acceptance Rate 17 of 23 submissions, 74%;

Overall Acceptance Rate 17 of 23 submissions, 74%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

27
Total Citations
View Citations
572
Total Downloads

Downloads (Last 12 months)81
Downloads (Last 6 weeks)3

Reflects downloads up to 14 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Swearngin AWu JZhang XGomez ECoughenour JStukenborg RGarg BHughes GHilliard ABigham JNichols J(2024)Towards Automated Accessibility Report Generation for Mobile AppsACM Transactions on Computer-Human Interaction10.1145/367496731:4(1-44)Online publication date: 19-Sep-2024
https://dl.acm.org/doi/10.1145/3674967
Lee JLee BChoi SSeo JChoe E(2024)Identify, Adapt, PersistProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36595858:2(1-21)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659585
Taeb MSwearngin ASchoop ECheng RJiang YNichols J(2024)AXNav: Replaying Accessibility Tests from Natural LanguageProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642777(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642777
Seixas Pereira LMatos MDuarte C(2024)Exploring Mobile Device Accessibility: Challenges, Insights, and Recommendations for Evaluation MethodologiesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642526(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642526
Perinello LGaggi O(2024)Accessibility of Mobile User Interfaces using Flutter and React Native2024 IEEE 21st Consumer Communications & Networking Conference (CCNC)10.1109/CCNC51664.2024.10454681(1-6)Online publication date: 6-Jan-2024
https://doi.org/10.1109/CCNC51664.2024.10454681
Zhang YChen SFan LChen CLi XChandra SBlincoe KTonella P(2023)Automated and Context-Aware Repair of Color-Related Accessibility Issues for Android AppsProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616329(1255-1267)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616329
Pinheiro MViana Wde Gois Ribeiro Darin T(2023)Why Should Red and Green Never Be Seen? Exploring Color Blindness Simulations as Tools to Create Chromatically Accessible GamesProceedings of the ACM on Human-Computer Interaction10.1145/36110267:CHI PLAY(165-196)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3611026
Ma SChen CKhalajzadeh HGrundy J(2023)A First Look at Dark Mode in Real-world Android AppsACM Transactions on Software Engineering and Methodology10.1145/360460733:1(1-26)Online publication date: 23-Nov-2023
https://dl.acm.org/doi/10.1145/3604607
Bouma-Sims EAcar Y(2023)Beyond the Boolean: How Programmers Ask About, Use, and Discuss GenderProceedings of the ACM on Human-Computer Interaction10.1145/35794617:CSCW1(1-31)Online publication date: 16-Apr-2023
https://dl.acm.org/doi/10.1145/3579461
Salehnamadi NHe ZMalek S(2023)Assistive-Technology Aided Manual Accessibility Testing in Mobile Apps, Powered by Record-and-ReplayProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580679(1-20)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580679
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents