research-article

Automated mobile UI test fragility: an exploratory assessment study on Android

Authors:

Riccardo Coppola,

Emanuele Raffero,

Marco TorchianoAuthors Info & Claims

INTUITEST 2016: Proceedings of the 2nd International Workshop on User Interface Test Automation

Pages 11 - 20

https://doi.org/10.1145/2945404.2945406

Published: 21 July 2016 Publication History

Abstract

Automated UI testing suffers from fragility due to continuous -although minor- changes in the UI of applications. Such fragility has been shown especially for the web domain, though no clear evidence is available for mobile applications. Our goal is to perform an exploratory assessment of the extent and causes of the fragiliy of UI automated tests for mobile applications. For this purpose, we analyzed a small test suite -that we developed using five different testing frameworks for an Android application (K-9 Mail) and observed the changes induced in the tests by the evolution of the UI. We found that up to 75% of code-based tests, and up to 100% of image recognition tests, had to be adapted because of the changes induced by the evolution of the application between two different versions. In addition we identified the main causes of such fragility: changes of identifiers, text or graphics, removal or relocation of elements, activity flow variation, execution time variation, and usage of physical buttons. The preliminary assessment showed that the fragility of UI tests can be a relevant issue also for mobile applications. A few common causes were found that can be used as the basis for providing guidelines for fragility avoidance and repair.

References

[1]

D. Amalfitano, A. R. Fasolino, and P. Tramontana. A gui crawling-based technique for android mobile application testing. In Software Testing, Verification and Validation Workshops (ICSTW), 2011 IEEE Fourth International Conference on, pages 252–261, 2011.

Digital Library

[2]

D. Amalfitano, A. R. Fasolino, and P. Tramontana. Mobiguitar: Automated model-based testing of mobile apps. Software, IEEE, pages 53–59, 2015.

[3]

Android Developer’s Guide. Testing display performance. http://developer.android.com/training/ testing/performance.html/.

[4]

Android Developer’s Guide. Testing support library. http://developer.android.com/tools/testing-supportlibrary/index.html/.

[5]

S. Berner, R. Weber, and R. K. Keller. Observations and lessons learned from automated testing. In Proceedings of the 27th International Conference on Software Engineering, ICSE ’05, pages 571–579, New York, NY, USA, 2005. ACM.

Digital Library

[6]

S. R. Choudhary, A. Gorla, and A. Orso. Automated test input generation for android: Are we there yet? In Proc. 30th IEEE/ACM International Conference on Automated Software Engineering, pages 429–440, 2015.

Digital Library

[7]

J. Gao, X. Bai, W.-T. Tsai, and T. Uehara. Mobile application testing: A tutorial. IEEE Computer, 47(2):46–55, 2014.

Digital Library

[8]

GitHub. K-9 mail - advanced email for android. https://github.com/k9mail/k-9/.

[9]

Google Code Archive. androidscreencast. https://code.google.com/archive/p/androidscreencast/.

[10]

C. Hu and I. Neamtiu. Automating gui testing for android applications. In Proceedings of the 6th International Workshop on Automation of Software Test, pages 77–83, 2011.

Digital Library

[11]

T. W. Knych and A. Baliga. Android application development and testability. In MOBILESoft’ 14, pages 37–40, 2014.

Digital Library

[12]

M. Leotta, D. Clerissi, F. Ricca, and P. Tonella. Capture-replay vs. programmable web testing: An empirical assessment during test case evolution. In Proc. of 20th Working Conference on Reverse Engineering (WCRE), 2013.

[13]

M. Leotta et al. Visual vs. dom-based web locators: An empirical study. Web Engineering. Springer International Publishing, pages 322–340, 2014.

[14]

A. M. Memon. Automatically repairing event sequence-based gui test suites for regression testing. ACM Trans. Softw. Eng. Methodol., 18(2):4:1–4:36, Nov. 2008.

Digital Library

[15]

Mobizen Homepage. http://www.mobizen.com/.

[16]

S. Gunasekaran and V. Bargavi. Survey on automation testing tools for mobile applications. International journal of Advanced Engineering Research and Science (IJIAERS), 2(11), 2015.

[17]

N. Saas, A. A. Bakar Husna, and N. Sham. Automated testing tools for mobile applications. In Information and Communication Technology for The Muslim World (ICT4M), 2014 The 5th International Conference on., 2014.

[18]

Selendroid Homepage. Quickstart. http://selendroid.io/quickStart.html/.

[19]

Vysor Homepage. http://www.vysor.io/.

[20]

C. Wohlin, P. Runeson, M. Höst, M. Ohlsson, B. Regnell, and A. Wesslén. Experimentation in Software Engineering - An Introduction. Kluwer Academic Publishers, 2000.

Digital Library

[21]

T. Yeh, C. Tsung-Hsiang, and R. C. Miller. Sikuli: using gui screenshots for search and automation. In Proc. 22nd annual ACM symposium on User interface software and technology., 2009.

Digital Library

Cited By

Rodríguez-Valdés Ovan der Vlist Kvan Dalen RMarín BVos T(2024)Scriptless and Seamless: Leveraging Probabilistic Models for Enhanced GUI Testing in Native Android ApplicationsResearch Challenges in Information Science10.1007/978-3-031-59468-7_10(83-91)Online publication date: 4-May-2024
https://doi.org/10.1007/978-3-031-59468-7_10
Azimi MYilmaz C(2023)AdapTV+: Enhancing Model-Based Test Adaptation for Smart TVs through Icon Recognition2023 IEEE 28th Pacific Rim International Symposium on Dependable Computing (PRDC)10.1109/PRDC59308.2023.00033(209-216)Online publication date: 24-Oct-2023
https://doi.org/10.1109/PRDC59308.2023.00033
Azimi MElgun CFirat AErata FYilmaz C(2023)AdapTV: A Model-Based Test Adaptation Approach for End-to-End User Interface Testing of Smart TVsIEEE Access10.1109/ACCESS.2023.326274611(32095-32118)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3262746
Show More Cited By

Index Terms

Automated mobile UI test fragility: an exploratory assessment study on Android
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

TERMINATOR: better automated UI test case prioritization
ESEC/FSE 2019: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Automated UI testing is an important component of the continuous integration process of software development. A modern web-based UI is an amalgam of reports from dozens of microservices written by multiple teams. Queries on a page that opens up another ...
An Experimental Study on Flakiness and Fragility of Randoop Regression Test Suites
Fundamentals of Software Engineering
Abstract
Randoop is a well-known tool that proposes a feedback-directed algorithm for automatic and random generation of unit tests for a given Java class. It automatically generates two test suites for the class under test: (1) an error-revealing test ...
Pro Android UI

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

INTUITEST 2016: Proceedings of the 2nd International Workshop on User Interface Test Automation

July 2016

20 pages

ISBN:9781450344128

DOI:10.1145/2945404

General Chairs:
Aho Pekka,
Tanja Vos

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 July 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ISSTA '16

Sponsor:

SIGSOFT

ISSTA '16: International Symposium on Software Testing and Analysis

July 21, 2016

Saarbrücken, Germany

Upcoming Conference

ISSTA '24

Sponsor:
sigsoft

33rd ACM SIGSOFT International Symposium on Software Testing and Analysis

September 16 - 20, 2024

Vienna , Austria

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
434
Total Downloads

Downloads (Last 12 months)40
Downloads (Last 6 weeks)3

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Rodríguez-Valdés Ovan der Vlist Kvan Dalen RMarín BVos T(2024)Scriptless and Seamless: Leveraging Probabilistic Models for Enhanced GUI Testing in Native Android ApplicationsResearch Challenges in Information Science10.1007/978-3-031-59468-7_10(83-91)Online publication date: 4-May-2024
https://doi.org/10.1007/978-3-031-59468-7_10
Azimi MYilmaz C(2023)AdapTV+: Enhancing Model-Based Test Adaptation for Smart TVs through Icon Recognition2023 IEEE 28th Pacific Rim International Symposium on Dependable Computing (PRDC)10.1109/PRDC59308.2023.00033(209-216)Online publication date: 24-Oct-2023
https://doi.org/10.1109/PRDC59308.2023.00033
Azimi MElgun CFirat AErata FYilmaz C(2023)AdapTV: A Model-Based Test Adaptation Approach for End-to-End User Interface Testing of Smart TVsIEEE Access10.1109/ACCESS.2023.326274611(32095-32118)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3262746
Nie LSaid KMa LZheng YZhao Y(2023)A systematic mapping study for graphical user interface testing on mobile appsIET Software10.1049/sfw2.1212317:3(249-267)Online publication date: 2-Mar-2023
https://dl.acm.org/doi/10.1049/sfw2.12123
Jansen TRicós FLuo Yvan der Vlist Kvan Dalen RAho PVos T(2022)Scriptless GUI Testing on Mobile Applications2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS57517.2022.00113(1103-1112)Online publication date: Dec-2022
https://doi.org/10.1109/QRS57517.2022.00113
Yoon JChung SShin KKim JHong SYoo S(2022)Repairing Fragile GUI Test Cases Using Word and Layout Embedding2022 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST53961.2022.00038(291-301)Online publication date: Apr-2022
https://doi.org/10.1109/ICST53961.2022.00038
Cruz LAbreu R(2021)On the Energy Footprint of Mobile Testing FrameworksIEEE Transactions on Software Engineering10.1109/TSE.2019.294616347:10(2260-2271)Online publication date: 1-Oct-2021
https://doi.org/10.1109/TSE.2019.2946163
Said KNie LAjibode AZhou X(2020)GUI testing for mobile applications: objectives, approaches and challengesProceedings of the 12th Asia-Pacific Symposium on Internetware10.1145/3457913.3457931(51-60)Online publication date: 1-Nov-2020
https://dl.acm.org/doi/10.1145/3457913.3457931
Pan ZChen JYao LChen Z(2020)Research on Functional Test of Mobile APP Based on Robot2020 IEEE 5th International Conference on Signal and Image Processing (ICSIP)10.1109/ICSIP49896.2020.9339300(960-964)Online publication date: 23-Oct-2020
https://doi.org/10.1109/ICSIP49896.2020.9339300
Uygun YOguz ROlmezogullari EAktas M(2020)On the Large-scale Graph Data Processing for User Interface Testing in Big Data Science Projects2020 IEEE International Conference on Big Data (Big Data)10.1109/BigData50022.2020.9378153(2049-2056)Online publication date: 10-Dec-2020
https://doi.org/10.1109/BigData50022.2020.9378153
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents