research-article

Fragility of layout-based and visual GUI test scripts: an assessment study on a hybrid mobile application

Authors:

Riccardo Coppola,

Marco TorchianoAuthors Info & Claims

A-TEST 2019: Proceedings of the 10th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation

Pages 28 - 34

https://doi.org/10.1145/3340433.3342824

Published: 26 August 2019 Publication History

Abstract

Context: Albeit different approaches exist for automated GUI testing of hybrid mobile applications, the practice appears to be not so commonly adopted by developers. A possible reason for such a low diffusion can be the fragility of the techniques, i.e. the frequent need for maintaining test cases when the GUI of the app is changed.

Goal: In this paper, we perform an assessment of the maintenance needed by test cases for a hybrid mobile app, and the related fragility causes.

Methods: We evaluated a small test suite with a Layout-based testing tool (Appium) and a Visual one (EyeAutomate) and observed the changes needed by tests during the co-evolution with the GUI of the app.

Results: We found that 20% Layout-based test methods and 30% Visual test methods had to be modified at least once, and that each release induced fragilities in 3-4% of the test methods.

Conclusion: Fragility of GUI tests can induce relevant maintenance efforts in test suites of large applications. Several principal causes for fragilities have been identified for the tested hybrid application, and guidelines for developers are deduced from them.

References

[1]

Emil Alégroth, Zebao Gao, Rafael Oliveira, and Atif Memon. 2015. Conceptualization and evaluation of component-based testing unified with visual gui testing: an empirical study. In Software Testing, Verification and Validation (ICST), 2015 IEEE 8th International Conference on. IEEE, 1–10.

[2]

Emil Alégroth, Arvid Karlsson, and Alexander Radway. 2018. Continuous Integration and Visual GUI Testing: Benefits and Drawbacks in Industrial Practice. In Software Testing, Verification and Validation (ICST), 2018 IEEE 11th International Conference on. IEEE, 172–181.

[3]

Luca Ardito, Riccardo Coppola, Maurizio Morisio, and Marco Torchiano. 2019. Espresso vs. EyeAutomate: An Experiment for the Comparison of Two Generations of Android GUI Testing. In Proceedings of the Evaluation and Assessment on Software Engineering (EASE ’19). ACM, New York, NY, USA, 13–22.

Digital Library

[4]

L. Ardito, R. Coppola, M. Torchiano, and E. Alegroth. 2018. Towards Automated Translation between Generations of GUI-based Tests for Mobile Devices. In Proceedings of INTUITESTBEDS 2018, joint Workshop of the 4th International Workshop on User Interface Test Automation, and 8th Workshop on TESting Techniques for event BasED Software. ACM.

Digital Library

[5]

Stefan Bosnic, Ištvan Papp, and Sebastian Novak. 2016. The development of hybrid mobile applications with Apache Cordova. In 2016 24th Telecommunications Forum (TELFOR). IEEE, 1–4.

[6]

Andreas Bruns, Andreas Kornstadt, and Dennis Wichmann. 2009. Web application tests with selenium. IEEE software 26, 5 (2009).

Digital Library

[7]

Victor R Basili-Gianluigi Caldiera and H Dieter Rombach. 1994. Goal question metric paradigm. Encyclopedia of software engineering 1 (1994), 528–532.

[8]

Claes Claes, Wohlin, R Runeson, Per, H Höst, Martin, CO Ohlsson, Magnus, R Regnell, Björn, and Anders Wesslén. 2012. Experimentation in software engineering. Springer.

[9]

Riccardo Coppola, Maurizio Morisio, and Marco Torchiano. 2017. Scripted GUI Testing of Android Apps: A Study on Diffusion, Evolution and Fragility. In Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering. ACM, 22–32.

Digital Library

[10]

R. Coppola, M. Morisio, and M. Torchiano. 2018. Maintenance of Android Widget-Based GUI Testing: A Taxonomy of Test Case Modification Causes. In 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). 151–158.

[11]

R. Coppola, M. Morisio, and M. Torchiano. 2018. Mobile GUI Testing Fragility: A Study on Open-Source Android Applications. IEEE Transactions on Reliability (2018), 1–24.

[12]

Riccardo Coppola, Maurizio Morisio, Marco Torchiano, and Luca Ardito. 2019. Scripted GUI testing of Android open-source apps: evolution of test code and fragility causes. Empirical Software Engineering (18 May 2019).

[13]

Isabelle Dalmasso, Soumya Kanti Datta, Christian Bonnet, and Navid Nikaein. 2013. Survey, comparison and evaluation of cross platform mobile application development tools. In 2013 9th International Wireless Communications and Mobile Computing Conference (IWCMC). IEEE, 323–328.

[14]

Ronald Jabangwe, Henry Edison, and Anh Nguyen Duc. 2018. Software engineering process models for mobile app development: a systematic literature review. Journal of Systems and Software 145 (2018), 98–111.

[15]

Mona Erfani Joorabchi, Ali Mesbah, and Philippe Kruchten. 2013. Real challenges in mobile app development. In Empirical Software Engineering and Measurement, 2013 ACM/IEEE International Symposium on. IEEE, 15–24.

[16]

P. S. Kochhar, F. Thung, N. Nagappan, T. Zimmermann, and D. Lo. 2015. Understanding the Test Automation Culture of App Developers. In 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST). 1–10.

[17]

Maurizio Leotta, Andrea Stocco, Filippo Ricca, and Paolo Tonella. 2018. Pesto: Automated migration of DOM-based Web tests towards the visual approach. Software Testing, Verification And Reliability 28, 4 (2018), e1665.

[18]

Mario Linares-Vásquez, Kevin Moran, and Denys Poshyvanyk. 2017. Continuous, evolutionary and large-scale: A new perspective for automated mobile app testing. In Software Maintenance and Evolution (ICSME), 2017 IEEE International Conference on. IEEE, 399–410.

[19]

Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. 2014. An empirical analysis of flaky tests. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 643–653.

Digital Library

[20]

I. Malavolta, S. Ruberto, T. Soru, and V. Terragni. 2015. End Users’ Perception of Hybrid Mobile Apps in the Google Play Store. In 2015 IEEE International Conference on Mobile Services. 25–32.

Digital Library

[21]

Gaurang Shah, Prayag Shah, and Rishikesh Muchhala. 2014. Software testing automation using appium. International Journal of Current Engineering and Technology 4, 5 (2014), 3528–3531.

Cited By

Prazina IBecirovic SCogo EOkanovic V(2023)Methods for Automatic Web Page Layout Testing and Analysis: A ReviewIEEE Access10.1109/ACCESS.2023.324254911(13948-13964)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3242549
Vos TAho PPastor Ricos FRodriguez‐Valdes OMulders A(2021) testar – scriptless testing through graphical user interface Software Testing, Verification and Reliability10.1002/stvr.177131:3Online publication date: 25-Apr-2021
https://doi.org/10.1002/stvr.1771
Ricós FAho PVos TBoigues IBlasco EMartínez H(2020)Deploying TESTAR to Enable Remote Testing in an Industrial CI Pipeline: A Case-Based EvaluationLeveraging Applications of Formal Methods, Verification and Validation: Verification Principles10.1007/978-3-030-61362-4_31(543-557)Online publication date: 29-Oct-2020
https://doi.org/10.1007/978-3-030-61362-4_31

Index Terms

Fragility of layout-based and visual GUI test scripts: an assessment study on a hybrid mobile application
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Scripted GUI Testing of Android Apps: A Study on Diffusion, Evolution and Fragility
PROMISE: Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering

Background. Evidence suggests that mobile applications are not thoroughly tested as their desktop counterparts. In particular GUI testing is generally limited. Like web-based applications, mobile apps suffer from GUI test fragility, i.e. GUI test classes ...
Scripted GUI testing of Android open-source apps: evolution of test code and fragility causes
Abstract
Evidence from empirical studies suggests that mobile applications are not thoroughly tested as their desktop counterparts. In particular, GUI testing is generally limited. Like web-based applications, mobile apps suffer from GUI testing fragility, ...
Multi-objective test case prioritization for GUI applications
SAC '13: Proceedings of the 28th Annual ACM Symposium on Applied Computing

Test case prioritization techniques are proposed to schedule execution of test cases in order to improve testing effectiveness. Various coverage criteria are used as surrogates for test case prioritization. They are expected to improve testing ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

A-TEST 2019: Proceedings of the 10th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation

August 2019

41 pages

ISBN:9781450368506

DOI:10.1145/3340433

General Chair:
Tanja E. J. Vos,
Program Chairs:
Wishnu Prasetya,
Sinem Getir

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 August 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ESEC/FSE '19

Sponsor:

SIGSOFT

ESEC/FSE '19: 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

August 26 - 27, 2019

Tallinn, Estonia

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
266
Total Downloads

Downloads (Last 12 months)19
Downloads (Last 6 weeks)3

Reflects downloads up to 07 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Prazina IBecirovic SCogo EOkanovic V(2023)Methods for Automatic Web Page Layout Testing and Analysis: A ReviewIEEE Access10.1109/ACCESS.2023.324254911(13948-13964)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3242549
Vos TAho PPastor Ricos FRodriguez‐Valdes OMulders A(2021) testar – scriptless testing through graphical user interface Software Testing, Verification and Reliability10.1002/stvr.177131:3Online publication date: 25-Apr-2021
https://doi.org/10.1002/stvr.1771
Ricós FAho PVos TBoigues IBlasco EMartínez H(2020)Deploying TESTAR to Enable Remote Testing in an Industrial CI Pipeline: A Case-Based EvaluationLeveraging Applications of Formal Methods, Verification and Validation: Verification Principles10.1007/978-3-030-61362-4_31(543-557)Online publication date: 29-Oct-2020
https://doi.org/10.1007/978-3-030-61362-4_31

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents