research-article

Espresso vs. EyeAutomate: An Experiment for the Comparison of Two Generations of Android GUI Testing

Authors:

Riccardo Coppola,

Maurizio Morisio,

Marco TorchianoAuthors Info & Claims

EASE '19: Proceedings of the 23rd International Conference on Evaluation and Assessment in Software Engineering

Pages 13 - 22

https://doi.org/10.1145/3319008.3319022

Published: 15 April 2019 Publication History

Abstract

Context: Different approaches exist for automated GUI testing of Android applications, each with its peculiarities, advantages, and drawbacks. The most common are either based on the structure of the GUI or use visual recognition.

Goal: In this paper, we present an empirical evaluation of two different GUI testing techniques with the use for each of a representative tool: (1) Visual GUI testing, with the use of EyeAutomate, and (2) Layout-based GUI testing, with the use of Espresso.

Method: We conducted an experiment with a population of 78 graduate students. The participants of the study were asked to create the same test suite for a popular, open-source Android app (Omni-Notes) with both the tools, and to answer a survey about their preference to the one or the other, and the perceived difficulties when developing the test scripts.

Results: By analyzing the outcomes of the delivered test suites (in terms of number of test scripts delivered and ratio of working ones) and the answers to the survey, we found that the participants showed similar productivity with both the tools, but the test suites developed with EyeAutomate were of higher quality (in terms of correctly working test scripts). The participants expressed a slight preference towards the EyeAutomate testing tool, reflecting a general complexity of Layout-based techniques -- represented by Espresso -- and some obstacles that may make the identification of components of the GUI quite a long and laborious task.

Conclusions: The evidence we collected can provide useful hints for researchers aiming at making GUI testing techniques for mobile applications more usable and effective.

References

[1]

{n. d.}. Omni Notes - App on Google Play. https://play.google.com/store/Apps/details?id=it.feio.android.omninotes&hl=en/.

[2]

Pekka Aho and Tanja Vos. 2018. Challenges in Automated Testing Through Graphical User Interface. In 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE.

[3]

Emil Alégroth and Robert Feldt. 2017. On the long-term use of visual gui testing in industrial practice: a case study. Empirical Software Engineering 22, 6 (2017), 2937--2971.

Digital Library

[4]

Emil Alégroth, Zebao Gao, Rafael Oliveira, and Atif Memon. 2015. Conceptualization and evaluation of component-based testing unified with visual gui testing: an empirical study. In Software Testing, Verification and Validation (ICST), 2015 IEEE 8th International Conference on. IEEE, 1--10.

[5]

Emil Alégroth, Arvid Karlsson, and Alexander Radway. 2018. Continuous Integration and Visual GUI Testing: Benefits and Drawbacks in Industrial Practice. In Software Testing, Verification and Validation (ICST), 2018 IEEE 11th International Conference on. IEEE, 172--181.

[6]

Domenico Amalfitano, Anna Rita Fasolino, Porfirio Tramontana, Bryan Ta, and Atif Memon. 2014. Mobiguitar-a tool for automated model-based testing of mobile apps. IEEE software 1 (2014), 1--1.

[7]

Luca Ardito, Riccardo Coppola, and Marco Torchiano. {n. d.}. Omni-Notes modified APK file. http://softeng.polito.it/coppola/omni-notes-good.apk.

[8]

Luca Ardito, Riccardo Coppola, and Marco Torchiano. {n. d.}. Responses to the survey questions. http://softeng.polito.it/coppola/survey_results.xlsx.

[9]

L. Ardito, R. Coppola, M. Torchiano, and E. Alegroth. 2018. Towards Automated Translation between Generations of GUI-based Tests for Mobile Devices. In Proceedings of INTUITESTBEDS 2018, joint Workshop of the 4th International Workshop on User Interface Test Automation, and 8th Workshop on TESting Techniques for event BasED Software. ACM.

Digital Library

[10]

Andreas Bruns, Andreas Kornstadt, and Dennis Wichmann. 2009. Web application tests with selenium. IEEE software 26, 5 (2009).

Digital Library

[11]

Victor R Basili-Gianluigi Caldiera and H Dieter Rombach. 1994. Goal question metric paradigm. Encyclopedia of software engineering 1 (1994), 528--532.

[12]

Claes Claes, Wohlin, R Runeson, Per, H Höst, Martin, CO Ohlsson, Magnus, R Regnell, Björn, and Anders Wesslén. 2012. Experimentation in software engineering. Springer.

[13]

Riccardo Coppola, Maurizio Morisio, and Marco Torchiano. 2017. Scripted GUI Testing of Android Apps: A Study on Diffusion, Evolution and Fragility. In Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering. ACM, 22--32.

Digital Library

[14]

R. Coppola, M. Morisio, and M. Torchiano. 2018. Mobile GUI Testing Fragility: A Study on Open-Source Android Applications. IEEE Transactions on Reliability (2018), 1--24.

[15]

Juliet M Corbin and Anselm Strauss. 1990. Grounded theory research: Procedures, canons, and evaluative criteria. Qualitative sociology 13, 1 (1990), 3--21.

[16]

Android Developers. {n. d.}. Espresso. https://developer.android.com/training/testing/espresso/.

[17]

Mattia Fazzini, Eduardo Noronha de A Freitas, Shauvik Roy Choudhary, and Alessandro Orso. 2017. Barista: A technique for recording, encoding, and running platform independent android tests. In Software Testing, Verification and Validation (ICST), 2017 IEEE International Conference on. IEEE, 149--160.

[18]

Lorenzo Gomez, Iulian Neamtiu, Tanzirul Azim, and Todd Millstein. 2013. Reran: Timing-and touch-sensitive record and replay for android. In Proceedings of the 2013 International Conference on Software Engineering. IEEE Press, 72--81.

Digital Library

[19]

IDC. {n. d.}. Smartphone Market Share 2017. https://www.idc.com/promo/smartphone-market-share/os.

[20]

Federico Iosue. {n. d.}. Open source note-taking application for Android. https://github.com/federicoiosue/Omni-Notes/.

[21]

P. S. Kochhar, F. Thung, N. Nagappan, T. Zimmermann, and D. Lo. 2015. Understanding the Test Automation Culture of App Developers. In 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST). 1--10.

[22]

Martin Kropp and Pamela Morales. 2010. Automated GUI testing on the Android platform. on Testing Software and Systems: Short Papers (2010), 67.

[23]

Maurizio Leotta, Diego Clerissi, Filippo Ricca, and Paolo Tonella. 2013. Capture-replay vs. programmable web testing: An empirical assessment during test case evolution. In Reverse Engineering (WCRE), 2013 20th Working Conference on. IEEE, 272--281.

[24]

Maurizio Leotta, Diego Clerissi, Filippo Ricca, and Paolo Tonella. 2014. Visual vs. DOM-based web locators: An empirical study. In International Conference on Web Engineering. Springer, 322--340.

[25]

Mario Linares-Vásquez, Carlos Bernal-Cárdenas, Kevin Moran, and Denys Poshyvanyk. 2017. How do developers test android applications?. In Software Maintenance and Evolution (ICSME), 2017 IEEE International Conference on. IEEE, 613--622.

[26]

Mario Linares-Vásquez, Kevin Moran, and Denys Poshyvanyk. 2017. Continuous, evolutionary and large-scale: A new perspective for automated mobile app testing. In Software Maintenance and Evolution (ICSME), 2017 IEEE International Conference on. IEEE, 399--410.

[27]

Aravind Machiry, Rohan Tahiliani, and Mayur Naik. 2013. Dynodroid: An input generation system for android apps. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. ACM, 224--234.

Digital Library

[28]

Yuhao Min and Shengcong Cai. 2018. Comparing Different Approaches of GUI Testing for Mobile Applications on Android Platform.

[29]

Ines Coimbra Morgado and Ana CR Paiva. 2015. The impact tool: Testing ui patterns on mobile applications. In 2015 30th IEEE/ACM International Conference on Automated Software Engineering(ASE). IEEE, 876--881.

Digital Library

[30]

Shiwangi Singh, Rucha Gadgil, and Ayushi Chudgor. 2014. Automated Testing of mobile applications using scripting Technique: A study on Appium. International Journal of Current Engineering and Technology (IJCET) 4, 5 (2014), 3627--3630.

[31]

AUQTUS Team. {n. d.}. Tools for Visual GUI Testing. http://eyeautomate.com/.

[32]

Porfirio Tramontana, Domenico Amalfitano, Nicola Amatucci, and Anna Rita Fasolino. 2018. Automated functional testing of mobile applications: a systematic mapping study. Software Quality Journal (2018), 1--53.

[33]

Wei Yang, Mukul R Prasad, and Tao Xie. 2013. A grey-box approach for automated GUI-model generation of mobile applications. In International Conference on Fundamental Approaches to Software Engineering. Springer, 250--265.

Digital Library

[34]

Tom Yeh, Tsung-Hsiang Chang, and Robert C Miller. 2009. Sikuli: using GUI screenshots for search and automation. In Proceedings of the 22nd annual ACM symposium on User interface software and technology. ACM, 183--192.

Digital Library

[35]

Yury Zhauniarovich, Anton Philippov, Olga Gadyatskaya, Bruno Crispo, and Fabio Massacci. 2015. Towards black box testing of android apps. In Availability, Reliability and Security (ARES), 2015 10th International Conference on. IEEE, 501--510.

Digital Library

Cited By

De Luca MFasolino ATramontana P(2024)Investigating the robustness of locators in template-based Web application testing using a GUI change classification modelJournal of Systems and Software10.1016/j.jss.2023.111932210:COnline publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1016/j.jss.2023.111932
Fulcini TCoppola RTorchiano MArdito L(2023)An analysis of widget layout attributes to support Android GUI-based testing2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW58534.2023.00033(117-125)Online publication date: Apr-2023
https://doi.org/10.1109/ICSTW58534.2023.00033
Gu RRojas J(2023)An Empirical Study on the Adoption of Scripted GUI Testing for Android Apps2023 38th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW)10.1109/ASEW60602.2023.00030(179-182)Online publication date: 11-Sep-2023
https://doi.org/10.1109/ASEW60602.2023.00030
Show More Cited By

Index Terms

Espresso vs. EyeAutomate: An Experiment for the Comparison of Two Generations of Android GUI Testing
1. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing theory, concepts and paradigms
      1. Mobile computing
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Empirical software validation
      2. Software defect analysis
        Software testing and debugging

Recommendations

Android Espresso Revealed: Writing Automated UI Tests
Testing of Mobile Applications in the Wild: A Large-Scale Empirical Study on Android Apps
ICPC '20: Proceedings of the 28th International Conference on Program Comprehension

Nowadays, mobile applications (a.k.a., apps) are used by over two billion users for every type of need, including social and emergency connectivity. Their pervasiveness in today's world has inspired the software testing research community in devising ...
A Static Approach to Prioritizing JUnit Test Cases

Test case prioritization is used in regression testing to schedule the execution order of test cases so as to expose faults earlier in testing. Over the past few years, many test case prioritization techniques have been proposed in the literature. Most ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

EASE '19: Proceedings of the 23rd International Conference on Evaluation and Assessment in Software Engineering

April 2019

345 pages

ISBN:9781450371452

DOI:10.1145/3319008

Program Chairs:
Shaukat Ali,
Vahid Garousi

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

IT University of Copenhagen

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 April 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

EASE '19

EASE '19: Evaluation and Assessment in Software Engineering

April 15 - 17, 2019

Copenhagen, Denmark

Acceptance Rates

EASE '19 Paper Acceptance Rate 20 of 73 submissions, 27%;

Overall Acceptance Rate 71 of 232 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
366
Total Downloads

Downloads (Last 12 months)40
Downloads (Last 6 weeks)2

Reflects downloads up to

Other Metrics

View Author Metrics

Citations

Cited By

De Luca MFasolino ATramontana P(2024)Investigating the robustness of locators in template-based Web application testing using a GUI change classification modelJournal of Systems and Software10.1016/j.jss.2023.111932210:COnline publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1016/j.jss.2023.111932
Fulcini TCoppola RTorchiano MArdito L(2023)An analysis of widget layout attributes to support Android GUI-based testing2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW58534.2023.00033(117-125)Online publication date: Apr-2023
https://doi.org/10.1109/ICSTW58534.2023.00033
Gu RRojas J(2023)An Empirical Study on the Adoption of Scripted GUI Testing for Android Apps2023 38th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW)10.1109/ASEW60602.2023.00030(179-182)Online publication date: 11-Sep-2023
https://doi.org/10.1109/ASEW60602.2023.00030
Bons AMarín BAho PVos T(2023)Scripted and scriptless GUI testing for web applications: An industrial caseInformation and Software Technology10.1016/j.infsof.2023.107172158(107172)Online publication date: Jun-2023
https://doi.org/10.1016/j.infsof.2023.107172
Qian JMa YLin CChen L(2022)Accelerating OCR-Based Widget Localization for Test Automation of GUI ApplicationsProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3556966(1-13)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3551349.3556966
Leotta MGarcía BRicca F(2022)An Empirical Study to Quantify the SetUp and Maintenance Benefits of Adopting WebDriverManagerQuality of Information and Communications Technology10.1007/978-3-031-14179-9_3(31-45)Online publication date: 5-Sep-2022
https://doi.org/10.1007/978-3-031-14179-9_3
Ardito LBottino ACoppola RLamberti FManigrasso FMorra LTorchiano M(2021)Feature Matching-based Approaches to Improve the Robustness of Android Visual GUI TestingACM Transactions on Software Engineering and Methodology10.1145/347742731:2(1-32)Online publication date: 17-Nov-2021
https://dl.acm.org/doi/10.1145/3477427
Junior NCosta HKarita LMachado ISoares L(2021)Experiences and Practices in GUI Functional Testing: A Software Practitioners’ ViewBrazilian Symposium on Software Engineering10.1145/3474624.3474640(195-204)Online publication date: 5-Oct-2021
https://doi.org/10.1145/3474624.3474640
Coppola RArdito LTorchiano MYaman SPrada RKifetew FCardozo N(2021)Automated translation of Android context-dependent gestures to visual GUI test instructionsProceedings of the 12th International Workshop on Automating TEST Case Design, Selection, and Evaluation10.1145/3472672.3473954(17-24)Online publication date: 23-Aug-2021
https://dl.acm.org/doi/10.1145/3472672.3473954
Althomali IKapfhammer GMcMinn P(2021)Automated visual classification of DOM‐based presentation failure reports for responsive web pagesSoftware Testing, Verification and Reliability10.1002/stvr.175631:4Online publication date: 14-Feb-2021
https://doi.org/10.1002/stvr.1756
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents