research-article

Preference-wise Testing of Android Apps via Test Amplification

Authors:

Xuandong LiAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology, Volume 32, Issue 1

Article No.: 4, Pages 1 - 37

https://doi.org/10.1145/3511804

Published: 13 February 2023 Publication History

Abstract

Preferences, the setting options provided by Android, are an essential part of Android apps. Preferences allow users to change app features and behaviors dynamically, and therefore their impacts need to be considered when testing the apps. Unfortunately, few test cases explicitly specify the assignments of valid values to the preferences, or configurations, under which they should be executed, and few existing mobile testing tools take the impact of preferences into account or provide help to testers in identifying and setting up the configurations for running the tests. This article presents the Prefest approach to effective testing of Android apps with preferences. Given an Android app and a set of test cases for the app, Prefest amplifies the test cases with a small number of configurations to exercise more behaviors and detect more bugs that are related to preferences. In an experimental evaluation conducted on real-world Android apps, amplified test cases produced by Prefest from automatically generated test cases covered significantly more code of the apps and detected seven real bugs, and the tool’s test amplification time was at the same order of magnitude as the running time of the input test cases. Prefest’s effectiveness and efficiency in amplifying programmer-written test cases was comparable with that in amplifying automatically generated test cases.

References

[1]

Bestoun S. Ahmed and Kamal Z. Zamli. 2010. PSTG: A T-way strategy adopting particle swarm optimization. In Proceedings of the4th Asia International Conference on Mathematical/Analytical Modelling and Computer Simulation. IEEE Computer Society, 1–5.

Digital Library

[2]

Domenico Amalfitano, Nicola Amatucci, Anna Rita Fasolino, Porfirio Tramontana, Emily Kowalczyk, and Atif M. Memon. 2015. Exploiting the saturation effect in automatic random testing of Android applications. In Proceedings of the 2nd ACM International Conference on Mobile Software Engineering and Systems. IEEE Press, 33–43.

[3]

Domenico Amalfitano, Anna Rita Fasolino, Porfirio Tramontana, Salvatore De Carmine, and Atif M. Memon. 2012. Using GUI ripping for automated testing of Android applications. In Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering. ACM, 258–261.

Digital Library

[4]

Saswat Anand, Mayur Naik, Mary Jean Harrold, and Hongseok Yang. 2012. Automated concolic testing of smartphone apps. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering. ACM, 59.

Digital Library

[5]

Joseph Annuzzi, Lauren Darcey, and Shane Conder. 2014. Introduction to Android Application Development: Android Essentials. Pearson Education.

[6]

AppiumConf. 2019. Appium: Automation for Apps. Retrieved from http://appium.io/.

[7]

Tanzirul Azim and Iulian Neamtiu. 2013. Targeted and depth-first exploration for systematic testing of Android apps. In ACM SIGPLAN Notices, Vol. 48. ACM, 641–660.

[8]

Roderick Bloem, Robert Koenighofer, Franz Röck, and Michael Tautschnig. 2014. Automating test-suite augmentation. In Proceedings of the 14th International Conference on Quality Software. IEEE, 67–72.

Digital Library

[9]

Mariano Ceccato, Luca Gazzola, Fitsum Meshesha Kifetew, Leonardo Mariani, Matteo Orrù, and Paolo Tonella. 2019. Toward in-vivo testing of mobile applications. In Proceedings of the IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW). IEEE, 137–143.

[10]

Wontae Choi, George Necula, and Koushik Sen. 2013. Guided GUI testing of Android apps with minimal restart and approximate learning. In ACM SIGPLAN Notices, Vol. 48. ACM, 623–640.

[11]

David M. Cohen, Siddhartha R. Dalal, Michael L. Fredman, and Gardner C. Patton. 1997. The AETG system: An approach to testing based on combinatorial design. IEEE Trans. Softw. Eng. 23, 7 (1997), 437–444.

Digital Library

[12]

Myra B. Cohen, Matthew B. Dwyer, and Jiangfan Shi. 2008. Constructing interaction test suites for Highly-configurable systems in the presence of constraints: A greedy approach. IEEE Trans. Softw. Eng. 34, 5 (2008), 633–650.

Digital Library

[13]

Benjamin Danglot, Oscar Vera-Perez, Zhongxing Yu, Andy Zaidman, Martin Monperrus, and Benoit Baudry. 2019. A snowballing literature study on test amplification. J. Syst. Softw. 157 (2019), 110398.

[14]

Thiago Botti de Assis, André Augusto Menegassi, and Andre Takeshi Endo. 2019. Amplifying tests for cross-platform apps through test patterns. In SEKE. 55–74.

[15]

Google Developers. 2021. Android Debug Bridge. Retrieved from https://developer.android.com/studio/command-line/adb.

[16]

Google Developers. 2021. Documentation of Settings for Android Developers. Retrieved from https://developer.android.com/guide/topics/ui/settings.

[17]

Google Developers. 2021. Google Play. Retrieved from https://play.google.com.

[18]

Zhen Dong, Marcel Böhme, Lucia Cojocaru, and Abhik Roychoudhury. 2020. Time-travel testing of Android apps. In Proceedings of the 42nd International Conference on Software Engineering (ICSE’20). 1–12.

Digital Library

[19]

Xiang Gao, Shin Hwei Tan, Zhen Dong, and Abhik Roychoudhury. 2018. Android testing via synthetic symbolic execution. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. 419–429.

Digital Library

[20]

Tianxiao Gu, Chengnian Sun, Xiaoxing Ma, Chun Cao, Chang Xu, Yuan Yao, Qirun Zhang, Jian Lu, and Zhendong Su. 2019. Practical GUI testing of Android applications via model abstraction and refinement. In Proceedings of the 41st International Conference on Software Engineering. 269–280.

Digital Library

[21]

Shuai Hao, Bin Liu, Suman Nath, William G. J. Halfond, and Ramesh Govindan. 2014. PUMA: Programmable UI-automation for large-scale dynamic analysis of mobile apps. In Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 204–217.

Digital Library

[22]

Sayed Hashimi, Satya Komatineni, and Dave MacLean. 2011. Pro Android 3. Apress.

[23]

Reyhaneh Jabbarvand, Jun-Wei Lin, and Sam Malek. 2019. Search-based energy testing of Android. In Proceedings of the 41st International Conference on Software Engineering. 1119–1130.

Digital Library

[24]

Casper S. Jensen, Mukul R. Prasad, and Anders Møller. 2013. Automated testing with targeted event sequence generation. In Proceedings of the International Symposium on Software Testing and Analysis. ACM, 67–77.

Digital Library

[25]

Taeyeon Ki, Chang Min Park, Karthik Dantu, Steven Y. Ko, and Lukasz Ziarek. 2019. Mimic: UI compatibility testing system for Android apps. In Proceedings of the 41st International Conference on Software Engineering. 246–256.

Digital Library

[26]

Satya Komatineni, Dave MacLean, and Sayed Y. Hashimi. 2012. Pro Android 4. Vol. 1. Springer.

Digital Library

[27]

Emily Kowalczyk, Myra B. Cohen, and Atif M. Memon. 2018. Configurations in Android testing: They matter. In Proceedings of the 1st International Workshop on Advances in Mobile App Analysis. 1–6.

Digital Library

[28]

Duling Lai and Julia Rubin. 2019. Goal-driven exploration for Android applications. In Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering. 115–127.

Digital Library

[29]

Yu Lei, Raghu Kacker, D. Richard Kuhn, Vadim Okun, and James Lawrence. 2007. IPOG: A general strategy for T-Way software testing. In Proceedings of the 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-based Systems. 549–556.

Digital Library

[30]

Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2019. Humanoid: A deep learning-based approach to automated black-box Android app testing. In Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering. 1070–1073.

Digital Library

[31]

Chieh-Jan Mike Liang, Nic Lane, Niels Brouwers, Li Zhang, Börje Karlsson, Ranveer Chandra, and Feng Zhao. 2013. Contextual Fuzzing: Automated Mobile App Testing under Dynamic Device and Environment Conditions. Technical Report. MSR-TR-2013-100, Microsoft Research.

[32]

Chieh-Jan Mike Liang, Nicholas D. Lane, Niels Brouwers, Li Zhang, Börje F. Karlsson, Hao Liu, Yan Liu, Jun Tang, Xiang Shan, Ranveer Chandra et al. 2014. Caiipa: Automated large-scale mobile app testing through contextual fuzzing. In Proceedings of the 20th Annual International Conference on Mobile Computing and Networking. 519–530.

Digital Library

[33]

Jun-Wei Lin, Reyhaneh Jabbarvand, and Sam Malek. 2019. Test transfer across mobile apps through semantic mapping. In Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering. 42–53.

Digital Library

[34]

Jun-Wei Lin, Navid Salehnamadi, and Sam Malek. 2020. Test automation in open-source Android apps: A large-scale empirical study. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering (ASE’20). Association for Computing Machinery, New York, NY, 1078–1089. DOI:

Digital Library

[35]

Yifei Lu, Minxue Pan, Juan Zhai, Tian Zhang, and Xuandong Li. 2019. Preference-wise testing for Android applications. In Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’19). Association for Computing Machinery, New York, NY, 268–278.

Digital Library

[36]

Aravind Machiry, Rohan Tahiliani, and Mayur Naik. 2013. Dynodroid: An input generation system for Android apps. In Proceedings of the 9th Joint Meeting on Foundations of Software Engineering. ACM, 224–234.

Digital Library

[37]

Riyadh Mahmood, Nariman Mirzaei, and Sam Malek. 2014. EvoDroid: Segmented evolutionary testing of Android apps. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 599–609.

Digital Library

[38]

Ke Mao, Mark Harman, and Yue Jia. 2016. Sapienz: Multi-objective automated testing for Android applications. In Proceedings of the 25th International Symposium on Software Testing and Analysis. ACM, 94–105.

Digital Library

[39]

C. C. Michael, G. E. McGraw, M. A. Schatz, and C. C. Walton. 1997. Genetic algorithms for dynamic test data generation. In Proceedings of the 12th International Conference on Automated Software Engineering. IEEE Computer Society, 307.

[40]

Nariman Mirzaei, Hamid Bagheri, Riyadh Mahmood, and Sam Malek. 2015. SIG-Droid: Automated system input generation for Android applications. In Proceedings of the IEEE 26th International Symposium on Software Reliability Engineering (ISSRE). IEEE Computer Society, 461–471.

Digital Library

[41]

Nariman Mirzaei, Joshua Garcia, Hamid Bagheri, Alireza Sadeghi, and Sam Malek. 2016. Reducing combinatorics in GUI testing of Android applications. In Proceedings of the 38th International Conference on Software Engineering. ACM, 559–570.

Digital Library

[42]

Monkeyrunner. 2019. monkeyrunner. Retrieved from https://developer.android.com/studio/test/monkeyrunner/.

[43]

Stas Negara, Naeem Esfahani, and Raymond P. L. Buse. 2019. Practical Android test recording with Espresso test recorder. In Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice. 193–202.

Digital Library

[44]

Changhai Nie and Hareton Leung. 2011. A survey of combinatorial testing. ACM Comput. Surv. 43, 2 (2011), 11.

Digital Library

[45]

Minxue Pan, An Huang, Guoxin Wang, Tian Zhang, and Xuandong Li. 2020. Reinforcement learning based curiosity-driven testing of Android applications. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 153–164.

Digital Library

[46]

pcqpcq. 2019. Open-source-android-apps. Retrieved from https://github.com/pcqpcq/open-source-android-apps/.

[47]

Fabiano Pecorelli, Gemma Catolino, Filomena Ferrucci, Andrea De Lucia, and Fabio Palomba. 2020. Testing of mobile applications in the wild: A large-scale empirical study on Android apps. In Proceedings of the 28th International Conference on Program Comprehension. 296–307.

Digital Library

[48]

RobotiumTech. 2019. Android UI Testing Robotium. Retrieved from https://github.com/RobotiumTech/robotium.

[49]

Alireza Sadeghi, Reyhaneh Jabbarvand, and Sam Malek. 2017. PATDdroid: Permission-aware GUI testing of Android. In Proceedings of the 11th Joint Meeting on Foundations of Software Engineering. ACM, 220–232.

[50]

Gian Luca Scoccia, Anthony Peruma, Virginia Pujols, Ben Christians, and Daniel E. Krutz. 2019. An empirical history of permission requests and mistakes in open source Android apps. In Proceedings of the 16th International Conference on Mining Software Repositories (MSR’19). IEEE Press, 597–601. DOI:

Digital Library

[51]

Toshiaki Shiba, Tatsuhiro Tsuchiya, and Tohru Kikuno. 2004. Using artificial life techniques to generate test cases for combinatorial testing. In Proceedings of the 28th Annual International Computer Software and Applications Conference. 72–77.

Digital Library

[52]

Wei Song, Xiangxing Qian, and Jeff Huang. 2017. EHBDroid: Beyond GUI testing for Android applications. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering. 27–37.

Digital Library

[53]

Helge Spieker, Arnaud Gotlieb, Dusica Marijan, and Morten Mossige. 2017. Reinforcement learning for automatic test case prioritization and selection in continuous integration. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis. 12–22.

Digital Library

[54]

Ting Su, Guozhu Meng, Yuting Chen, Ke Wu, Weiming Yang, Yao Yao, Geguang Pu, Yang Liu, and Zhendong Su. 2017. Guided, stochastic model-based GUI testing of Android apps. In Proceedings of the 11th Joint Meeting on Foundations of Software Engineering. 245–256.

Digital Library

[55]

Shin Hwei Tan and Ziqiang Li. 2020. Collaborative bug finding for Android apps. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (ICSE’20). Association for Computing Machinery, New York, NY, 1335–1347. DOI:

Digital Library

[56]

Nikolai Tillmann and Wolfram Schulte. 2006. Unit tests reloaded: Parameterized unit testing with symbolic execution. IEEE Softw. 23, 4 (2006), 38–47.

Digital Library

[57]

Wenyu Wang, Dengfeng Li, Wei Yang, Yurui Cao, Zhenwen Zhang, Yuetang Deng, and Tao Xie. 2018. An empirical study of Android test generation tools in industrial cases. In Proceedings of the 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 738–748.

Digital Library

[58]

Ziyuan Wang, Baowen Xu, and Changhai Nie. 2008. Greedy heuristic algorithms to generate variable strength combinatorial test suite. In Proceedings of the 8th International Conference on Quality Software. IEEE Computer Society, 155–160.

Digital Library

[59]

Zhihong Xu and Gregg Rothermel. 2009. Directed test suite augmentation. In Proceedings of the 16th Asia-Pacific Software Engineering Conference. IEEE, 406–413.

Digital Library

[60]

Pingyu Zhang and Sebastian Elbaum. 2012. Amplifying tests to validate exception handling code. In Proceedings of the 34th International Conference on Software Engineering (ICSE). IEEE, 595–605.

Digital Library

[61]

Pingyu Zhang and Sebastian Elbaum. 2014. Amplifying tests to validate exception handling code: An extended study in the mobile application domain. ACM Trans. Softw. Eng. Methodol. 23, 4 (2014), 1–28.

Digital Library

Cited By

Wang J(2024)Research on Intelligent Analysis Method for the Impact of Running APP Software on Physical Fitness Indicators of College StudentsEAI Endorsed Transactions on Pervasive Health and Technology10.4108/eetpht.10.550610Online publication date: 26-Apr-2024
https://doi.org/10.4108/eetpht.10.5506
Marinho E(2024)Characterizing Resource Interaction Failures in Mobile ApplicationsProceedings of the 28th ACM International Systems and Software Product Line Conference - Volume B10.1145/3646548.3676601(11-16)Online publication date: 2-Sep-2024
https://dl.acm.org/doi/10.1145/3646548.3676601
Ji RZhu TZhu XChen CPan MZhang T(2023)Vision-Based Widget Mapping for Test Migration Across Mobile Platforms: Are We There Yet?2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)10.1109/ASE56229.2023.00068(1416-1428)Online publication date: 11-Sep-2023
https://doi.org/10.1109/ASE56229.2023.00068

Index Terms

Preference-wise Testing of Android Apps via Test Amplification
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Preference-wise testing for Android applications
ESEC/FSE 2019: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Preferences, the setting options provided by Android, are an essential part of Android apps. Preferences allow users to change app features and behaviors dynamically, and therefore, need to be thoroughly tested. Unfortunately, the specific preferences ...
A framework for testing Android apps by reusing test cases
MOBILESoft '19: Proceedings of the 6th International Conference on Mobile Software Engineering and Systems

Android apps are generally developed by an individual developer or a small team of developers, and the developers may not have experience of testing Android apps or they may not have experience of testing any software systems. Furthermore, even an ...
A compatibility testing platform for android multimedia applications

Along with the widespread use of smartphones, Android has become one of the major platforms for multimedia applications (apps). However, due to the fast evolution of Android operating system and the fragmentation of Android devices, it becomes important ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology

ACM Transactions on Software Engineering and Methodology Volume 32, Issue 1

January 2023

954 pages

ISSN:1049-331X

EISSN:1557-7392

DOI:10.1145/3572890

Editor:
Mauro Pezzè
USI Università della Svizzera italiana and SIT Schaffhausen Institute of Technology, Switzerland

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 February 2023

Online AM: 19 April 2022

Accepted: 14 January 2022

Revised: 14 December 2021

Received: 16 August 2020

Published in TOSEM Volume 32, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

Leading-edge Technology Program of Jiangsu Natural Science Foundation
National Natural Science Foundation of China
Hong Kong RGC General Research Fund (GRF)

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
663
Total Downloads

Downloads (Last 12 months)295
Downloads (Last 6 weeks)16

Reflects downloads up to 09 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang J(2024)Research on Intelligent Analysis Method for the Impact of Running APP Software on Physical Fitness Indicators of College StudentsEAI Endorsed Transactions on Pervasive Health and Technology10.4108/eetpht.10.550610Online publication date: 26-Apr-2024
https://doi.org/10.4108/eetpht.10.5506
Marinho E(2024)Characterizing Resource Interaction Failures in Mobile ApplicationsProceedings of the 28th ACM International Systems and Software Product Line Conference - Volume B10.1145/3646548.3676601(11-16)Online publication date: 2-Sep-2024
https://dl.acm.org/doi/10.1145/3646548.3676601
Ji RZhu TZhu XChen CPan MZhang T(2023)Vision-Based Widget Mapping for Test Migration Across Mobile Platforms: Are We There Yet?2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)10.1109/ASE56229.2023.00068(1416-1428)Online publication date: 11-Sep-2023
https://doi.org/10.1109/ASE56229.2023.00068
Nie LSaid KMa LZheng YZhao Y(2023)A systematic mapping study for graphical user interface testing on mobile appsIET Software10.1049/sfw2.1212317:3(249-267)Online publication date: 2-Mar-2023
https://dl.acm.org/doi/10.1049/sfw2.12123

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents