research-article

Preference-wise testing for Android applications

Authors:

Xuandong LiAuthors Info & Claims

ESEC/FSE 2019: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Pages 268 - 278

https://doi.org/10.1145/3338906.3338980

Published: 12 August 2019 Publication History

Abstract

Preferences, the setting options provided by Android, are an essential part of Android apps. Preferences allow users to change app features and behaviors dynamically, and therefore, need to be thoroughly tested. Unfortunately, the specific preferences used in test cases are typically not explicitly specified, forcing testers to manually set options or blindly try different option combinations. To effectively test the impacts of different preference options, this paper presents PREFEST, as a preference-wise enhanced automatic testing approach, for Android apps. Given a set of test cases, PREFEST can locate the preferences that may affect the test cases with a static and dynamic combined analysis on the app under test, and execute these test cases only under necessary option combinations. The evaluation shows that PREFEST can improve 6.8% code coverage and 12.3% branch coverage and find five more real bugs compared to testing with the original test cases. The test cost is reduced by 99% for both the number of test cases and the testing time, compared to testing under pairwise combination of options.

References

[1]

Bestoun S Ahmed and Kamal Z Zamli. 2010. PSTG: A T-Way Strategy Adopting Particle Swarm Optimization. In Proceedings of the 2010 Fourth Asia International Conference on Mathematical/Analytical Modelling and Computer Simulation. IEEE Computer Society, 1–5.

Digital Library

[2]

Domenico Amalfitano, Nicola Amatucci, Anna Rita Fasolino, Porfirio Tramontana, Emily Kowalczyk, and Atif M Memon. 2015. Exploiting the saturation effect in automatic random testing of Android applications. In Proceedings of the Second ACM International Conference on Mobile Software Engineering and Systems. IEEE Press, 33–43.

Digital Library

[3]

Domenico Amalfitano, Anna Rita Fasolino, Porfirio Tramontana, Salvatore De Carmine, and Atif M Memon. 2012. Using GUI ripping for automated testing of Android applications. In Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering. ACM, 258–261.

Digital Library

[4]

Saswat Anand, Mayur Naik, Mary Jean Harrold, and Hongseok Yang. 2012. Automated concolic testing of smartphone apps. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering. ACM, 59.

Digital Library

[5]

AppiumConf. 2019. Appium : Automation for Apps. http://appium.io/. {online, accessed 15-Feb-2019}.

[6]

Steven Arzt, Siegfried Rasthofer, Christian Fritz, Eric Bodden, Alexandre Bartel, Jacques Klein, Yves Le Traon, Damien Octeau, and Patrick McDaniel. 2014.

Digital Library

[7]

Flowdroid: Precise context, flow, field, object-sensitive and lifecycle-aware taint analysis for Android apps. Acm Sigplan Notices 49, 6 (2014), 259–269.

Digital Library

[8]

Tanzirul Azim and Iulian Neamtiu. 2013. Targeted and depth-first exploration for systematic testing of Android apps. In Acm Sigplan Notices, Vol. 48. ACM, 641–660.

Digital Library

[9]

Wontae Choi, George Necula, and Koushik Sen. 2013. Guided GUI testing of Android apps with minimal restart and approximate learning. In Acm Sigplan Notices, Vol. 48. ACM, 623–640.

Digital Library

[10]

Shauvik Roy Choudhary, Alessandra Gorla, and Alessandro Orso. 2015. Automated Test Input Generation for Android: Are We There Yet?(E). (2015), 429–440.

[11]

David M Cohen, Siddhartha R Dalal, Michael L Fredman, and Gardner C Patton. 1997. The AETG System: An Approach to Testing Based on Combinatorial Design. IEEE Transactions on Software Engineering 23, 7 (1997), 437–444.

Digital Library

[12]

Myra B Cohen, Matthew B Dwyer, and Jiangfan Shi. 2008. Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach. IEEE Transactions on Software Engineering 34, 5 (2008), 633–650.

Digital Library

[13]

Google Developers. 2018. Documentation of Settings for Android Developers. https://developer.android.com/guide/topics/ui/settings. {online, accessed 01-Sep- 2018}.

[14]

Google Developers. 2019. UI/Application Exerciser Monkey. https://developer. android.com/studio/test/monkey. {online, accessed 15-Feb-2019}.

[15]

EclEmmaTeam. 2019. JaCoCo: Java Code Coverage Library. https://www. eclemma.org/jacoco/. {online, accessed 15-Feb-2019}.

[16]

Shuai Hao, Bin Liu, Suman Nath, William GJ Halfond, and Ramesh Govindan. 2014. PUMA: programmable UI-automation for large-scale dynamic analysis of mobile apps. In Proceedings of the 12th annual international conference on Mobile systems, applications, and services. ACM, 204–217.

Digital Library

[17]

Casper S Jensen, Mukul R Prasad, and Anders Møller. 2013. Automated testing with targeted event sequence generation. In Proceedings of the 2013 International Symposium on Software Testing and Analysis. ACM, 67–77.

Digital Library

[18]

Patrick Lam, Eric Bodden, Ondrej Lhoták, and Laurie Hendren. 2011. The Soot framework for Java program analysis: a retrospective. In Cetus Users and Compiler Infastructure Workshop (CETUS 2011), Vol. 15. 35.

[19]

Yu Lei, Raghu Kacker, D Richard Kuhn, Vadim Okun, and James Lawrence. 2007. IPOG: A General Strategy for T-Way Software Testing. In Proceedings of the 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems. IEEE Computer Society, 549–556.

Digital Library

[20]

Aravind Machiry, Rohan Tahiliani, and Mayur Naik. 2013. Dynodroid: An input generation system for Android apps. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. ACM, 224–234.

Digital Library

[21]

Riyadh Mahmood, Nariman Mirzaei, and Sam Malek. 2014. Evodroid: Segmented evolutionary testing of Android apps. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 599–609.

Digital Library

[22]

Ke Mao, Mark Harman, and Yue Jia. 2016. Sapienz: multi-objective automated testing for Android applications. In Proceedings of the 25th International Symposium on Software Testing and Analysis. ACM, 94–105.

Digital Library

[23]

CC Michael, GE McGraw, MA Schatz, and CC Walton. 1997. Genetic algorithms for dynamic test data generation. In Proceedings of the 12th international conference on Automated software engineering (formerly: KBSE). IEEE Computer Society, 307.

Digital Library

[24]

Nariman Mirzaei, Hamid Bagheri, Riyadh Mahmood, and Sam Malek. 2015. SIGDroid: Automated system input generation for Android applications. In Proceedings of the 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE). IEEE Computer Society, 461–471.

Digital Library

[25]

Nariman Mirzaei, Joshua Garcia, Hamid Bagheri, Alireza Sadeghi, and Sam Malek. 2016. Reducing combinatorics in GUI testing of Android applications. In Proceedings of the 38th International Conference on Software Engineering. ACM, 559–570.

Digital Library

[26]

monkeyrunner. 2019. monkeyrunner. https://developer.android.com/studio/test/ monkeyrunner/. {online, accessed 15-Feb-2019}.

[27]

Changhai Nie and Hareton Leung. 2011. A survey of combinatorial testing. ACM Computing Surveys (CSUR) 43, 2 (2011), 11.

Digital Library

[28]

pcqpcq. 2019. open-source-android-apps. https://github.com/pcqpcq/opensource-android-apps/. {online, accessed 15-Feb-2019}.

[29]

RobotiumTech. 2019. Android UI Testing Robotium. https://github.com/ RobotiumTech/robotium. {online, accessed 15-Feb-2019}.

[30]

Alireza Sadeghi, Reyhaneh Jabbarvand, and Sam Malek. 2017. PATDdroid: permission-aware GUI testing of Android. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. ACM, 220–232.

Digital Library

[31]

Toshiaki Shiba, Tatsuhiro Tsuchiya, and Tohru Kikuno. 2004. Using Artificial Life Techniques to Generate Test Cases for Combinatorial Testing. In Proceedings of the 28th Annual International Computer Software and Applications Conference-Volume 01. IEEE Computer Society, 72–77.

Digital Library

[32]

Skylot. 2019. jadx : Dex to Java decompiler. https://github.com/skylot/jadx/. {online, accessed 15-Feb-2019}.

[33]

Wei Song, Xiangxing Qian, and Jeff Huang. 2017. EHBDroid: beyond GUI testing for Android applications. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering. IEEE Press, 27–37.

Digital Library

[34]

Ting Su, Guozhu Meng, Yuting Chen, Ke Wu, Weiming Yang, Yao Yao, Geguang Pu, Yang Liu, and Zhendong Su. 2017. Guided, stochastic model-based GUI testing of Android apps. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. ACM, 245–256.

Digital Library

Cited By

Lu YHou WPan MLi XSu Z(2024)Understanding and Finding Java Decompiler BugsProceedings of the ACM on Programming Languages10.1145/36498608:OOPSLA1(1380-1406)Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1145/3649860
Cao SPan MPei YYang WZhang TWang LLi XRoychoudhury APaiva AAbreu RStorey M(2024)Comprehensive Semantic Repair of Obsolete GUI Test Scripts for Mobile ApplicationsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639108(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639108
Sun JSu TLiu KPeng CZhang ZPu GXie TSu Z(2023)Characterizing and Finding System Setting-Related Defects in Android AppsIEEE Transactions on Software Engineering10.1109/TSE.2023.323644949:4(2941-2963)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TSE.2023.3236449
Show More Cited By

Index Terms

Preference-wise testing for Android applications
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Preference-wise Testing of Android Apps via Test Amplification
Preferences, the setting options provided by Android, are an essential part of Android apps. Preferences allow users to change app features and behaviors dynamically, and therefore their impacts need to be considered when testing the apps. Unfortunately, ...
Sapienz: multi-objective automated testing for Android applications
ISSTA 2016: Proceedings of the 25th International Symposium on Software Testing and Analysis

We introduce Sapienz, an approach to Android testing that uses multi-objective search-based testing to automatically explore and optimise test sequences, minimising length, while simultaneously maximising coverage and fault revelation. Sapienz combines ...
A compatibility testing platform for android multimedia applications

Along with the widespread use of smartphones, Android has become one of the major platforms for multimedia applications (apps). However, due to the fast evolution of Android operating system and the fragmentation of Android devices, it becomes important ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEC/FSE 2019: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

August 2019

1264 pages

ISBN:9781450355728

DOI:10.1145/3338906

General Chairs:
Marlon Dumas
University of Tartu, Estonia
,
Dietmar Pfahl
University of Tartu, Estonia
,
Program Chairs:
Sven Apel
Saarland University, Germany
,
Alessandra Russo
Imperial College, UK

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 August 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China

Conference

ESEC/FSE '19

Sponsor:

SIGSOFT

ESEC/FSE '19: 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

August 26 - 30, 2019

Tallinn, Estonia

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
483
Total Downloads

Downloads (Last 12 months)42
Downloads (Last 6 weeks)7

Reflects downloads up to 09 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lu YHou WPan MLi XSu Z(2024)Understanding and Finding Java Decompiler BugsProceedings of the ACM on Programming Languages10.1145/36498608:OOPSLA1(1380-1406)Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1145/3649860
Cao SPan MPei YYang WZhang TWang LLi XRoychoudhury APaiva AAbreu RStorey M(2024)Comprehensive Semantic Repair of Obsolete GUI Test Scripts for Mobile ApplicationsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639108(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639108
Sun JSu TLiu KPeng CZhang ZPu GXie TSu Z(2023)Characterizing and Finding System Setting-Related Defects in Android AppsIEEE Transactions on Software Engineering10.1109/TSE.2023.323644949:4(2941-2963)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TSE.2023.3236449
Auer MStahlbauer AFraser G(2023)Android Fuzzing: Balancing User-Inputs and Intents2023 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST57152.2023.00013(37-48)Online publication date: Apr-2023
https://doi.org/10.1109/ICST57152.2023.00013
Deng XYan JZhang SYan JZhang J(2023)Variable-strength combinatorial testing of exported activities based on misexposure predictionJournal of Systems and Software10.1016/j.jss.2023.111773204(111773)Online publication date: Oct-2023
https://doi.org/10.1016/j.jss.2023.111773
Marinho EFerreira FDiniz JFigueiredo E(2023)Evaluating testing strategies for resource related failures in mobile applicationsSoftware Quality Journal10.1007/s11219-023-09627-331:4(1121-1147)Online publication date: 21-Apr-2023
https://doi.org/10.1007/s11219-023-09627-3
Su TYan YWang JSun JXiong YPu GWang KSu Z(2021)Fully automated functional fuzzing of Android apps for detecting non-crashing logic bugsProceedings of the ACM on Programming Languages10.1145/34855335:OOPSLA(1-31)Online publication date: 15-Oct-2021
https://dl.acm.org/doi/10.1145/3485533
Su TWang JSu ZSpinellis DGousios GChechik MDi Penta M(2021)Benchmarking automated GUI testing for Android against real-world bugsProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3468264.3468620(119-130)Online publication date: 20-Aug-2021
https://dl.acm.org/doi/10.1145/3468264.3468620
Sun JSu TLi JDong ZPu GXie TSu ZCadar CZhang X(2021)Understanding and finding system setting-related defects in Android appsProceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3460319.3464806(204-215)Online publication date: 11-Jul-2021
https://dl.acm.org/doi/10.1145/3460319.3464806
Song WHan MHuang J(2021)IMGDroidProceedings of the 43rd International Conference on Software Engineering10.1109/ICSE43902.2021.00080(823-834)Online publication date: 22-May-2021
https://dl.acm.org/doi/10.1109/ICSE43902.2021.00080
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents