research-article

CydiOS: A Model-Based Testing Framework for iOS Apps

Authors:

Yongsheng Fang,

Chenxiong Qian,

Xiapu LuoAuthors Info & Claims

ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

Pages 1 - 13

https://doi.org/10.1145/3597926.3598033

Published: 13 July 2023 Publication History

Abstract

To make an app stand out in an increasingly competitive market, developers must ensure its quality to deliver a better user experience. UI testing is a popular technique for quality assurance, which can thoroughly test the app from the users’ perspective. However, while considerable research has already studied UI testing on the Android platform, there is no research on iOS. This paper introduces CydiOS, a novel approach to performing model-based testing for iOS apps. CydiOS enhances the existing static analysis to build a more complete static model for the app under test. We propose an approach to retrieve runtime information to obtain real-time app context that can be mapped in the model. To improve the effectiveness of UI testing, we also introduce a potential-aware search algorithm to guide testing execution. We compare CydiOS with four representative algorithms(i.e., random, depth-first, stoat, and ape). We have evaluated CydiOS on 50 popular apps from App Store, and the results show that CydiOS outperforms other tools, achieving both higher code coverage and screen coverage. We open source CydiOS at https://github.com/SoftWare2022Testing/CydiOS, and a demo video can be found there.

References

[1]

2022. A jailbreak detection bypass for modern iOS jailbreaks. https:/https://github.com/jjolano/shadow/

[2]

2022. A lightweight and modular front-end framework for developing fast and powerful web interfaces. https://getuikit.com/

[3]

2022. addAction. https://developer.apple.com/documentation/uikit/uialertcontroller/1620094-addaction

[4]

2022. Android Debug Bridge (adb). https://developer.android.com/studio/command-line/adb

[5]

2022. App Review. https://developer.apple.com/app-store/review/

[6]

2022. Apple Developer Forums. https://developer.apple.com/forums/

[7]

2022. Brief discussion about iOS swizzle. https://juejin.cn/post/6844903856497754126

[8]

2022. checkra. https://checkra.in/

[9]

2022. class-dump: Generate Objective-C headers from Mach-O files. https://github.com/nygard/class-dump

[10]

2022. Collaborative List of Open-Source iOS Apps. https://github.com/dkhamsing/open-source-ios-apps

[11]

2022. Current ChildView Controller. https://stackoverflow.com/questions/14405490/current-childview-controller/

[12]

2022. Cydia Substrate. http://www.cydiasubstrate.com/

[13]

2022. Detect if iOS App is written in Swift. https://stackoverflow.com/questions/32882208/detect-if-ios-app-is-written-in-swift

[14]

2022. Dynamic instrumentation toolkit for developers, reverse-engineers, and security researchers. https://frida.re/

[15]

2022. Facebook WebDriverAgent. https://github.com/facebookarchive/WebDriverAgent/

[16]

2022. Fastmonkey. https://github.com/zhangzhao4444/Fastmonkey.git

[17]

2022. Global Premium Smartphone Market Sales Reach Highest Ever in 2021. https://www.counterpointresearch.com/global-premium-smartphone-market-2021/

[18]

2022. How to view crash reports in XCode. https://stackoverflow.com/questions/69123921/how-to-view-crash-reports-in-xcode

[19]

2022. introduction to ios test automation with xcuitest. https://testautomationu.applitools.com/introduction-to-ios-test-automation-with-xcuitest/chapter1.html/

[20]

2022. iOS App Dev Tutorials. https://developer.apple.com/tutorials/app-dev-training/

[21]

2022. iOS Interface Builder utility, implemented in python. https://github.com/davidquesada/ibtool

[22]

2022. iOS UIWindow. https://developer.apple.com/documentation/uikit/uiwindow

[23]

2022. iPhone Users and Sales Stats for 2022. https://backlinko.com/iphone-users/

[24]

2022. Jailbreak Downloads. https://theideviceblog.com/jailbreak-downloads/

[25]

2022. objc_msgsend. https://developer.apple.com/documentation/objectivec/1456712-objc_msgsend

[26]

2022. Programming with Objective-C: About Objective-C. https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/ProgrammingWithObjectiveC/Introduction/Introduction.html

[27]

2022. Simulate touch events for iOS User mode. https://github.com/Ret70/PTFakeTouch

[28]

2022. Swift. Method Dispatch. https://maxim-kryloff.medium.com/swift-method-dispatch-4ac7efab0388

[29]

2022. SwiftMonkey: A framework for doing randomised UI testing of iOS apps. https://github.com/zalando/SwiftMonkey/

[30]

2022. Testing with Xcode Code Coverage. https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/07-code_coverage.html

[31]

2022. The Right Way to Swizzle in Objective-C. https://newrelic.com/blog/best-practices/right-way-to-swizzle

[32]

2022. The UIAutomation Driver for iOS. http://appium.io/docs/en/drivers/ios-uiautomation/

[33]

2022. The View Controller Hierarchy. https://developer.apple.com/library/archive/featuredarticles/ViewControllerPGforiPhoneOS/TheViewControllerHierarchy.html

[34]

2022. The XCUITest Driver for iOS. https://appium.io/docs/en/drivers/ios-xcuitest/

[35]

2022. theos framework. https://github.com/theos/theos

[36]

2022. UINavigationController. https://developer.apple.com/documentation/uikit/uinavigationcontroller

[37]

2022. UITabBarController. https://developer.apple.com/documentation/uikit/uitabbarcontroller

[38]

2022. UIViewController presentedviewcontroller. https://developer.apple.com/documentation/uikit/uiviewcontroller/1621407-presentedviewcontroller

[39]

2022. Understanding Navigation in iOS. https://guides.codepath.com/ios/Understanding-Navigation-in-iOS

[40]

2022. Understanding Windows and Screens. https://developer.apple.com/library/archive/documentation/WindowsViews/Conceptual/WindowAndScreenGuide/WindowScreenRolesinApp/WindowScreenRolesinApp.html

[41]

2022. Unicorn Engine Introduction. https://ctf-wiki.mahaloz.re/reverse/unicorn/introduction/

[42]

2022. Using Segues. https://developer.android.com/studio/test/other-testing-tools/monkey

[43]

2022. Using Segues. https://developer.apple.com/library/archive/featuredarticles/ViewControllerPGforiPhoneOS/UsingSegues.html

[44]

2022. Write automated tests with UI Automator. https://developer.android.com/training/testing/other-components/ui-automator

[45]

Domenico Amalfitano, Anna Rita Fasolino, Porfirio Tramontana, Bryan Dzung Ta, and Atif M Memon. 2014. MobiGUITAR: Automated model-based testing of mobile apps. IEEE software, 32, 5 (2014), 53–59.

[46]

Saswat Anand, Mayur Naik, Mary Jean Harrold, and Hongseok Yang. 2012. Automated concolic testing of smartphone apps. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering. 1–11.

Digital Library

[47]

Tanzirul Azim and Iulian Neamtiu. 2013. Targeted and depth-first exploration for systematic testing of android apps. In Proceedings of the 2013 ACM SIGPLAN international conference on Object oriented programming systems languages & applications. 641–660.

Digital Library

[48]

Young-Min Baek and Doo-Hwan Bae. 2016. Automated model-based android gui testing using multi-level gui comparison criteria. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. 238–249.

Digital Library

[49]

Gogul Balakrishnan and Thomas Reps. 2004. Analyzing memory accesses in x86 executables. In Compiler Construction: 13th International Conference, CC 2004, Held as Part of the Joint European Conferences on Theory and Practice of Software, ETAPS 2004, Barcelona, Spain, March 29-April 2, 2004. Proceedings 13. 5–23.

[50]

Patrick Carter, Collin Mulliner, Martina Lindorfer, William Robertson, and Engin Kirda. 2016. Curiousdroid: automated user interface interaction for android application analysis sandboxes. In International Conference on Financial Cryptography and Data Security. 231–249.

[51]

Kai Chen, Xueqiang Wang, Yi Chen, Peng Wang, Yeonjoon Lee, XiaoFeng Wang, Bin Ma, Aohui Wang, Yingjun Zhang, and Wei Zou. 2016. Following devil’s footprints: Cross-platform analysis of potentially harmful libraries on android and ios. In 2016 IEEE Symposium on Security and Privacy (SP). 357–376.

[52]

Wontae Choi, George Necula, and Koushik Sen. 2013. Guided gui testing of android apps with minimal restart and approximate learning. Acm Sigplan Notices, 48, 10 (2013), 623–640.

Digital Library

[53]

Zhui Deng, Brendan Saltaformaggio, Xiangyu Zhang, and Dongyan Xu. 2015. iris: Vetting private api abuse in ios applications. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security. 44–56.

Digital Library

[54]

Manuel Egele, Christopher Kruegel, Engin Kirda, and Giovanni Vigna. 2011. PiOS: Detecting Privacy Leaks in iOS Applications. In NDSS. 177–183.

[55]

Johannes Feichtner, David Missmann, and Raphael Spreitzer. 2018. Automated Binary Analysis on iOS: A Case Study on Cryptographic Misuse in iOS Applications. In Proceedings of the 11th ACM Conference on Security & Privacy in Wireless and Mobile Networks. 236–247.

Digital Library

[56]

Xiang Gao, Shin Hwei Tan, Zhen Dong, and Abhik Roychoudhury. 2018. Android testing via synthetic symbolic execution. In 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE). 419–429.

Digital Library

[57]

Tianxiao Gu, Chengnian Sun, Xiaoxing Ma, Chun Cao, Chang Xu, Yuan Yao, Qirun Zhang, Jian Lu, and Zhendong Su. 2019. Practical GUI testing of Android applications via model abstraction and refinement. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). 269–280.

Digital Library

[58]

Vignir Gudmundsson, Mikael Lindvall, Luca Aceto, Johann Bergthorsson, and Dharmalingam Ganesan. 2016. Model-based Testing of Mobile Systems–An Empirical Study on QuizUp Android App. arXiv preprint arXiv:1606.00503.

[59]

Shuai Hao, Bin Liu, Suman Nath, William GJ Halfond, and Ramesh Govindan. 2014. Puma: Programmable ui-automation for large-scale dynamic analysis of mobile apps. In Proceedings of the 12th annual international conference on Mobile systems, applications, and services. 204–217.

Digital Library

[60]

Taeyeon Ki, Alexander Simeonov, Chang Min Park, Karthik Dantu, Steven Y Ko, and Lukasz Ziarek. 2017. Fully automated ui testing system for large-scale android apps using multiple devices. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services. 185–185.

Digital Library

[61]

Konrad Kollnig, Anastasia Shuba, Reuben Binns, Max Van Kleek, and Nigel Shadbolt. 2021. Are iPhones Really Better for Privacy? Comparative Study of iOS and Android Apps. arXiv preprint arXiv:2109.13722.

[62]

Yeonjoon Lee, Xueqiang Wang, Kwangwuk Lee, Xiaojing Liao, XiaoFeng Wang, Tongxin Li, and Xianghang Mi. 2019. Understanding $iOS-based$ Crowdturfing Through Hidden $UI$ Analysis. In 28th USENIX Security Symposium (USENIX Security 19). 765–781.

[63]

Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2017. Droidbot: a lightweight ui-guided test input generator for android. In 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C). 23–26.

[64]

Changlin Liu and Xusheng Xiao. 2021. ProMal: precise window transition graphs for Android via synergy of program analysis and machine learning. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). 144–146.

Digital Library

[65]

Xing Liu, Jiqiang Liu, Sencun Zhu, Wei Wang, and Xiangliang Zhang. 2019. Privacy risk analysis and mitigation of analytics libraries in the android ecosystem. IEEE Transactions on Mobile Computing, 19, 5 (2019), 1184–1199.

[66]

Aravind Machiry, Rohan Tahiliani, and Mayur Naik. 2013. Dynodroid: An input generation system for android apps. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. 224–234.

Digital Library

[67]

Riyadh Mahmood, Nariman Mirzaei, and Sam Malek. 2014. Evodroid: Segmented evolutionary testing of android apps. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. 599–609.

Digital Library

[68]

Ke Mao, Mark Harman, and Yue Jia. 2016. Sapienz: Multi-objective automated testing for android applications. In Proceedings of the 25th international symposium on software testing and analysis. 94–105.

Digital Library

[69]

Leon J Osterweil and Lloyd D Fosdick. 1976. Program testing techniques using simulated execution. ACM SIGSIM Simulation Digest.

[70]

Minxue Pan, An Huang, Guoxin Wang, Tian Zhang, and Xuandong Li. 2020. Reinforcement learning based curiosity-driven testing of Android applications. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 153–164.

Digital Library

[71]

Huy Xuan Pham, Hung Manh La, David Feil-Seifer, and Aria Nefian. 2018. Cooperative and distributed reinforcement learning of drones for field coverage. arXiv preprint arXiv:1803.07250.

[72]

Huy X Pham, Hung M La, David Feil-Seifer, and Luan V Nguyen. 2018. Autonomous uav navigation using reinforcement learning. arXiv preprint arXiv:1801.05086.

[73]

Xue Qin, Hao Zhong, and Xiaoyin Wang. 2019. Testmig: Migrating gui test cases from ios to android. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. 284–295.

Digital Library

[74]

Wei Song, Xiangxing Qian, and Jeff Huang. 2017. EHBDroid: Beyond GUI testing for Android applications. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). 27–37.

[75]

Ting Su, Guozhu Meng, Yuting Chen, Ke Wu, Weiming Yang, Yao Yao, Geguang Pu, Yang Liu, and Zhendong Su. 2017. Guided, stochastic model-based GUI testing of Android apps. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. 245–256.

Digital Library

[76]

Saghar Talebipour, Yixue Zhao, Luka Dojcilović, Chenggang Li, and Nenad Medvidović. 2021. UI Test Migration Across Mobile Platforms. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). 756–767.

[77]

Yutian Tang, Haoyu Wang, Xian Zhan, Xiapu Luo, Yajin Zhou, Hao Zhou, Qiben Yan, Yulei Sui, and Jacky Wai Keung. 2021. A systematical study on application performance management libraries for apps. IEEE Transactions on Software Engineering.

[78]

Aditya Atul Tirodkar and Sundeep Singh Khandpur. 2019. EarlGrey: iOS UI automation testing framework. In 2019 IEEE/ACM 6th International Conference on Mobile Software Engineering and Systems (MOBILESoft). 12–15.

[79]

Heila Van Der Merwe, Brink Van Der Merwe, and Willem Visser. 2012. Verifying android applications using Java PathFinder. ACM SIGSOFT Software Engineering Notes, 37, 6 (2012), 1–5.

Digital Library

[80]

Shengqian Yang, Haowei Wu, Hailong Zhang, Yan Wang, Chandrasekar Swaminathan, Dacong Yan, and Atanas Rountev. 2018. Static window transition graphs for Android. Automated Software Engineering, 25, 4 (2018), 833–873.

Digital Library

[81]

Xia Zeng, Dengfeng Li, Wujie Zheng, Fan Xia, Yuetang Deng, Wing Lam, Wei Yang, and Tao Xie. 2016. Automated test input generation for android: Are we really there yet in an industrial case? In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. 987–992.

Digital Library

Cited By

Tombeng MPungus SLontaan RKorompis RTerok R(2023)Integrated Attendance System using NFC Technology2023 5th International Conference on Cybernetics and Intelligent System (ICORIS)10.1109/ICORIS60118.2023.10352289(1-6)Online publication date: 6-Oct-2023
https://doi.org/10.1109/ICORIS60118.2023.10352289

Index Terms

CydiOS: A Model-Based Testing Framework for iOS Apps
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Code smells in iOS apps: how do they compare to Android?
MOBILESoft '17: Proceedings of the 4th International Conference on Mobile Software Engineering and Systems

With billions of app downloads, the Apple App Store and Google Play Store succeeded to conquer mobile devices. However, this success also challenges app developers to publish high-quality apps to keep attracting and satisfying end-users. In particular, ...
Binary compatible graphics support in Android for running iOS apps
Middleware '17: Proceedings of the 18th ACM/IFIP/USENIX Middleware Conference

Mobile apps make extensive use of GPUs on smartphones and tablets to access Web content. To support pervasive Web content, we introduce three key OS techniques for binary graphics compatibility necessary to build a real-world system to run iOS and ...
A Deep Dive into the Featured iOS Apps
Internetware '23: Proceedings of the 14th Asia-Pacific Symposium on Internetware

Millions of apps in markets have made it difficult for mobile users to find fancy and high quality apps. Mobile app markets have deployed mechanisms to recommend apps to users. Apple usually features apps in the iOS App Store, and mobile users could ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

July 2023

1554 pages

ISBN:9798400702211

DOI:10.1145/3597926

General Chair:
René Just
University of Washington, USA
,
Program Chair:
Gordon Fraser
University of Passau, Germany

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ISSTA '23

Sponsor:

SIGSOFT

ISSTA '23: 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

July 17 - 21, 2023

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '24

Sponsor:
sigsoft

33rd ACM SIGSOFT International Symposium on Software Testing and Analysis

September 16 - 20, 2024

Vienna , Austria

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
354
Total Downloads

Downloads (Last 12 months)354
Downloads (Last 6 weeks)14

Reflects downloads up to

Other Metrics

View Author Metrics

Citations

Cited By

Tombeng MPungus SLontaan RKorompis RTerok R(2023)Integrated Attendance System using NFC Technology2023 5th International Conference on Cybernetics and Intelligent System (ICORIS)10.1109/ICORIS60118.2023.10352289(1-6)Online publication date: 6-Oct-2023
https://doi.org/10.1109/ICORIS60118.2023.10352289

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents