research-article

Seenomaly: vision-based linting of GUI animation effects against design-don't guidelines

Authors:

Zhenchang Xing,

Jinshui WangAuthors Info & Claims

ICSE '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

Pages 1286 - 1297

https://doi.org/10.1145/3377811.3380411

Published: 01 October 2020 Publication History

Abstract

GUI animations, such as card movement, menu slide in/out, snackbar display, provide appealing user experience and enhance the usability of mobile applications. These GUI animations should not violate the platform's UI design guidelines (referred to as design-don't guideline in this work) regarding component motion and interaction, content appearing and disappearing, and elevation and shadow changes. However, none of existing static code analysis, functional GUI testing and GUI image comparison techniques can "see" the GUI animations on the scree, and thus they cannot support the linting of GUI animations against design-don't guidelines. In this work, we formulate this GUI animation linting problem as a multi-class screencast classification task, but we do not have sufficient labeled GUI animations to train the classifier. Instead, we propose an unsupervised, computer-vision based adversarial autoencoder to solve this linting problem. Our autoencoder learns to group similar GUI animations by "seeing" lots of unlabeled real-application GUI animations and learning to generate them. As the first work of its kind, we build the datasets of synthetic and real-world GUI animations. Through experiments on these datasets, we systematically investigate the learning capability of our model and its effectiveness and practicality for linting GUI animations, and identify the challenges in this linting problem for future work.

References

[1]

[n.d.]. http://tools.android.com/tips/lint.

[2]

[n.d.]. https://github.com/stylelint/stylelint.

[3]

Samet Akcay, Amir Atapour-Abarghouei, and Toby P Breckon. 2018. Ganomaly: Semi-supervised anomaly detection via adversarial training. In Asian Conference on Computer Vision. Springer, 622--637.

[4]

Young-Min Baek and Doo-Hwan Bae. 2016. Automated model-based android gui testing using multi-level gui comparison criteria. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. ACM, 238--249.

Digital Library

[5]

Chunyang Chen, Sidong Feng, Zhenchang Xing, Linda Liu, Shengdong Zhao, and Jinshui Wang. 2019. Gallery DC: Design Search and Knowledge Discovery through Auto-created GUI Component Gallery. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1--22.

[6]

Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018. From ui design image to gui skeleton: a neural machine translator to bootstrap mobile gui implementation. In Proceedings of the 40th International Conference on Software Engineering. ACM, 665--676.

Digital Library

[7]

Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xiwei Xu, Liming Zhu, Guoqiang Li, and Jinshui Wang. 2020. Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learning. In 42nd International Conference on Software Engineering (ICSE '20). ACM, New York, NY, 13 pages.

Digital Library

[8]

Sen Chen, Lingling Fan, Chunyang Chen, Ting Su, Wenhe Li, Yang Liu, and Lihua Xu. 2019. Storydroid: Automated generation of storyboard for Android apps. In Proceedings of the 41st International Conference on Software Engineering. IEEE Press, 596--607.

Digital Library

[9]

Sen Chen, Lingling Fan, Chunyang Chen, Minhui Xue, Yang Liu, and Lihua Xu. 2019. GUI-Squatting Attack: Automated Generation of Android Phishing Apps. IEEE Transactions on Dependable and Secure Computing (2019).

Digital Library

[10]

Sen Chen, Minhui Xue, Lingling Fan, Shuang Hao, Lihua Xu, Haojin Zhu, and Bo Li. 2018. Automated poisoning attacks and defenses in malware detection systems: An adversarial machine learning approach. computers & security 73 (2018), 326--344.

[11]

Christian Degott, Nataniel P Borges Jr, and Andreas Zeller. 2019. Learning user interface element interactions. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, 296--306.

Digital Library

[12]

Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In Proceedings of the 30th Annual Symposium on User Interface Software and Technology (UIST '17).

Digital Library

[13]

Android Developers. 2012. Ui/application exerciser monkey.

[14]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672--2680.

[15]

Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. 2014. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 1725--1732.

Digital Library

[16]

Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013).

[17]

B Kiran, Dilip Thomas, and Ranjith Parakkal. 2018. An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos. Journal of Imaging 4, 2 (2018), 36.

[18]

Tomi Lämsä. 2017. Comparison of GUI testing tools for Android applications. (2017).

[19]

Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. 2015. Autoencoding beyond pixels using a learned similarity metric. arXiv preprint arXiv:1512.09300 (2015).

[20]

Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. 2015. Adversarial autoencoders. arXiv preprint arXiv:1511.05644 (2015).

[21]

Jefferson Ryan Medel and Andreas Savakis. 2016. Anomaly detection in video using predictive convolutional long short-term memory networks. arXiv preprint arXiv:1612.00390 (2016).

[22]

Nariman Mirzaei, Joshua Garcia, Hamid Bagheri, Alireza Sadeghi, and Sam Malek. 2016. Reducing combinatorics in GUI testing of android applications. In Software Engineering (ICSE), 2016 IEEE/ACM 38th International Conference on. IEEE, 559--570.

Digital Library

[23]

Kevin Moran, Carlos BernalCrdenas, Michael Curcio, Richard Bonett, and Denys Poshyvanyk. 2018. Machine learning-based prototyping of graphical user interfaces for mobile apps. arXiv preprint arXiv:1802.02312 (2018).

[24]

Kevin Moran, Boyang Li, Carlos Bernal-Cárdenas, Dan Jelf, and Denys Poshyvanyk. 2018. Automated reporting of GUI design violations for mobile apps. In Proceedings of the 40th International Conference on Software Engineering. ACM, 165--175.

Digital Library

[25]

Martin J Osborne and Ariel Rubinstein. 1994. A course in game theory. MIT press.

[26]

Alec Radford, Luke Metz, and Soumith Chintala. 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015).

[27]

Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Ursula Schmidt-Erfurth, and Georg Langs. 2017. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In International Conference on Information Processing in Medical Imaging. Springer, 146--157.

[28]

Karen Simonyan and Andrew Zisserman. 2014. Two-stream convolutional networks for action recognition in videos. In Advances in neural information processing systems. 568--576.

[29]

Khurram Soomro, Amir Roshan Zamir, and M Shah. 2012. A dataset of 101 human action classes from videos in the wild. Center for Research in Computer Vision (2012).

[30]

Ting Su, Guozhu Meng, Yuting Chen, Ke Wu, Weiming Yang, Yao Yao, Geguang Pu, Yang Liu, and Zhendong Su. 2017. Guided, stochastic model-based GUI testing of Android apps. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. ACM, 245--256.

Digital Library

[31]

Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. 2018. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1199--1208.

[32]

Chongbin Tang, Sen Chen, Lingling Fan, Lihua Xu, Yang Liu, Zhushou Tang, and Liang Dou. 2019. A large-scale empirical study on industrial fake apps. In Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice. IEEE Press, 183--192.

Digital Library

[33]

Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. 2015. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision. 4489--4497.

Digital Library

[34]

Amin Ullah, Jamil Ahmad, Khan Muhammad, Muhammad Sajjad, and Sung Wook Baik. 2017. Action recognition in video sequences using deep bi-directional LSTM with CNN features. IEEE Access 6 (2017), 1155--1166.

[35]

Xu Wang, Chunyang Chen, and Zhenchang Xing. 2019. Domain-specific machine translation with recurrent neural network for software localization. Empirical Software Engineering 24, 6 (2019), 3514--3545.

[36]

Thomas D White, Gordon Fraser, and Guy J Brown. 2019. Improving random GUI testing with image-based widget detection. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, 307--317.

Digital Library

[37]

Matthew D Zeiler, Dilip Krishnan, Graham W Taylor, and Robert Fergus. 2010. Deconvolutional networks. In Cvpr, Vol. 10. 7.

[38]

Samer Zein, Norsaremah Salleh, and John Grundy. 2016. A systematic mapping study of mobile application testing techniques. Journal of Systems and Software 117 (2016), 334--356.

Digital Library

[39]

Dehai Zhao, Zhenchang Xing, Chunyang Chen, Xin Xia, and Guoqiang Li. 2019. ActionNet: vision-based workflow action recognition from programming screen-casts. In Proceedings of the 41st International Conference on Software Engineering. IEEE Press, 350--361.

Digital Library

Cited By

Yang BLi S(2024)UISGPT: Automated Mobile UI Design Smell Detection with Large Language ModelsElectronics10.3390/electronics1316312713:16(3127)Online publication date: 7-Aug-2024
https://doi.org/10.3390/electronics13163127
Yang BLi S(2024)UIGuider: Detecting Implicit Design Guidelines Using a Domain Knowledge Graph ApproachElectronics10.3390/electronics1307121013:7(1210)Online publication date: 26-Mar-2024
https://doi.org/10.3390/electronics13071210
Yu XLi CPan MLi XFilkov VRay BZhou M(2024)DroidCoder: Enhanced Android Code Completion with Context-Enriched Retrieval-Augmented GenerationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695063(681-693)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695063
Show More Cited By

Index Terms

Seenomaly: vision-based linting of GUI animation effects against design-don't guidelines
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software organization and properties
    1. Extra-functional properties
      1. Software usability

Recommendations

Finding a New Voice: Transitioning Designers from GUI to VUI Design
CUI '21: Proceedings of the 3rd Conference on Conversational User Interfaces

As Voice User Interfaces (VUIs) become widely popular, designers must handle new usability challenges. However, compared to other established domains such as Graphical User Interfaces (GUIs), VUI designers have fewer resources (training support, ...
What’s The Talk on VUI Guidelines? A Meta-Analysis of Guidelines for Voice User Interface Design
CUI '23: Proceedings of the 5th International Conference on Conversational User Interfaces

Over the past decade, voice user interface (VUI) design has been steadily growing, along with a growing VUI presence in consumer markets. However, there is currently a lack of widely-established guidelines for VUI design. While many sets of VUI ...
Design guidelines for hands-free speech interaction
MobileHCI '18: Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct

As research on speech interfaces continues to grow in the field of HCI, there is a need to develop design guidelines that help solve usability and learnability issues that exist in hands-free speech interfaces. While several sets of established ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

June 2020

1640 pages

ISBN:9781450371216

DOI:10.1145/3377811

General Chairs:
Gregg Rothermel
North Carolina State University
,
Doo-Hwan Bae
KAIST, South Korea

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

KIISE: Korean Institute of Information Scientists and Engineers
IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '20

Sponsor:

SIGSOFT

ICSE '20: 42nd International Conference on Software Engineering

June 27 - July 19, 2020

Seoul, South Korea

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

47
Total Citations
View Citations
322
Total Downloads

Downloads (Last 12 months)81
Downloads (Last 6 weeks)4

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yang BLi S(2024)UISGPT: Automated Mobile UI Design Smell Detection with Large Language ModelsElectronics10.3390/electronics1316312713:16(3127)Online publication date: 7-Aug-2024
https://doi.org/10.3390/electronics13163127
Yang BLi S(2024)UIGuider: Detecting Implicit Design Guidelines Using a Domain Knowledge Graph ApproachElectronics10.3390/electronics1307121013:7(1210)Online publication date: 26-Mar-2024
https://doi.org/10.3390/electronics13071210
Yu XLi CPan MLi XFilkov VRay BZhou M(2024)DroidCoder: Enhanced Android Code Completion with Context-Enriched Retrieval-Augmented GenerationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695063(681-693)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695063
Li SGao CZhang JZhang YLiu YGu JPeng YLyu M(2024)Less Cybersickness, Please: Demystifying and Detecting Stereoscopic Visual Inconsistencies in Virtual Reality AppsProceedings of the ACM on Software Engineering10.1145/36608031:FSE(2167-2189)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660803
Baral KJohnson JMahmud JSalma SFazzini MRubin JOffutt JMoran KSpinellis DConstantinou EBacchelli A(2024)Automating GUI-based Test Oracles for Mobile AppsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644930(309-321)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644930
Nie LSaid KHu M(2024)SoK: An Exhaustive Taxonomy of Display Issues for Mobile ApplicationsProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645193(537-548)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645193
Ahmad MGeewax JMacvean AKarger DMa KRoychoudhury APaiva AAbreu RStorey MAniche MNagappan N(2024)API Governance at ScaleProceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice10.1145/3639477.3639713(430-440)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639477.3639713
Lu YKnearem TDutta SBlass JKliman-Silver CBentley F(2024)AI Is Not Enough: A Hybrid Technical Approach to AI Adoption in UI Linting With HeuristicsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3637135(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3637135
Gao SWang YLiu H(2024)UiAnalyzerExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.122408239:COnline publication date: 17-Apr-2024
https://dl.acm.org/doi/10.1016/j.eswa.2023.122408
Yang BLi S(2024)GUI2DSVec: Detecting Visual Design Smells Based on Semantic Embedding of GUI Images and ComponentsMobile and Ubiquitous Systems: Computing, Networking and Services10.1007/978-3-031-63992-0_29(445-457)Online publication date: 19-Jul-2024
https://doi.org/10.1007/978-3-031-63992-0_29
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents