research-article

UIED: a hybrid tool for GUI element detection

Authors:

Zhenchang Xing,

Chunyang ChenAuthors Info & Claims

ESEC/FSE 2020: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Pages 1655 - 1659

https://doi.org/10.1145/3368089.3417940

Published: 08 November 2020 Publication History

Abstract

Graphical User Interface (GUI) elements detection is critical for many GUI automation and GUI testing tasks. Acquiring the accurate positions and classes of GUI elements is also the very first step to conduct GUI reverse engineering or perform GUI testing. In this paper, we implement a User Iterface Element Detection (UIED), a toolkit designed to provide user with a simple and easy-to-use platform to achieve accurate GUI element detection. UIED integrates multiple detection methods including old-fashioned computer vision (CV) approaches and deep learning models to handle diverse and complicated GUI images. Besides, it equips with a novel customized GUI element detection methods to produce state-of-the-art detection results. Our tool enables the user to change and edit the detection result in an interactive dashboard. Finally, it exports the detected UI elements in the GUI image to design files that can be further edited in popular UI design tools such as Sketch and Photoshop. UIED is evaluated to be capable of accurate detection and useful for downstream works.

Tool URL: <a>http://uied.online</a>

Github Link: <a>https://github.com/MulongXie/UIED</a>

Supplementary Material

Auxiliary Teaser Video (fse20demo-p40-p-teaser.mp4)

Main presentation

Download
17.56 MB

Auxiliary Presentation Video (fse20demo-p40-p-video.mp4)

Main presentation

Download
34.29 MB

References

[1]

Lingfeng Bao, Jing Li, Zhenchang Xing, Xinyu Wang, and Bo Zhou. 2015. scvRipper: video scraping tool for modeling developers' behavior using interaction data. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 2. IEEE, 673-676.

[2]

Carlos Bernal-Cardenas, Nathan Cooper, Kevin Moran, Oscar Chaparro, Andrian Marcus, and Denys Poshyvanyk. 2020. Translating Video Recordings of Mobile App Usages into Replayable Scenarios. In 42nd International Conference on Software Engineering (ICSE '20). ACM, New York, NY.

[3]

Karl Bridge and Michael Satran. 2018. Windows Accessibility API overview. Retrieved March 2, 2020 from https://docs.microsoft.com/en-us/windows/win32/ winauto/windows-automation-api-portal

[4]

J. Canny. 1986. A Computational Approach to Edge Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-8, 6 (Nov 1986 ), 679-698. https://doi.org/10.1109/TPAMI. 1986.4767851

Digital Library

[5]

Chunyang Chen, Sidong Feng, Zhenchang Xing, Linda Liu, Shengdong Zhao, and Jinshui Wang. 2019. Gallery DC : Design Search and Knowledge Discovery through Auto-created GUI Component Gallery. Proceedings of the ACM on Human-Computer Interaction 3, CSCW ( 2019 ), 1-22.

[6]

Jieshan Chen, Mulong Xie, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, and Guoqiang Li. 2020. Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination? arXiv: 2008. 05132 [cs.CV]

[7]

Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jefrey Nichols, and Ranjitha Kumar. 2017. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. 845-854.

Digital Library

[8]

Google Developers. 2020. Protocol Bufers  |  Google Developers. https://developers.google.com/protocol-bufers

[9]

Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. 2019. Centernet: Keypoint triplets for object detection. In Proceedings of the IEEE International Conference on Computer Vision. 6569-6578.

[10]

Google. 2019. UI Automator. Retrieved March 2, 2020 from https://developer. android.com/training/testing/ui-automator

[11]

Google. 2020. Build more accessible apps. Retrieved March 2, 2020 from https: //developer.android.com/guide/topics/ui/accessibility

[12]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770-778.

[13]

Feng Lin, Chen Song, Xiaowei Xu, Lora Cavuoto, and Wenyao Xu. 2016. Sensing from the bottom: Smart insole enabled patient handling activity recognition through manifold learning. In 2016 IEEE First International Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE). IEEE, 254-263.

[14]

Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. 2017. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2117-2125.

[15]

Microsoft. 2016. Introducing Spy++. Retrieved March 2, 2020 from https://docs.microsoft.com/en-us/visualstudio/debugger/introducing-spyincrement?view= vs-2019

[16]

Kevin Moran, Boyang Li, Carlos Bernal-Cárdenas, Dan Jelf, and Denys Poshyvanyk. 2018. Automated reporting of GUI design violations for mobile apps. In Proceedings of the 40th International Conference on Software Engineering. 165-175.

Digital Library

[17]

Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse engineering mobile application user interfaces with remaui (t). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 248-259.

Digital Library

[18]

Suporn Pongnumkul, Mira Dontcheva, Wilmot Li, Jue Wang, Lubomir Bourdev, Shai Avidan, and Michael F Cohen. 2011. Pause-and-play: automatically linking screencast video tutorials with applications. In Proceedings of the 24th annual ACM symposium on User interface software and technology. 135-144.

Digital Library

[19]

Dilip K. Prasad, Maylor K.H. Leung, Chai Quek, and Siu-Yeung Cho. 2012. A novel framework for making dominant point detection methods non-parametric. Image and Vision Computing 30, 11 ( 2012 ), 843-859. https://doi.org/10.1016/j. imavis. 2012. 06.010

Digital Library

[20]

Ju Qian, Zhengyu Shang, Shuoyan Yan, Yan Wang, and Lin Chen. 2020. RoScript: A Visual Script Driven Truly Non-Intrusive Robotic Testing System for Touch Screen Applications. In 42nd International Conference on Software Engineering (ICSE '20). ACM, New York, NY.

[21]

Joseph Redmon and Ali Farhadi. 2018. Yolov3: An incremental improvement. arXiv preprint arXiv: 1804. 02767 ( 2018 ).

[22]

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems. 91-99.

[23]

H. Samet and M. Tamminen. 1988. Eficient component labeling of images of arbitrary dimension represented by linear bintrees. IEEE Transactions on Pattern Analysis and Machine Intelligence 10, 4 ( 1988 ), 579-586. https://doi.org/10.1109/ 34.3918

Digital Library

[24]

Ray Smith. 2007. An overview of the Tesseract OCR engine. In Ninth International Conference on Document Analysis and Recognition (ICDAR 2007 ), Vol. 2. IEEE, 629-633.

[25]

Satoshi Suzuki and KeiichiA be. 1985. Topological structural analysis of digitized binary images by border following. Computer Vision, Graphics, and Image Processing 30, 1 ( 1985 ), 32-46. https://doi.org/10.1016/ 0734-189X ( 85 ) 90016-7

[26]

OpenCV team. 2020. https://opencv.org/

[27]

Pytorch Team. 2020. https://pytorch.org/

[28]

Shane Torbert. 2016. Applied computer science. Springer.

[29]

Thomas D White, Gordon Fraser, and Guy J Brown. 2019. Improving random GUI testing with image-based widget detection. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. 307-317.

Digital Library

[30]

Tom Yeh, Tsung-Hsiang Chang, and Robert C Miller. 2009. Sikuli: using GUI screenshots for search and automation. In Proceedings of the 22nd annual ACM symposium on User interface software and technology. 183-192.

Digital Library

[31]

Chen Yongxin, Zhang Tonghui, and Chen Jie. 2019. UI2code: How to Fine-tune Background and Foreground Analysis. Retrieved Feb 23, 2020 from https://laptrinhx.com/ui2code-how-to-fine-tune-background-andforeground-analysis-2293652041/

[32]

Dehai Zhao, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, Guoqiang Li, and Jinshui Wang. 2020. Seenomaly: Vision-Based Linting of GUI Animation Efects Against Design-Don't Guidelines. In 42nd International Conference on Software Engineering (ICSE '20). ACM, New York, NY, 12 pages. https://doi.org/ 10.1145/3377811.3380411

Digital Library

[33]

Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang. 2017. EAST: an eficient and accurate scene text detector. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 5551-5560.

Cited By

Kumar SNitin Yadav M(2024)An Effective GDP-LSTM and SDQL-Based Finite State Testing of GUIApplied Sciences10.3390/app1402054914:2(549)Online publication date: 8-Jan-2024
https://doi.org/10.3390/app14020549
Özdal MBulut Ş(2024)FÜTÜRİST ÖZELLİKLERİN 21. YÜZYIL GRAFİK TASARIMINA ETKİSİİnönü Üniversitesi Kültür ve Sanat Dergisi10.22252/ijca.143789710:1(63-75)Online publication date: 5-Aug-2024
https://doi.org/10.22252/ijca.1437897
Feng SLu HJiang JXiong THuang LLiang YLi XDeng YAleti AFilkov VRay BZhou M(2024)Enabling Cost-Effective UI Automation Testing with Retrieval-Based LLMs: A Case Study in WeChatProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695260(1973-1978)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695260
Show More Cited By

Index Terms

UIED: a hybrid tool for GUI element detection
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Graphical user interfaces
2. Software and its engineering
  1. Software creation and management
    1. Software development techniques

Recommendations

Improving random GUI testing with image-based widget detection
ISSTA 2019: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis

Graphical User Interfaces (GUIs) are amongst the most common user interfaces, enabling interactions with applications through mouse movements and key presses. Tools for automated testing of programs through their GUI exist, however they usually rely on ...
Object detection for graphical user interface: old fashioned or deep learning or a combination?
ESEC/FSE 2020: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Detecting Graphical User Interface (GUI) elements in GUI images is a domain-specific object detection task. It supports many software engineering tasks, such as GUI animation and testing, GUI search and code generation. Existing studies for GUI element ...
GUI Element Detection from Mobile UI Images Using YOLOv5
Mobile Web and Intelligent Information Systems
Abstract
In mobile application development, building a consistent user interface (UI) might be a costly and time-consuming process. This is especially the case if an organization has a separate team for each mobile platform such as iOS and Android. In this ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEC/FSE 2020: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

November 2020

1703 pages

ISBN:9781450370431

DOI:10.1145/3368089

General Chair:
Prem Devanbu
University of California at Davis, USA
,
Program Chairs:
Myra Cohen
Iowa State University, USA
,
Thomas Zimmermann
Microsoft Research, USA

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 November 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ESEC/FSE '20

Sponsor:

SIGSOFT

ESEC/FSE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

November 8 - 13, 2020

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

58
Total Citations
View Citations
1,127
Total Downloads

Downloads (Last 12 months)341
Downloads (Last 6 weeks)37

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kumar SNitin Yadav M(2024)An Effective GDP-LSTM and SDQL-Based Finite State Testing of GUIApplied Sciences10.3390/app1402054914:2(549)Online publication date: 8-Jan-2024
https://doi.org/10.3390/app14020549
Özdal MBulut Ş(2024)FÜTÜRİST ÖZELLİKLERİN 21. YÜZYIL GRAFİK TASARIMINA ETKİSİİnönü Üniversitesi Kültür ve Sanat Dergisi10.22252/ijca.143789710:1(63-75)Online publication date: 5-Aug-2024
https://doi.org/10.22252/ijca.1437897
Feng SLu HJiang JXiong THuang LLiang YLi XDeng YAleti AFilkov VRay BZhou M(2024)Enabling Cost-Effective UI Automation Testing with Retrieval-Based LLMs: A Case Study in WeChatProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695260(1973-1978)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695260
Cao SChen RPan MYang WLi XFilkov VRay BZhou M(2024)Beyond Manual Modeling: Automating GUI Model Generation Using Design DocumentsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695032(91-103)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695032
Liu SWang SSun K(2024)Having Difficulty Understanding Manuals? Automatically Converting User Manuals into Instructional VideosProceedings of the ACM on Human-Computer Interaction10.1145/36602458:EICS(1-19)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3660245
Emami PJiang YGuo ZLeiva L(2024)Impact of Design Decisions in Scanpath ModelingProceedings of the ACM on Human-Computer Interaction10.1145/36556028:ETRA(1-16)Online publication date: 28-May-2024
https://dl.acm.org/doi/10.1145/3655602
Zhang LWang SJia XZheng ZYan YGao LLi YXu M(2024)LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task AutomationProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676382(1-13)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676382
Jiang YZhou CGarg VOulasvirta A(2024)Graph4GUI: Graph Neural Networks for Representing Graphical User InterfacesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642822(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642822
Yang SVermeulen JFitzmaurice GMatejka J(2024)AQuA: Automated Question-Answering in Software Tutorial Videos with Visual AnchorsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642752(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642752
Feng SMa SWang HKong DChen C(2024)MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI ModelingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642350(1-14)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642350
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents