research-article

Gallery D.C: auto-created GUI component gallery for design search and knowledge discovery

Authors:

Zhenchang XingAuthors Info & Claims

ICSE '22: Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings

Pages 80 - 84

https://doi.org/10.1145/3510454.3516873

Published: 19 October 2022 Publication History

Abstract

GUI design is an integral part of software development. The process of designing a mobile application typically starts with the ideation and inspiration search from existing designs. However, existing information-retrieval based, and database-query based methods cannot efficiently gain inspirations in three requirements: design practicality, design granularity and design knowledge discovery. In this paper we propose a web application, called Gallery D.C. that aims to facilitate the process of user interface design through real world GUI component search. Gallery D.C. indexes GUI component designs using reverse engineering and deep learning based computer vision techniques on millions of real world applications. To perform an advanced design search and knowledge discovery, our approach extracts information about size, color, component type, and text information to help designers explore multi-faceted design space and distill higher-order of design knowledge. Gallery D.C. is well received via an informal evaluation with 7 professional designers.

Web Link: http://mui-collection.herokuapp.com/.

Demo Video Link: https://youtu.be/zVmsz_wY5OQ.

References

[1]

2021. Dribbble. https://dribbble.com/.

[2]

2021. UI Automator. https://developer.android.com/training/testing/uiautomator.

[3]

Carlos Bernal-Cárdenas, Kevin Moran, Michele Tufano, Zichang Liu, Linyong Nan, Zhehan Shi, and Denys Poshyvanyk. 2019. Guigle: a GUI search engine for Android apps. In Proceedings of the 41st International Conference on Software Engineering: Companion Proceedings. IEEE Press, 71--74.

Digital Library

[4]

Chunyang Chen, Sidong Feng, Zhengyang Liu, Zhenchang Xing, and Shengdong Zhao. 2020. From lost to found: Discover missing ui design semantics through recovering missing tags. Proceedings of the ACM on Human-Computer Interaction 4, CSCW2 (2020), 1--22.

Digital Library

[5]

Chunyang Chen, Sidong Feng, Zhenchang Xing, Linda Liu, Shengdong Zhao, and Jinshui Wang. 2019. Gallery DC: Design Search and Knowledge Discovery through Auto-created GUI Component Gallery. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 180.

[6]

Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018. From ui design image to gui skeleton: a neural machine translator to bootstrap mobile gui implementation. In Proceedings of the 40th International Conference on Software Engineering. ACM, 665--676.

Digital Library

[7]

Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xin Xia, Liming Zhu, John Grundy, and Jinshui Wang. 2020. Wireframe-based UI design search through image autoencoder. ACM Transactions on Software Engineering and Methodology (TOSEM) 29, 3 (2020), 1--31.

Digital Library

[8]

Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xiwei Xu, Liming Zhut, Guoqiang Li, and Jinshui Wang. 2020. Unblind your apps: Predicting natural-language labels for mobile gui components by deep learning. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE). IEEE, 322--334.

Digital Library

[9]

Jieshan Chen, Mulong Xie, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, and Guoqiang Li. 2020. Object detection for graphical user interface: old fashioned or deep learning or a combination?. In proceedings of the 28th ACM joint meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1202--1214.

Digital Library

[10]

Qiuyuan Chen, Chunyang Chen, Safwat Hassan, Zhengchang Xing, Xin Xia, and Ahmed E Hassan. 2021. How should I improve the UI of my app? a study of user reviews of popular apps in the Google Play. ACM Transactions on Software Engineering and Methodology (TOSEM) 30, 3 (2021), 1--38.

Digital Library

[11]

Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. ACM, 845--854.

Digital Library

[12]

Sidong Feng, Suyu Ma, Jinzhong Yu, Chunyang Chen, Tingting Zhou, and Yankun Zhen. 2021. Auto-icon: An automated code generation tool for icon designs assisting in ui development. In 26th International Conference on Intelligent User Interfaces. 59--69.

Digital Library

[13]

Bernard J Jansen. 1998. The graphical user interface. ACM SIGCHI Bulletin 30, 2 (1998), 22--26.

Digital Library

[14]

Zhe Liu, Chunyang Chen, Junjie Wang, Yuekai Huang, Jun Hu, and Qing Wang. 2020. Owl eyes: Spotting ui display issues via visual understanding. In 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 398--409.

Digital Library

[15]

Marc Najork and Janet L Wiener. 2001. Breadth-first crawling yields high-quality pages. In Proceedings of the 10th international conference on World Wide Web. ACM, 114--118.

Digital Library

[16]

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems. 91--99.

[17]

Mulong Xie, Sidong Feng, Zhenchang Xing, Jieshan Chen, and Chunyang Chen. 2020. UIED: a hybrid tool for GUI element detection. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1655--1659.

Digital Library

[18]

Tianming Zhao, Chunyang Chen, Yuanning Liu, and Xiaodong Zhu. 2021. GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 748--760.

Digital Library

Cited By

Wei JCourbis ALambolais TXu BBernard PDray GMaalej WFilkov VRay BZhou M(2024)Getting Inspiration for Feature Elicitation: App Store- vs. LLM-based ApproachProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695591(857-869)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695591
Feng SLu HJiang JXiong THuang LLiang YLi XDeng YAleti AFilkov VRay BZhou M(2024)Enabling Cost-Effective UI Automation Testing with Retrieval-Based LLMs: A Case Study in WeChatProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695260(1973-1978)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695260
Feng SMa SWang HKong DChen C(2024)MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI ModelingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642350(1-14)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642350
Show More Cited By

Index Terms

Gallery D.C: auto-created GUI component gallery for design search and knowledge discovery
1. Software and its engineering

Recommendations

Gallery D.C.: Design Search and Knowledge Discovery through Auto-created GUI Component Gallery

Online communities like Dribbble and GraphicBurger allow GUI designers to share their design artwork and learn from each other. These design sharing platforms are important sources for design inspiration, but our survey with GUI designers suggests ...
SIGGRAPH '12: ACM SIGGRAPH 2012 Art Gallery
SIGGRAPH '09: ACM SIGGRAPH 2009 Art Gallery

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '22: Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings

May 2022

394 pages

ISBN:9781450392235

DOI:10.1145/3510454

General Chair:
Matthew B Dwyer
University of Virginia

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 October 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '22

Sponsor:

SIGSOFT

ICSE '22: 44th International Conference on Software Engineering

May 21 - 29, 2022

Pennsylvania, Pittsburgh

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
95
Total Downloads

Downloads (Last 12 months)37
Downloads (Last 6 weeks)7

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wei JCourbis ALambolais TXu BBernard PDray GMaalej WFilkov VRay BZhou M(2024)Getting Inspiration for Feature Elicitation: App Store- vs. LLM-based ApproachProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695591(857-869)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695591
Feng SLu HJiang JXiong THuang LLiang YLi XDeng YAleti AFilkov VRay BZhou M(2024)Enabling Cost-Effective UI Automation Testing with Retrieval-Based LLMs: A Case Study in WeChatProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695260(1973-1978)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695260
Feng SMa SWang HKong DChen C(2024)MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI ModelingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642350(1-14)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642350
Zhong RShin DMeza RKlasnja PColusso LHsieh G(2024)AI-Assisted Causal Pathway Diagram for Human-Centered DesignProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642179(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642179
Feng SChen CXing Z(2023)Video2Action: Reducing Human Interactions in Action Annotation of App Tutorial VideosProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606778(1-15)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606778
Wei J(2023)Enhancing Requirements Elicitation through App Stores Mining: Health Monitoring App Case Study2023 IEEE 31st International Requirements Engineering Conference (RE)10.1109/RE57278.2023.00062(396-400)Online publication date: Sep-2023
https://doi.org/10.1109/RE57278.2023.00062
Wei JCourbis ALambolais TXu BBernard PDray G(2023)Boosting GUI Prototyping with Diffusion Models2023 IEEE 31st International Requirements Engineering Conference (RE)10.1109/RE57278.2023.00035(275-280)Online publication date: Sep-2023
https://doi.org/10.1109/RE57278.2023.00035
Feng SXie MChen CGrundy JPollock LPenta M(2023)Efficiency Matters: Speeding Up Automated Testing with GUI Rendering InferenceProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00084(906-918)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00084
Raptis GKatsini C(2023)Transforming Ideas into Code: A Novel Approach to Design-to-Development Workflow2023 International Conference on Computer and Applications (ICCA)10.1109/ICCA59364.2023.10401412(1-6)Online publication date: 28-Nov-2023
https://doi.org/10.1109/ICCA59364.2023.10401412

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents