Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article
Open access

From Lost to Found: Discover Missing UI Design Semantics through Recovering Missing Tags

Published: 15 October 2020 Publication History

Abstract

Design sharing sites provide UI designers with a platform to share their works and also an opportunity to get inspiration from others' designs. To facilitate management and search of millions of UI design images, many design sharing sites adopt collaborative tagging systems by distributing the work of categorization to the community. However, designers often do not know how to properly tag one design image with compact textual description, resulting in unclear, incomplete, and inconsistent tags for uploaded examples which impede retrieval, according to our empirical study and interview with four professional designers. Based on a deep neural network, we introduce a novel approach for encoding both the visual and textual information to recover the missing tags for existing UI examples so that they can be more easily found by text queries. We achieve 82.72% accuracy in the tag prediction. Through a simulation test of 5 queries, our system on average returns hundreds more results than the default Dribbble search, leading to better relatedness, diversity and satisfaction.

References

[1]
1998. Design Jobs and Portfolios | Coroflot. https://www.coroflot.com/.
[2]
2005. Automatic image annotation. https://en.wikipedia.org/wiki/Automatic_image_annotation.
[3]
2005. Behance :: Best of Behance. https://www.behance.net/.
[4]
2010. Dribbble - Discover the world's top designers & creatives. https://dribbble.com/.
[5]
2014. Design - Material Design. https://material.io/design/.
[6]
2017. Essential Design Principles. https://developer.apple.com/videos/play/wwdc2017/802/.
[7]
2017. Flickr: Find your inspiration. https://www.flickr.com/.
[8]
2018. Dribbble 2018 Year in Review. https://dribbble.com/year/2018.
[9]
2020. Delicious. https://del.icio.us/.
[10]
2020. GloVe: Global Vectors for Word Representation. https://nlp.stanford.edu/projects/glove.
[11]
2020. Quora: A place to share knowledge and better understand the world. https://www.quora.com.
[12]
Rakesh Agrawal, Ramakrishnan Srikant, et al. 1994. Fast algorithms for mining association rules. In Proc. 20th int. conf. very large data bases, VLDB, Vol. 1215. 487--499.
[13]
Khalid Alharbi and Tom Yeh. 2015. Collect, decompile, extract, stats, and diff: Mining design pattern changes in Android apps. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM, 515--524.
[14]
Mathieu Bastian, Sebastien Heymann, Mathieu Jacomy, et al. 2009. Gephi: an open source software for exploring and manipulating networks. ICWSM 8 (2009), 361--362.
[15]
Farnaz Behrang, Steven P Reiss, and Alessandro Orso. 2018. GUIfetch: supporting app design and development through GUI search. In Proceedings of the 5th International Conference on Mobile Software Engineering and Systems. ACM, 236--246.
[16]
Kerstin Bischoff, Claudiu S Firan,Wolfgang Nejdl, and Raluca Paiu. 2008. Can all tags be used for search?. In Proceedings of the 17th ACM conference on Information and knowledge management. 193--202.
[17]
Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. 2008. Fast unfolding of communities in large networks. Journal of Statistical Mechanics: Theory and Experiment 2008, 10 (2008), P10008.
[18]
Zoya Bylinskii, Sami Alsheikh, Spandan Madan, Adria Recasens, Kimberli Zhong, Hanspeter Pfister, Fredo Durand, and Aude Oliva. 2017. Understanding infographics through textual and visual tag prediction. arXiv preprint arXiv:1709.09215 (2017).
[19]
Chunyang Chen, Xi Chen, Jiamou Sun, Zhenchang Xing, and Guoqiang Li. 2018. Data-driven proactive policy assurance of post quality in community q&a sites. Proceedings of the ACM on human-computer interaction 2, CSCW (2018), 1--22.
[20]
Chunyang Chen, Sidong Feng, Zhenchang Xing, Linda Liu, Shengdong Zhao, and Jinshui Wang. 2019. Gallery DC: Design Search and Knowledge Discovery through Auto-created GUI Component Gallery. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1--22.
[21]
Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018. From UI design image to GUI skeleton: a neural machine translator to bootstrap mobile GUI implementation. In Proceedings of the 40th International Conference on Software Engineering. ACM, 665--676.
[22]
Chunyang Chen, Zhenchang Xing, and Yang Liu. 2017. By the community & for the community: a deep learning approach to assist collaborative editing in q&a sites. Proceedings of the ACM on Human-Computer Interaction 1, CSCW (2017), 32.
[23]
Chunyang Chen, Zhenchang Xing, and Ximing Wang. 2017. Unsupervised software-specific morphological forms inference from informal discussions. In Proceedings of the 39th International Conference on Software Engineering. IEEE Press, 450--461.
[24]
Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xin Xia, Liming Zhu, John Grundy, and Jinshui Wang. 2020. Wireframe-based UI design search through image autoencoder. ACM Transactions on Software Engineering and Methodology (TOSEM) 29, 3 (2020), 1--31.
[25]
Jieshan Chen, Zhenchang Xing, Sidong Feng, Chunyang Chen, Xiwei Xu, Lingming Zhu, and Guoqiang Li. 2020. Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination?. In ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering.
[26]
Minmin Chen, Alice Zheng, and Kilian Weinberger. 2013. Fast image tagging. In International conference on machine learning. 1274--1282.
[27]
Sen Chen, Lingling Fan, Chunyang Chen, Ting Su,Wenhe Li, Yang Liu, and Lihua Xu. 2019. STORYDROID: Automated Generation of Storyboard for Android Apps. In Proceedings of the 41st International Conference on Software Engineering. ACM.
[28]
Joohee Choi and Yla Tausczik. 2018. Will Too Many Editors Spoil The Tag?: Conflicts and Alignment in Q&A Categorization. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 38.
[29]
Ian G Clifton. 2015. Android User Interface Design: Implementing Material Design for Developers. Addison-Wesley Professional.
[30]
Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine learning 20, 3 (1995), 273--297.
[31]
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. ACM, 845--854.
[32]
Wonyong Eom, Sihyoung Lee, Wesley De Neve, and Yong Man Ro. 2011. Improving image tag recommendation using favorite image context. In 2011 18th IEEE International Conference on Image Processing. IEEE, 2445--2448.
[33]
Michael P Fay and Michael A Proschan. 2010. Wilcoxon-Mann-Whitney or t-test? On assumptions for hypothesis tests and multiple interpretations of decision rules. Statistics surveys 4 (2010), 1.
[34]
Michael Fischer, Richard R Yang, and Monica S Lam. 2018. ImagineNet: Style Transfer from Fine Art to Graphical User Interfaces. (2018).
[35]
Wai-Tat Fu. 2008. The microstructures of social tagging: a rational model. In Proceedings of the 2008 ACM conference on Computer supported cooperative work. ACM, 229--238.
[36]
Wai-Tat Fu, Thomas Kannampallil, Ruogu Kang, and Jibo He. 2010. Semantic imitation in social tagging. ACM Transactions on Computer-Human Interaction (TOCHI) 17, 3 (2010), 12.
[37]
Wilbert O Galitz. 2007. The essential guide to user interface design: an introduction to GUI design principles and techniques. John Wiley & Sons.
[38]
Scott A Golder and Bernardo A Huberman. 2006. Usage patterns of collaborative tagging systems. Journal of information science 32, 2 (2006), 198--208.
[39]
Harry Halpin, Valentin Robu, and Hana Shepherd. 2007. The complex dynamics of collaborative tagging. In Proceedings of the 16th international conference on World Wide Web. ACM, 211--220.
[40]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep Residual Learning for Image Recognition. CoRR abs/1512.03385 (2015). arXiv:1512.03385 http://arxiv.org/abs/1512.03385
[41]
Ali Jahanian, Phillip Isola, and Donglai Wei. 2017. Mining Visual Evolution in 21 Years of Web Design. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 2676--2682.
[42]
Ali Jahanian, Shaiyan Keshvari, SVN Vishwanathan, and Jan P Allebach. 2017. Colors--Messengers of Concepts: Visual Design Mining for Learning Color Semantics. ACM Transactions on Computer-Human Interaction(TOCHI) 24, 1 (2017).
[43]
Bernard J Jansen. 1998. The graphical user interface. ACM SIGCHI Bulletin 30, 2 (1998), 22--26.
[44]
Richard Socher Li-jia Li Kai Li Jia Deng, Wei Dong and Li Fei-fei. 2009. Imagenet: A large-scale hierarchical image database. In In CVPR.
[45]
Mahdi M Kalayeh, Haroon Idrees, and Mubarak Shah. 2014. NMF-KNN: Image annotation using weighted multi-view non-negative matrix factorization. In Proceedings of the IEEE conference on computer vision and pattern recognition. 184--191.
[46]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[47]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097--1105.
[48]
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278--2324.
[49]
Vladimir I Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, Vol. 10. 707--710.
[50]
Guo Li, Haiyi Zhu, Tun Lu, Xianghua Ding, and Ning Gu. 2015. Is it good to be like wikipedia?: Exploring the trade-offs of introducing collaborative editing model to q&a sites. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing. ACM, 1080--1091.
[51]
Lei Li, Daqing He, Danchen Zhang, Yu Chi, and Chengzhi Zhang. 2017. Types of Tags for Annotating Academic Blogs. (2017).
[52]
Huizhi Liang, Yue Xu, Yuefeng Li, and Richi Nayak. 2008. Collaborative filtering recommender systems using tag information. In 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, Vol. 3. IEEE, 59--62.
[53]
Bin Liu and Grigorios Tsoumakas. 2019. Synthetic Oversampling of Multi-Label Data based on Local Label Distribution. arXiv preprint arXiv:1905.00609 (2019).
[54]
Thomas F Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar. 2018. Learning Design Semantics for Mobile Apps. In The 31st Annual ACM Symposium on User Interface Software and Technology. ACM, 569--579.
[55]
Zhe Liu, Chunyang Chen, Junjie Wang, Yuekai Huang, Jun Hu, and Qing Wang. 2020. Owl Eyes: Spotting UI Display Issues via Visual Understanding. In Proceedings of the 35th International Conference on Automated Software Engineering.
[56]
Yuexin Ma, Xinge Zhu, Yujing Sun, and Bingzheng Yan. 2017. Image tagging by joint deep visual-semantic propagation. In Pacific Rim Conference on Multimedia. Springer, 25--35.
[57]
George Macgregor and Emma McCulloch. 2006. Collaborative tagging as a knowledge organisation and resource discovery tool. Library review 55, 5 (2006), 291--300.
[58]
Mark Melenhorst and Mark van Setten. 2007. Usefulness of tags in providing access to large information systems. In 2007 IEEE International Professional Communication Conference. IEEE, 1--9.
[59]
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems. 3111--3119.
[60]
Gilad Mishne. 2006. Autotag: a collaborative approach to automated tag assignment for weblog posts. In Proceedings of the 15th international conference on World Wide Web. 953--954.
[61]
Venkatesh N Murthy, Subhransu Maji, and R Manmatha. 2015. Automatic image annotation using deep learning representations. In Proceedings of the 5th ACM on International Conference on Multimedia Retrieval. ACM, 603--606.
[62]
Marc Najork and Janet L Wiener. 2001. Breadth-first crawling yields high-quality pages. In Proceedings of the 10th international conference on World Wide Web. ACM, 114--118.
[63]
Theresa Neil. 2014. Mobile design pattern gallery: UI patterns for smartphone apps. " O'Reilly Media, Inc.".
[64]
J Ross Quinlan. 1983. Learning efficient classification procedures and their application to chess end games. In Machine learning. Springer, 463--482.
[65]
Yogesh Singh Rawat and Mohan S Kankanhalli. 2016. ConTagNet: Exploiting user context for image tag recommendation. In Proceedings of the 24th ACM international conference on Multimedia. 1102--1106.
[66]
Steven P Reiss, Yun Miao, and Qi Xin. 2018. Seeking the user interface. Automated Software Engineering 25, 1 (2018), 157--193.
[67]
Shilad Sen, Shyong K Lam, Al Mamunur Rashid, Dan Cosley, Dan Frankowski, Jeremy Osterhouse, F Maxwell Harper, and John Riedl. 2006. Tagging, communities, vocabulary, evolution. In Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work. 181--190.
[68]
Börkur Sigurbjörnsson and Roelof Van Zwol. 2008. Flickr tag recommendation based on collective knowledge. In Proceedings of the 17th international conference on World Wide Web. ACM, 327--336.
[69]
Craig Silverstein, Hannes Marais, Monika Henzinger, and Michael Moricz. 1999. Analysis of a very large web search engine query log. In ACm SIGIR Forum, Vol. 33. ACM, 6--12.
[70]
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2013. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 (2013).
[71]
Fabian M Suchanek, Milan Vojnovic, and Dinan Gunawardena. 2008. Social tags: meaning and suggestions. In Proceedings of the 17th ACM conference on Information and knowledge management. ACM, 223--232.
[72]
Amanda Swearngin, Mira Dontcheva, Wilmot Li, Joel Brandt, Morgan Dixon, and Andrew J Ko. 2018. Rewire: Interface Design Assistance from Examples. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 504.
[73]
Karen HL Tso-Sutter, Leandro Balby Marinho, and Lars Schmidt-Thieme. 2008. Tag-aware recommender systems by fusion of collaborative filtering algorithms. In Proceedings of the 2008 ACM symposium on Applied computing. 1995--1999.
[74]
Xiang-Yang Wang, Jun-Feng Wu, and Hong-Ying Yang. 2010. Robust image retrieval based on color histogram of local feature regions. Multimedia Tools and Applications 49, 2 (2010), 323--345.
[75]
Robert Wetzker, Carsten Zimmermann, Christian Bauckhage, and Sahin Albayrak. 2010. I tag, you tag: translating tags for advanced user models. In Proceedings of the third ACM international conference on Web search and data mining. ACM, 71--80.
[76]
Zhichen Xu, Yun Fu, Jianchang Mao, and Difu Su. 2006. Towards the semantic web: Collaborative tag suggestions. In Collaborative web tagging workshop at WWW2006, Edinburgh, Scotland.
[77]
Dehai Zhao, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, Guoqiang Li, and JinshuiWang. 2020. Seenomaly: Vision-Based Linting of GUI Animation Effects Against Design-Don?t Guidelines. In 42nd International Conference on Software Engineering (ICSE'20). ACM, New York, NY.
[78]
Shuyu Zheng, Ziniu Hu, and Yun Ma. 2019. FaceOff: Assisting the Manifestation Design of Web Graphical User Interface. (2019).

Cited By

View all
  • (2024)Mobile-UI-Repair: a deep learning based UI smell detection technique for mobile user interfacePeerJ Computer Science10.7717/peerj-cs.202810(e2028)Online publication date: 16-May-2024
  • (2024)A Practice Theory Perspective on Dribbble and the Evolving Design IndustrySocial Media + Society10.1177/2056305124122860110:1Online publication date: 19-Feb-2024
  • (2024)On Improving Management of Duplicate Video-Based Bug ReportsProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings10.1145/3639478.3639786(201-203)Online publication date: 14-Apr-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction
Proceedings of the ACM on Human-Computer Interaction  Volume 4, Issue CSCW2
CSCW
October 2020
2310 pages
EISSN:2573-0142
DOI:10.1145/3430143
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2020
Published in PACMHCI Volume 4, Issue CSCW2

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. deep learning
  2. designers
  3. graphical user interface
  4. semantics
  5. tags

Qualifiers

  • Research-article

Funding Sources

  • ANU-Data61 Collaborative Research Project

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)295
  • Downloads (Last 6 weeks)51
Reflects downloads up to 09 Nov 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Mobile-UI-Repair: a deep learning based UI smell detection technique for mobile user interfacePeerJ Computer Science10.7717/peerj-cs.202810(e2028)Online publication date: 16-May-2024
  • (2024)A Practice Theory Perspective on Dribbble and the Evolving Design IndustrySocial Media + Society10.1177/2056305124122860110:1Online publication date: 19-Feb-2024
  • (2024)On Improving Management of Duplicate Video-Based Bug ReportsProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings10.1145/3639478.3639786(201-203)Online publication date: 14-Apr-2024
  • (2024)Deep Heterogeneous Contrastive Hyper-Graph Learning for In-the-Wild Context-Aware Human Activity RecognitionProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314447:4(1-23)Online publication date: 12-Jan-2024
  • (2024)Spatial-Temporal Masked Autoencoder for Multi-Device Wearable Human Activity RecognitionProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314157:4(1-25)Online publication date: 12-Jan-2024
  • (2024)MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI ModelingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642350(1-14)Online publication date: 11-May-2024
  • (2024)Automatically Detecting Reflow Accessibility Issues in Responsive Web PagesProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639229(1-13)Online publication date: 20-May-2024
  • (2024)Semantic GUI Scene Learning and Video Alignment for Detecting Duplicate Video-based Bug ReportsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639163(1-13)Online publication date: 20-May-2024
  • (2024)Prompting Is All You Need: Automated Android Bug Replay with Large Language ModelsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3608137(1-13)Online publication date: 20-May-2024
  • (2024)A Systematic Review of Human Activity Recognition Based on Mobile Devices: Overview, Progress and TrendsIEEE Communications Surveys & Tutorials10.1109/COMST.2024.335759126:2(890-929)Online publication date: 23-Jan-2024
  • Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Get Access

Login options

Full Access

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media