research-article

Free access

Just Accepted

GUing: A Mobile GUI Search Engine using a Vision-Language Model

Authors:

Anne-Lise Courbis,

Thomas Lambolais,

Pierre Louis Bernard,

Walid MaalejAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology

Accepted on 01 October 2024

https://doi.org/10.1145/3702993

Online AM: 08 November 2024 Publication History

Abstract

Graphical User Interfaces (GUIs) are central to app development projects. App developers may use the GUIs of other apps as a means of requirements refinement and rapid prototyping or as a source of inspiration for designing and improving their own apps. Recent research has thus suggested retrieving relevant GUI designs that match a certain text query from screenshot datasets acquired through crowdsourced or automated exploration of GUIs. However, such text-to-GUI retrieval approaches only leverage the textual information of the GUI elements, neglecting visual information such as icons or background images. In addition, retrieved screenshots are not steered by app developers and lack app features that require particular input data.

To overcome these limitations, this paper proposes GUing, a GUI search engine based on a vision-language model called GUIClip, which we trained specifically for the problem of designing app GUIs. For this, we first collected from Google Play app introduction images which display the most representative screenshots and are often captioned (i.e. labelled) by app vendors. Then, we developed an automated pipeline to classify, crop, and extract the captions from these images. This resulted in a large dataset which we share with this paper: including 303k app screenshots, out of which 135k have captions. We used this dataset to train a novel vision-language model, which is, to the best of our knowledge, the first of its kind for GUI retrieval. We evaluated our approach on various datasets from related work and in a manual experiment. The results demonstrate that our model outperforms previous approaches in text-to-GUI retrieval achieving a Recall@10 of up to 0.69 and a HIT@10 of 0.91. We also explored the performance of GUIClip for other GUI tasks including GUI classification and sketch-to-GUI retrieval with encouraging results.

References

[1]

Afnan A. Al-Subaihin, Federica Sarro, Sue Black, Licia Capra, and Mark Harman. 2021. App Store Effects on Software Engineering Practices. IEEE Transactions on Software Engineering 47, 2 (2021), 300–319. https://doi.org/10.1109/TSE.2019.2891715

Digital Library

[2]

AppBrain. 2024. Google Play Ranking in the United States (Oct 2023). https://www.appbrain.com/stats/google-play-rankings. Accessed: 2023-10-01.

[3]

Sunil Arya, David M. Mount, Nathan S. Netanyahu, Ruth Silverman, and Angela Y. Wu. 1998. An optimal algorithm for approximate nearest neighbor searching in fixed dimensions. J. ACM 45, 6 (1998), 891–923. https://doi.org/10.1145/293347.293348

Digital Library

[4]

Farnaz Behrang, Steven P. Reiss, and Alessandro Orso. 2018. GUIfetch: Supporting app design and development through GUI search. Proceedings - International Conference on Software Engineering (2018), 236–246. https://doi.org/10.1145/3197231.3197244

Digital Library

[5]

Yoshua Bengio. 2009. Learning deep architectures for AI. Vol. 2. 1–27 pages. https://doi.org/10.1561/2200000006

Digital Library

[6]

Carlos Bernal-Cardenas, Kevin Moran, Michele Tufano, Zichang Liu, Linyong Nan, Zhehan Shi, and Denys Poshyvanyk. 2019. Guigle: A GUI search engine for android apps. Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion, ICSE-Companion 2019 (2019), 71–74. https://doi.org/10.1109/ICSE-Companion.2019.00041 arXiv:1901.00891

Digital Library

[7]

Nis Bornoe and Jan Stage. 2013. Supporting usability engineering in small software development organizations. In Proceedings of the The 36th Information Systems Research Conference in Scandinavia (IRIS 36). 1–12.

[8]

Sara Bunian, Kai Li, Chaima Jemmali, Casper Harteveld, Yun Fu, and Magy Seif Seif El-Nasr. 2021. VINS: Visual Search for Mobile User Interface Design. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764.3445762

Digital Library

[9]

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-End Object Detection with Transformers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 12346 LNCS (2020), 213–229. https://doi.org/10.1007/978-3-030-58452-8_13 arXiv:2005.12872

Digital Library

[10]

Chunyang Chen, Sidong Feng, Zhenchang Xing, Linda Liu, Shengdong Zhao, and Jinshui Wang. 2019. Gallery D.C.: Design search and knowledge discovery through auto-created GUI component gallery. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019). https://doi.org/10.1145/3359282

Digital Library

[11]

Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018. From UI design image to GUI skeleton: A neural machine translator to bootstrap mobile GUI implementation. In Proceedings - International Conference on Software Engineering, Vol. 6. 665–676. https://doi.org/10.1145/3180155.3180240

Digital Library

[12]

Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xin Xia, Liming Zhu, John Grundy, and Jinshui Wang. 2020. Wireframe-based UI Design Search through Image Autoencoder. ACM Transactions on Software Engineering and Methodology 29, 3 (2020). https://doi.org/10.1145/3391613 arXiv:2103.07085

Digital Library

[13]

Qiuyuan Chen, Chunyang Chen, Safwat Hassan, Zhengchang Xing, Xin Xia, and Ahmed E. Hassan. 2021. How Should I Improve the UI of My App?: A Study of User Reviews of Popular Apps in the Google Play. ACM Transactions on Software Engineering and Methodology 30, 3 (2021), 1–37. https://doi.org/10.1145/3447808

Digital Library

[14]

Sen Chen, Lingling Fan, Chunyang Chen, and Yang Liu. 2023. Automatically Distilling Storyboard With Rich Features for Android Apps. IEEE Transactions on Software Engineering 49, 2 (2023), 667–683. https://doi.org/10.1109/TSE.2022.3159548 arXiv:2203.06420

[15]

Sen Chen, Lingling Fan, Chunyang Chen, Ting Su, Wenhe Li, Yang Liu, and Lihua Xu. 2019. StoryDroid: Automated Generation of Storyboard for Android Apps. In Proceedings - International Conference on Software Engineering, Vol. 2019-May. IEEE, 596–607. https://doi.org/10.1109/ICSE.2019.00070 arXiv:1902.00476

Digital Library

[16]

Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A mobile app dataset for building data-driven design applications. UIST 2017 - Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (2017), 845–854. https://doi.org/10.1145/3126594.3126651

Digital Library

[17]

Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, Vol. 1. 4171–4186. arXiv:1810.04805 https://arxiv.org/abs/1810.04805

[18]

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image Is Worth 16X16 Words: Transformers for Image Recognition At Scale. In ICLR 2021 - 9th International Conference on Learning Representations. arXiv:2010.11929

[19]

Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. The Faiss library. (2024). arXiv:2401.08281 [cs.LG]

[20]

Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. 2022. A Survey of Vision-Language Pre-Trained Models. IJCAI International Joint Conference on Artificial Intelligence (2022), 5436–5443. https://doi.org/10.24963/ijcai.2022/762 arXiv:2202.10936

[21]

Explosion. 2024. Prodigy - An annotation tool for AI, Machine Learning and NLP. https://prodi.gy/. Accessed: 2024-3-10.

[22]

Sidong Feng, Chunyang Chen, and Zhenchang Xing. 2022. Gallery D.C.: Auto-created GUI Component Gallery for Design Search and Knowledge Discovery. In Proceedings - International Conference on Software Engineering, Vol. 1. Association for Computing Machinery, 80–84. https://doi.org/10.1109/ICSE-Companion55297.2022.9793764 arXiv:2204.06700

Digital Library

[23]

Alessio Ferrari and Paola Spoletini. 2023. Strategies, Benefits and Challenges of App Store-inspired Requirements Elicitation. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1290–1302. https://doi.org/10.1109/ICSE48619.2023.00114

Digital Library

[24]

Google. 2024. Add preview assets to showcase your app - Play Console Help. https://support.google.com/googleplay/android-developer/answer/9866151?hl=en&sjid=206438066775745925-EU. Accessed: 2024-3-10.

[25]

Marlo Haering, Muneera Bano, Didar Zowghi, Matthew Kearney, and Walid Maalej. 2021. Automating the Evaluation of Education Apps with App Store Data. IEEE Transactions on Learning Technologies 14, 1 (2021), 16–27. https://doi.org/10.1109/TLT.2021.3055121

[26]

Marlo Haering, Christoph Stanik, and Walid Maalej. 2021. Automatically Matching Bug Reports With Related App Reviews. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). 970–981. https://doi.org/10.1109/ICSE43902.2021.00092

Digital Library

[27]

Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang, Wentao Han, Minlie Huang, Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song, Jie Tang, Ji Rong Wen, Jinhui Yuan, Wayne Xin Zhao, and Jun Zhu. 2021. Pre-trained models: Past, present and future. AI Open 2, August 2021 (2021), 225–250. https://doi.org/10.1016/j.aiopen.2021.08.002

[28]

Safwat Hassan, Cor Paul Bezemer, and Ahmed E. Hassan. 2020. Studying Bad Updates of Top Free-to-Download Apps in the Google Play Store. IEEE Transactions on Software Engineering 46, 7 (2020), 773–793. https://doi.org/10.1109/TSE.2018.2869395

[29]

Forrest Huang, John F. Canny, and Jefrey Nichols. 2019. Swire: Sketch-based User Interface Retrieval. Conference on Human Factors in Computing Systems - Proceedings (2019), 1–10. https://doi.org/10.1145/3290605.3300334

Digital Library

[30]

Nic Hughart. 2023. 50 Best App Ideas For 2024. https://buildfire.com/best-app-ideas. Accessed: 2024-3-10.

[31]

Kristian Kolthoff, Christian Bartelt, and Simone Paolo Ponzetto. 2023. Data-driven prototyping via natural-language-based GUI retrieval. Automated Software Engineering 30, 1 (2023), 13. https://doi.org/10.1007/s10515-023-00377-x

Digital Library

[32]

Luis A. Leiva, Asutosh Hota, and Antti Oulasvirta. 2020. Enrico: A Dataset for Topic Modeling of Mobile UI Designs. Extended Abstracts - 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services: Expanding the Horizon of Mobile Interaction, MobileHCI 2020 (2020). https://doi.org/10.1145/3406324.3410710

Digital Library

[33]

Luis A. Leiva, Asutosh Hota, and Antti Oulasvirta. 2023. Describing UI Screenshots in Natural Language. ACM Transactions on Intelligent Systems and Technology 14, 1 (2023), 1–28. https://doi.org/10.1145/3564702

Digital Library

[34]

Chenxia Li, Weiwei Liu, Ruoyu Guo, Xiaoting Yin, Kaitao Jiang, Yongkun Du, Yuning Du, Lingfeng Zhu, Baohua Lai, Xiaoguang Hu, Dianhai Yu, and Yanjun Ma. 2022. PP-OCRv3: More Attempts for the Improvement of Ultra Lightweight OCR System. (2022). arXiv:2206.03001 http://arxiv.org/abs/2206.03001

[35]

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. (2023). arXiv:2301.12597 http://arxiv.org/abs/2301.12597

[36]

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. Proceedings of Machine Learning Research 162, 2 (2022), 12888–12900. arXiv:2201.12086

[37]

Toby Jia Jun Li, Lindsay Popowski, Tom M. Mitchell, and Brad A. Myers. 2021. Screen2vec: Semantic embedding of GUI screens and GUI components. Conference on Human Factors in Computing Systems - Proceedings (2021). https://doi.org/10.1145/3411764.3445049 arXiv:2101.11103

Digital Library

[38]

Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. 2020. Widget captioning: Generating natural language description for mobile user interface elements. EMNLP 2020 - 2020 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference 2015 (2020), 5495–5510. https://doi.org/10.18653/v1/2020.emnlp-main.443 arXiv:2010.04295

[39]

Tsung Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference. https://doi.org/10.1007/978-3-319-10602-1_48 arXiv:1405.0312

[40]

Thomas F. Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar. 2018. Learning design semantics for mobile apps. UIST 2018 - Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (2018), 569–579. https://doi.org/10.1145/3242587.3242650

Digital Library

[41]

Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. https://doi.org/10.48550/arXiv.1711.05101 arXiv:1711.05101 [cs, math].

[42]

Walid Maalej. 2009. Task-First or Context-First? Tool Integration Revisited. In 2009 IEEE/ACM International Conference on Automated Software Engineering. 344–355. https://doi.org/10.1109/ASE.2009.36 ISSN: 1938-4300.

Digital Library

[43]

Walid Maalej, Volodymyr Biryuk, Jialiang Wei, and Fabian Panse. 2024. On the Automated Processing of User Feedback. https://doi.org/10.48550/arXiv.2407.15519 arXiv:2407.15519 [cs].

[44]

Walid Maalej, Hans-Jörg Happel, and Asarnusch Rashid. 2009. When users become collaborators: towards continuous and context-aware user input. In Proceedings of the 24th ACM SIGPLAN conference companion on Object oriented programming systems languages and applications (OOPSLA ’09). Association for Computing Machinery, New York, NY, USA, 981–990. https://doi.org/10.1145/1639950.1640068

Digital Library

[45]

Daniel Martens and Walid Maalej. 2019. Extracting and Analyzing Context Information in User-Support Conversations on Twitter. In 2019 IEEE 27th International Requirements Engineering Conference (RE). 131–141. https://doi.org/10.1109/RE.2019.00024

[46]

Daniel Martens and Walid Maalej. 2019. Release Early, Release Often, and Watch Your Users’ Emotions: Lessons From Emotional Patterns. IEEE Software 36, 5 (Sept. 2019), 32–37. https://doi.org/10.1109/MS.2019.2923603 Conference Name: IEEE Software.

Digital Library

[47]

William Martin, Federica Sarro, Yue Jia, Yuanyuan Zhang, and Mark Harman. 2017. A Survey of App Store Analysis for Software Engineering. IEEE Transactions on Software Engineering 43, 9 (2017), 817–847. https://doi.org/10.1109/TSE.2016.2630689

Digital Library

[48]

Soumik Mohian and Christoph Csallner. 2022. PSDoodle: Fast App Screen Search via Partial Screen Doodle. Proceedings - 9th IEEE/ACM International Conference on Mobile Software Engineering and Systems, MOBILESoft 2022 January (2022), 89–99. https://doi.org/10.1145/3524613.3527816

Digital Library

[49]

Soumik Mohian and Christoph Csallner. 2023. Searching Mobile App Screens via Text + Doodle. (2023). arXiv:2305.06165 http://arxiv.org/abs/2305.06165

[50]

Ron Mokady, Amir Hertz, and Amit H Bermano. 2021. ClipCap: CLIP Prefix for Image Captioning. (2021). arXiv:2111.09734 https://arxiv.org/abs/2111.09734

[51]

Lloyd Montgomery, Davide Fucci, Abir Bouraffa, Lisa Scholz, and Walid Maalej. 2022. Empirical research on requirements quality: a systematic mapping study. Requirements Engineering 27, 2 (June 2022), 183–209. https://doi.org/10.1007/s00766-021-00367-z

Digital Library

[52]

Kevin Moran, Carlos Bernal-Cardenas, Michael Curcio, Richard Bonett, and Denys Poshyvanyk. 2020. Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps. IEEE Transactions on Software Engineering 46, 2 (2020), 196–221. https://doi.org/10.1109/TSE.2018.2844788 arXiv:1802.02312

[53]

Kevin Moran, Boyang Li, Carlos Bernal-Cárdenas, Dan Jelf, and Denys Poshyvanyk. 2018. Automated reporting of GUI design violations for mobile apps. In 40th International Conference on Software Engineering. 165–175. https://doi.org/10.1145/3180155.3180246 arXiv:1802.04732

Digital Library

[54]

Kevin Moran, Ali Yachnes, George Purnell, Junayed Mahmud, Michele Tufano, Carlos Bernal Cardenas, Denys Poshyvanyk, and Zach H’Doubler. 2022. An Empirical Investigation into the Use of Image Captioning for Automated Software Documentation. Proceedings - 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2022 (2022), 514–525. https://doi.org/10.1109/SANER53432.2022.00069 arXiv:2301.01224

[55]

Facundo Olano. 2015. Google Play Scraper. https://github.com/facundoolano/google-play-scraper. Accessed: 2024-3-10.

[56]

OpenAI. 2021. CLIP. https://github.com/openai/CLIP/blob/main/clip/clip.py. Accessed: 2024-3-10.

[57]

OpenAI. 2021. Model Card openai/clip-vit-base-patch32 on HuggingFace. https://huggingface.co/openai/clip-vit-base-patch32. Accessed: 2024-3-10.

[58]

Yen Dieu Pham, Davide Fucci, and Walid Maalej. 2018. A first implementation of a design thinking workshop during a mobile app development course project. In Proceedings of the 2nd International Workshop on Software Engineering Education for Millennials (SEEM ’18). Association for Computing Machinery, New York, NY, USA, 56–63. https://doi.org/10.1145/3194779.3194785

Digital Library

[59]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 8748–8763. arXiv:2103.00020 http://arxiv.org/abs/2103.00020https://proceedings.mlr.press/v139/radford21a.html

[60]

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21 (2020), 1–67. arXiv:1910.10683

[61]

Tobias Roehm, Nigar Gurbanova, Bernd Bruegge, Christophe Joubert, and Walid Maalej. 2013. Monitoring user interactions for supporting failure reproduction. In 2013 21st International Conference on Program Comprehension (ICPC). 73–82. https://doi.org/10.1109/ICPC.2013.6613835 ISSN: 1092-8138.

[62]

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. 2022. High-Resolution Image Synthesis with Latent Diffusion Models. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2022-June (2022), 10674–10685. https://doi.org/10.1109/CVPR52688.2022.01042 arXiv:2112.10752

[63]

Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings. 1–14. arXiv:1409.1556

[64]

Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. 2022. FLAVA: A Foundational Language And Vision Alignment Model. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2022-June (2022), 15617–15629. https://doi.org/10.1109/CVPR52688.2022.01519 arXiv:2112.04482

[65]

Filip Sondej. 2020. Autocorrect. https://github.com/filyp/autocorrect. Accessed: 2024-3-10.

[66]

Peter M. Stahl. 2022. Lingua. https://github.com/pemistahl/lingua-py. Accessed: 2024-3-10.

[67]

Statista. 2024. Number of available applications in the Google Play Store from December 2009 to December 2023. https://www.statista.com/statistics/266210/number-of-available-applications-in-the-google-play-store/. Accessed: 2024-5-10.

[68]

Srinivas Sunkara, Maria Wang, Lijuan Liu, Gilles Baechler, Yu Chung Hsiao, Jindong Chen, Abhanshu Sharma, and James Stout. 2022. Towards Better Semantic Understanding of Mobile Interfaces. Proceedings - International Conference on Computational Linguistics, COLING 29, 1 (2022), 5636–5650. arXiv:2210.02663

[69]

Vladimir Terekhov. 2023. 138 Features to Consider While Developing a Mobile App. https://attractgroup.com/blog/most-comprehensive-list-of-mobile-app-features-while-developing-a-mobile-application. Accessed: 2024-3-10.

[70]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems 2017-Decem, Nips (2017), 5999–6009. arXiv:1706.03762

[71]

Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. 2021. Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning. In UIST 2021 - Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology. 498–510. https://doi.org/10.1145/3472749.3474765 arXiv:2108.03353

Digital Library

[72]

Jialiang Wei, Anne-Lise Courbis, Thomas Lambolais, Gérard Dray, and Walid Maalej. 2024. On AI-Inspired UI-Design. http://arxiv.org/abs/2406.13631 arXiv:2406.13631 [cs].

[73]

Jialiang Wei, Anne-Lise Courbis, Thomas Lambolais, Binbin Xu, Pierre Louis Bernard, Gérard Dray, and Walid Maalej. 2024. Getting Inspiration for Feature Elicitation: App Store- vs. LLM-based Approach. https://doi.org/10.48550/arXiv.2408.17404 arXiv:2408.17404 [cs].

[74]

Jason Wu, Rebecca Krosnick, Eldon Schoop, Amanda Swearngin, Jeffrey P. Bigham, and Jeffrey Nichols. 2023. Never-ending Learning of User Interfaces. (2023). https://doi.org/10.1145/3586183.3606824 arXiv:2308.08726

Digital Library

[75]

Jason Wu, Yi-Hao Peng, Amanda Li, Amanda Swearngin, Jeffrey P. Bigham, and Jeffrey Nichols. 2024. UIClip: A Data-driven Model for Assessing User Interface Design. https://doi.org/10.48550/arXiv.2404.12500 arXiv:2404.12500 [cs].

[76]

Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. C-Pack: Packaged Resources To Advance General Chinese Embedding. (2023). arXiv:2309.07597 http://arxiv.org/abs/2309.07597

[77]

Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. 2024. Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs. https://doi.org/10.48550/arXiv.2404.05719 arXiv:2404.05719 [cs].

[78]

Alaa Zaki and Mohamed Abdallah. 2023. MASC: A Dataset for the Development and Classification of Mobile Applications Screens. (2023), 1–15. https://doi.org/10.21203/rs.3.rs-3786876/v1

[79]

Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. 2024. Vision-Language Models for Vision Tasks: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 46, 8 (Aug. 2024), 5625–5644. https://doi.org/10.1109/TPAMI.2024.3369699 Conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence.

Digital Library

[80]

Xiangyu Zhang, Lingling Fan, Sen Chen, Yucheng Su, and Boyuan Li. 2023. Scene-Driven Exploration and GUI Modeling for Android Apps. (2023). arXiv:2308.10228 http://arxiv.org/abs/2308.10228

Cited By

Raza AFarooq UNaseem KAlam SEhtisham Khan MMohammad AZakri WYasir Khan M(2024)A focused review on organic electrochemical transistors: A potential futuristic technological application in microelectronicsMicrochemical Journal10.1016/j.microc.2024.111737207(111737)Online publication date: Dec-2024
https://doi.org/10.1016/j.microc.2024.111737

Index Terms

GUing: A Mobile GUI Search Engine using a Vision-Language Model
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Graphical user interfaces
2. Software and its engineering
  1. Software creation and management
    1. Designing software

Recommendations

A Mobile App Search Engine

With the popularity of mobile apps on mobile devices based on iOS, Android, Blackberry and Windows Phone operating systems, the numbers of mobile apps in each of the respective native app stores are increasing in leaps and bounds. Currently there are ...
Automated model-based Android GUI testing using multi-level GUI comparison criteria
ASE '16: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering

Automated Graphical User Interface (GUI) testing is one of the most widely used techniques to detect faults in mobile applications (apps) and to test functionality and usability. GUI testing exercises behaviors of an application under test (AUT) by ...
Analyzing GUI running fluency for Android apps
MSCC '16: Proceedings of the 3rd ACM Workshop on Mobile Sensing, Computing and Communication

Android as a free open platform has become increasingly popular and been widespread adopted in mobile, tablet, and other devices. However, a great number of issues, such as inadequate quality and the fragmentation phenomenon, have emerged, enhancing the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology

ACM Transactions on Software Engineering and Methodology Just Accepted

EISSN:1557-7392

Table of Contents

Copyright © 2024 Copyright held by the owner/author(s).

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Online AM: 08 November 2024

Accepted: 01 October 2024

Revised: 26 September 2024

Received: 30 April 2024

Check for updates

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
182
Total Downloads

Downloads (Last 12 months)182
Downloads (Last 6 weeks)34

Reflects downloads up to 09 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Raza AFarooq UNaseem KAlam SEhtisham Khan MMohammad AZakri WYasir Khan M(2024)A focused review on organic electrochemical transistors: A potential futuristic technological application in microelectronicsMicrochemical Journal10.1016/j.microc.2024.111737207(111737)Online publication date: Dec-2024
https://doi.org/10.1016/j.microc.2024.111737

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Figures

Tables

Media