Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

GUI2DSVec: Detecting Visual Design Smells Based on Semantic Embedding of GUI Images and Components

  • Conference paper
  • First Online:
Mobile and Ubiquitous Systems: Computing, Networking and Services (MobiQuitous 2023)

  • 179 Accesses

Abstract

Visual graphical user interface testing (VGT) can simulate end-user behavior on essentially any device, and data-driven VGT approaches can automatically discern certain visual design imperfections. The visual information representing GUI images and the semantic information of components are crucial for data-driven design defect detection methodologies. The GUI representation techniques in the existing VGT approaches are incapable of unifying multimodal semantic information such as text content, screen images, metadata, et al., but require subsequent manual execution and implementation. To address the dual issues, this paper presents GUI2DSVec (GUI to Design Smell Vector), a novel GUI representation method that automatically extracts semantic information from GUI images and components to embedding vectors without manual exertion. By representing image and component metadata into vector space, this study introduces an automated end-to-end mobile application design smell detection method based on GUI representation. Performance validation on 64,759 real Android UIs demonstrates GUI2DSVec’s efficiency in detecting visual design smells (accuracy@20 of 0.77). Additionally, the paper analyzes the impact of discrete representation modules and model variations on the performance of the smell detection method.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    The definition of design smell is inspired by code smell [13].

References

  1. Html5 specification finalized, squabbling over specs continues (2014). https://tinyurl.com/3ckk6sk2

  2. Google material design (2023). https://m2.material.io/components/

  3. Tesseract ocr (2023). https://tesseract-ocr.github.io/

  4. Alégroth, E., Gao, Z., Oliveira, R., Memon, A.: Conceptualization and evaluation of component-based testing unified with visual GUI testing: an empirical study. In: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pp. 1–10. IEEE (2015)

    Google Scholar 

  5. Alotaibi, A.S., Chiou, P.T., Halfond, W.G.: Automated repair of size-based inaccessibility issues in mobile applications. In: 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 730–742. IEEE (2021)

    Google Scholar 

  6. Amalfitano, D., Fasolino, A.R., Tramontana, P., Ta, B.D., Memon, A.M.: Mobiguitar: automated model-based testing of mobile apps. IEEE Softw. 32(5), 53–59 (2014)

    Article  Google Scholar 

  7. Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 6, 679–698 (1986)

    Article  Google Scholar 

  8. Chen, C., Feng, S., Xing, Z., Liu, L., Zhao, S., Wang, J.: Gallery dc: design search and knowledge discovery through auto-created GUI component gallery. Proc. ACM Hum.-Comput. Interact. 3(CSCW), 1–22 (2019)

    Google Scholar 

  9. Chen, J., et al.: Object detection for graphical user interface: old fashioned or deep learning or a combination? In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1202–1214 (2020)

    Google Scholar 

  10. Chen, K., Li, Y., Chen, Y., Fan, C., Hu, Z., Yang, W.: Glib: towards automated test oracle for graphically-rich applications. In: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1093–1104 (2021)

    Google Scholar 

  11. Cover, T., Hart, P.: Nearest neighbor pattern classification. IEEE Trans. Inf. Theory 13(1), 21–27 (1967)

    Article  Google Scholar 

  12. Deka, B., et al.: Rico: a mobile app dataset for building data-driven design applications. In: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pp. 845–854 (2017)

    Google Scholar 

  13. Fowler, M., Beck, K.: Refactoring: Improving the design of existing code. In: 11th European Conference, Jyväskylä, Finland (1997)

    Google Scholar 

  14. Galitz, W.O.: The Essential Guide to user Interface Design: an Introduction to GUI Design Principles and Techniques. Wiley, Hoboken (2007)

    Google Scholar 

  15. Kaluža, M., Troskot, K., Vukelić, B.: Comparison of front-end frameworks for web applications development. Zbornik Veleučilišta u Rijeci 6(1), 261–282 (2018)

    Article  Google Scholar 

  16. Li, T.J.J., Chen, J., Xia, H., Mitchell, T.M., Myers, B.A.: Multi-modal repairs of conversational breakdowns in task-oriented dialogs. In: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, pp. 1094–1107 (2020)

    Google Scholar 

  17. Li, T.J.J., Popowski, L., Mitchell, T., Myers, B.A.: Screen2vec: Semantic embedding of GUI screens and GUI components. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–15 (2021)

    Google Scholar 

  18. Li, T.J.J., Riva, O.: Kite: Building conversational bots from mobile apps. In: Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services, pp. 96–109 (2018)

    Google Scholar 

  19. Liu, Z.: Woodpecker: identifying and fixing android UI display issues. In: Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, pp. 334–336 (2022)

    Google Scholar 

  20. Liu, Z., Chen, C., Wang, J., Huang, Y., Hu, J., Wang, Q.: Owl eyes: Spotting UI display issues via visual understanding. In: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, pp. 398–409 (2020)

    Google Scholar 

  21. Liu, Z., Chen, C., Wang, J., Huang, Y., Hu, J., Wang, Q.: Nighthawk: fully automated localizing UI display issues via visual understanding. IEEE Trans. Software Eng. 49(1), 403–418 (2022)

    Article  Google Scholar 

  22. Moran, K., Li, B., Bernal-Cárdenas, C., Jelf, D., Poshyvanyk, D.: Automated reporting of GUI design violations for mobile apps. In: Proceedings of the 40th International Conference on Software Engineering, pp. 165–175 (2018)

    Google Scholar 

  23. Nielsen, J., Molich, R.: Heuristic evaluation of user interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 1990, pp. 249–256. Association for Computing Machinery, New York, NY, USA (1990)

    Google Scholar 

  24. Reimers, N., Gurevych, I.: Sentence-bert: sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019)

  25. Stone, M.: Cross-validatory choice and assessment of statistical predictions. J. Roy. Stat. Soc. Ser. B Methodol. 36(2), 111–133 (1974)

    Article  MathSciNet  Google Scholar 

  26. Su, T., et al.: Guided, stochastic model-based GUI testing of android apps. In: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, pp. 245–256 (2017)

    Google Scholar 

  27. Su, Y., et al.: The metamorphosis: automatic detection of scaling issues for mobile apps. In: 37th IEEE/ACM International Conference on Automated Software Engineering, pp. 1–12 (2022)

    Google Scholar 

  28. White, L., Almezen, H.: Generating test cases for GUI responsibilities using complete interaction sequences. In: Proceedings 11th International Symposium on Software Reliability Engineering. ISSRE 2000, pp. 110–121. IEEE (2000)

    Google Scholar 

  29. Xing, Y., Huang, J., Lai, Y.: Research and analysis of the front-end frameworks and libraries in e-business development. In: Proceedings of the 2019 11th International Conference on Computer and Automation Engineering, pp. 68–72 (2019)

    Google Scholar 

  30. Yang, B., Xing, Z., Xia, X., Chen, C., Ye, D., Li, S.: Don’t do that! hunting down visual design smells in complex UIS against design guidelines. In: 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pp. 761–772. IEEE (2021)

    Google Scholar 

  31. Zhang, Z., Feng, Y., Ernst, M.D., Porst, S., Dillig, I.: Checking conformance of applications against GUI policies. In: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 95–106 (2021)

    Google Scholar 

  32. Zhao, D., et al.: Seenomaly: vision-based linting of GUI animation effects against design-don’t guidelines. In: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pp. 1286–1297 (2020)

    Google Scholar 

  33. Zhou, X., et al.: East: an efficient and accurate scene text detector. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 5551–5560 (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Zhejiang University, Hangzhou, China

    Bo Yang & Shanping Li

Authors
  1. Bo Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shanping Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shanping Li .

Editor information

Editors and Affiliations

  1. Deakin University, Melbourne, VIC, Australia

    Arkady Zaslavsky

  2. Chongqing University of Posts and Telecommunications, Chongqing, China

    Zhaolong Ning

  3. Athens University of Economics and Business, Athens, Greece

    Vana Kalogeraki

  4. Swinburne University of Technology, Melbourne, VIC, Australia

    Dimitrios Georgakopoulos

  5. University of Pittsburgh, Pittsburgh, PA, USA

    Panos K. Chrysanthis

Rights and permissions

Reprints and permissions

Copyright information

© 2024 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yang, B., Li, S. (2024). GUI2DSVec: Detecting Visual Design Smells Based on Semantic Embedding of GUI Images and Components. In: Zaslavsky, A., Ning, Z., Kalogeraki, V., Georgakopoulos, D., Chrysanthis, P.K. (eds) Mobile and Ubiquitous Systems: Computing, Networking and Services. MobiQuitous 2023. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 594. Springer, Cham. https://doi.org/10.1007/978-3-031-63992-0_29

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-63992-0_29

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-63991-3

  • Online ISBN: 978-3-031-63992-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation