poster

Bridging the Gap Between UX Practitioners’ Work Practices and AI-Enabled Design Support Tools

Authors:

Chengzhi Zhang,

Toby Jia-Jun LiAuthors Info & Claims

CHI EA '22: CHI Conference on Human Factors in Computing Systems Extended Abstracts

Article No.: 268, Pages 1 - 7

https://doi.org/10.1145/3491101.3519809

Published: 28 April 2022 Publication History

Abstract

User interface (UI) and user experience (UX) design have become an indispensable part of today’s tech industry. Recently, much progress has been made in machine-learning-enabled design support tools for UX designers. However, few of these tools have been adopted by practitioners. To learn the underlying reasons and understand user needs for bridging this gap, we conducted a retrospective analysis with 8 UX professionals to understand their practice and identify opportunities for future research. We found that the current AI-enabled systems to support UX work mainly work on graphical interface elements, while design activities that involve more ‘design thinking” such as user interviews and user testings are more helpful for designers. Many current systems were also designed for overly-simplistic and generic use scenarios. We identified 4 areas in the UX workflow that can benefit from additional AI-enabled assistance: design inspiration search, design alternative exploration, design system customization, and design guideline violation check.

Supplementary Material

MP4 File (3491101.3519809-talk-video.mp4)

Talk Video

Download
15.87 MB

References

[1]

Mike Adams. 2022. Cure your Zoom amnesia with Grain 2.0. https://grain.co/blog/announcing-grain-2-0

[2]

Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018. From UI Design Image to GUI Skeleton: A Neural Machine Translator to Bootstrap Mobile GUI Implementation. In Proceedings of the 40th International Conference on Software Engineering (Gothenburg, Sweden) (ICSE ’18). Association for Computing Machinery, New York, NY, USA, 665–676. https://doi.org/10.1145/3180155.3180240

Digital Library

[3]

Michael Chromik, Florian Lachner, and Andreas Butz. 2020. ML for UX? - An Inventory and Predictions on the Use of Machine Learning Techniques for UX Research. In Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society. Association for Computing Machinery, New York, NY, USA, Article 57, 11 pages. https://doi.org/10.1145/3419249.3420163

Digital Library

[4]

Andy Coenen, Luke Davis, Daphne Ippolito, Emily Reif, and Ann Yuan. 2021. Wordcraft: a Human-AI Collaborative Editor for Story Writing. arXiv preprint arXiv:2107.07430(2021).

[5]

Lucas Colusso, Cynthia L. Bennett, Gary Hsieh, and Sean A. Munson. 2017. Translational Resources: Reducing the Gap Between Academic Research and HCI Practice. In Proceedings of the 2017 Conference on Designing Interactive Systems (Edinburgh, United Kingdom) (DIS ’17). Association for Computing Machinery, New York, NY, USA, 957–968. https://doi.org/10.1145/3064663.3064667

Digital Library

[6]

Scott Davidoff, Min Kyung Lee, Anind K Dey, and John Zimmerman. 2007. Rapidly exploring application design through speed dating. In International Conference on Ubiquitous Computing. Springer, 429–446. https://doi.org/10.1007/978-3-540-74853-3_25

[7]

Nicholas Davis, Chih-PIn Hsiao, Kunwar Yashraj Singh, Lisa Li, and Brian Magerko. 2016. Empirically Studying Participatory Sense-Making in Abstract Drawing with a Co-Creative Cognitive Agent. In Proceedings of the 21st International Conference on Intelligent User Interfaces (Sonoma, California, USA) (IUI ’16). Association for Computing Machinery, New York, NY, USA, 196–207. https://doi.org/10.1145/2856767.2856795

Digital Library

[8]

Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST ’17). Association for Computing Machinery, New York, NY, USA, 845–854. https://doi.org/10.1145/3126594.3126651

Digital Library

[9]

Zhen Feng, Jiaqi Fang, Bo Cai, and Yingtao Zhang. 2021. GUIS2Code: A Computer Vision Tool to Generate Code Automatically from Graphical User Interface Sketches. In International Conference on Artificial Neural Networks. Springer, 53–65.

Digital Library

[10]

Nishit Gajjar, Vinoth Pandian Sermuga Pandian, Sarah Suleri, and Matthias Jarke. 2021. Akin: Generating UI Wireframes From UI Design Patterns Using Deep Learning. (2021), 40–42. https://doi.org/10.1145/3397482.3450727

[11]

Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2015. A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576(2015).

[12]

Katy Ilonka Gero and Lydia B Chilton. 2019. Metaphoria: An algorithmic companion for metaphor creation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.

Digital Library

[13]

Leo A Goodman. 1961. Snowball sampling. The annals of mathematical statistics(1961), 148–170.

[14]

Colin M. Gray. 2014. Evolution of Design Competence in UX Practice. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 1645–1654. https://doi.org/10.1145/2556288.2557264

Digital Library

[15]

David Ha and Douglas Eck. 2017. A neural representation of sketch drawings. arXiv preprint arXiv:1704.03477(2017).

[16]

Eric Horvitz. 1999. Principles of Mixed-Initiative User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Pittsburgh, Pennsylvania, USA) (CHI ’99). Association for Computing Machinery, New York, NY, USA, 159–166. https://doi.org/10.1145/302979.303030

Digital Library

[17]

Forrest Huang, John F. Canny, and Jeffrey Nichols. 2019. Swire: Sketch-Based User Interface Retrieval. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–10. https://doi.org/10.1145/3290605.3300334

Digital Library

[18]

Forrest Huang, Gang Li, Xin Zhou, John F Canny, and Yang Li. 2021. Creating User Interface Mock-ups from High-Level Text Descriptions with Deep-Learning Models. arXiv preprint arXiv:2110.07775(2021).

[19]

Yue Jiang, Yuwen Lu, Jeffrey Nichols, Wolfgang Stuerzlinger, Chun Yu, Christof Lutteroth, Yang Li, Ranjitha Kumar, and Toby Jia-Jun Li. 2022. Computational Approaches for Understanding, Generating, and Adapting User Interfaces. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA. https://doi.org/10.1145/3491101.3504030

Digital Library

[20]

Pegah Karimi, Jeba Rezwana, Safat Siddiqui, Mary Lou Maher, and Nasrin Dehbozorgi. 2020. Creative sketching partner: an analysis of human-AI co-creativity. In Proceedings of the 25th International Conference on Intelligent User Interfaces. 221–230.

Digital Library

[21]

Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114(2013).

[22]

Max Kreminski, Melanie Dickinson, Michael Mateas, and Noah Wardrip-Fruin. 2020. Why Are We Like This?: The AI Architecture of a Co-Creative Storytelling Game. In International Conference on the Foundations of Digital Games (Bugibba, Malta) (FDG ’20). Association for Computing Machinery, New York, NY, USA, Article 13, 4 pages. https://doi.org/10.1145/3402942.3402953

Digital Library

[23]

James A Landay. 1996. SILK: sketching interfaces like krazy. In Conference companion on Human factors in computing systems. 398–399.

Digital Library

[24]

Effie Lai-Chong Law, Paul Van Schaik, and Virpi Roto. 2014. Attitudes towards user experience (UX) measurement. International Journal of Human-Computer Studies 72, 6 (2014), 526–541.

[25]

Hsin-Ying Lee, Lu Jiang, Irfan Essa, Phuong B Le, Haifeng Gong, Ming-Hsuan Yang, and Weilong Yang. 2020. Neural Design Network: Graphic Layout Generation with Constraints. (2020). https://doi.org/10.1007/978-3-030-58580-8

[26]

Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu. 2019. LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators. 7th International Conference on Learning Representations, ICLR 2019 (jan 2019). arxiv:1901.06767

[27]

Toby Jia-Jun Li, Lindsay Popowski, Tom Mitchell, and Brad A Myers. 2021. Screen2Vec: Semantic Embedding of GUI Screens and GUI Components. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–15.

[28]

Q Vera Liao, Moninder Singh, Yunfeng Zhang, and Rachel Bellamy. 2021. Introduction to explainable ai. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1–3.

Digital Library

[29]

Ryan Louie, Andy Coenen, Cheng Zhi Huang, Michael Terry, and Carrie J Cai. 2020. Novice-AI music co-creation via AI-steering tools for deep generative models. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.

Digital Library

[30]

Kevin Moran, Carlos Bernal-Cardenas, Michael Curcio, Richard Bonett, and Denys Poshyvanyk. 2020. Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps. IEEE Transactions on Software Engineering 46, 2 (feb 2020), 196–221. https://doi.org/10.1109/TSE.2018.2844788

[31]

Michael J Muller and Sarah Kuhn. 1993. Participatory design. Commun. ACM 36, 6 (1993), 24–28.

Digital Library

[32]

Brad Myers, Scott E. Hudson, and Randy Pausch. 2000. Past, Present, and Future of User Interface Software Tools. ACM Trans. Comput.-Hum. Interact. 7, 1 (mar 2000), 3–28. https://doi.org/10.1145/344949.344959

Digital Library

[33]

Brad A. Myers. 1995. User Interface Software Tools. ACM Trans. Comput.-Hum. Interact. 2, 1 (mar 1995), 64–103. https://doi.org/10.1145/200968.200971

Digital Library

[34]

Brad A Myers, Dario A Giuse, Roger B Dannenberg, Brad Vander Zanden, David S Kosbie, Edward Pervin, Andrew Mickish, and Philippe Marchal. 1995. GARNET comprehensive support for graphical, highly interactive user interfaces. In Readings in Human–Computer Interaction. Elsevier, 357–371.

[35]

Jakob Nielsen. 2017. A 100-Year View of User Experience (by Jakob Nielsen). https://www.nngroup.com/articles/100-years-ux/

[36]

Donald A Norman. 2010. The research-Practice Gap: The need for translational developers. interactions 17, 4 (2010), 9–12.

Digital Library

[37]

Changhoon Oh, Jungwoo Song, Jinhan Choi, Seonghyeon Kim, Sungwoo Lee, and Bongwon Suh. 2018. I Lead, You Help but Only with Enough Details: Understanding User Experience of Co-Creation with Artificial Intelligence. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3173574.3174223

Digital Library

[38]

Kara Pernice. 2018. Affinity Diagramming: Collaboratively Sort UX Findings & Design Ideas. https://www.nngroup.com/articles/affinity-diagram/

[39]

P Jonathon Phillips, Carina A Hahn, Peter C Fontana, David A Broniatowski, and Mark A Przybocki. 2020. Four principles of explainable artificial intelligence. Gaithersburg, Maryland(2020).

[40]

Mitchel Resnick, Brad Myers, Kumiyo Nakakoji, Ben Shneiderman, Randy Pausch, Ted Selker, and Mike Eisenberg. 2005. Design principles for tools to support creative thinking. (2005).

[41]

Joy Robinson, Candice Lanius, and Ryan Weber. 2018. The past, present, and future of UX empirical research. Communication Design Quarterly Review 5, 3 (2018), 10–23.

Digital Library

[42]

[42] Maria Rosala.2019. https://www.nngroup.com/articles/thematic-analysis/

[43]

Emma Rose and Josh Tenenberg. 2017. Making practice-level struggles visible: researching UX practice to inform pedagogy. Communication Design Quarterly Review 5, 1 (2017), 89–97.

Digital Library

[44]

Daniel M Russell and Ed H Chi. 2014. Looking back: Retrospective study methods for HCI. In Ways of Knowing in HCI. Springer, 373–393.

[45]

[45] Kim Salazar.2020. https://www.nngroup.com/articles/contextual-inquiry/

[46]

Vinoth Pandian Sermuga Pandian, Sarah Suleri, and Prof. Dr. Matthias Jarke. 2021. UISketch: A Large-Scale Dataset of UI Element Sketches. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, Article 319, 14 pages. https://doi.org/10.1145/3411764.3445784

Digital Library

[47]

Ben Shneiderman, Michael Leavitt, 2006. Research-based web design & usability guidelines. Washington DC, Department of Health and Human Services (2006).

[48]

Erik Stolterman. 2008. The nature of design practice and implications for interaction design research. International Journal of Design 2, 1 (2008).

[49]

Hariharan Subramonyam, Colleen Seifert, and Eytan Adar. 2021. ProtoAI: Model-Informed Prototyping for AI-Powered Interfaces. In 26th International Conference on Intelligent User Interfaces. Association for Computing Machinery, New York, NY, USA, 48–58. https://doi.org/10.1145/3397481.3450640

[50]

Amanda Swearngin, Chenglong Wang, Alannah Oleson, James Fogarty, and Amy J Ko. 2020. Scout: Rapid exploration of interface layout alternatives through high-level design constraints. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.

Digital Library

[51]

Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. 2021. Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning. In The 34th Annual ACM Symposium on User Interface Software and Technology. 498–510.

Digital Library

[52]

Marcus Winter and Phil Jackson. 2020. Flatpack ML: How to support designers in creating a new generation of customizable machine learning applications. In International Conference on Human-Computer Interaction. Springer, 175–193.

Digital Library

[53]

Jacob O Wobbrock, Shaun K Kane, Krzysztof Z Gajos, Susumu Harada, and Jon Froehlich. 2011. Ability-based design: Concept, principles and examples. ACM Transactions on Accessible Computing (TACCESS) 3, 3 (2011), 1–27.

Digital Library

[54]

Bo Yang, Zhenchang Xing, Xin Xia, Chunyang Chen, Deheng Ye, and Shanping Li. 2021. Don’t do that! hunting down visual design smells in complex uis against design guidelines. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 761–772.

Digital Library

[55]

Qian Yang, Alex Scuito, John Zimmerman, Jodi Forlizzi, and Aaron Steinfeld. 2018. Investigating How Experienced UX Designers Effectively Work with Machine Learning. In Proceedings of the 2018 Designing Interactive Systems Conference (Hong Kong, China) (DIS ’18). Association for Computing Machinery, New York, NY, USA, 585–596. https://doi.org/10.1145/3196709.3196730

Digital Library

[56]

Qian Yang, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020. Re-examining whether, why, and how human-AI interaction is uniquely difficult to design. In Proceedings of the 2020 chi conference on human factors in computing systems. 1–13.

Digital Library

[57]

ZiCheng Zhang, CongYing Han, and TianDe Guo. 2021. ExSinGAN: Learning an Explainable Generative Model from a Single Image. arXiv preprint arXiv:2105.07350(2021).

[58]

Tianming Zhao, Chunyang Chen, Yuanning Liu, and Xiaodong Zhu. 2021. GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 748–760. https://doi.org/10.1109/ICSE43902.2021.00074

Digital Library

Cited By

(2024)Nowe media i technologie w komunikacji marketingowej – wybrane obszary aplikacji10.22367/uekat.9788378758884Online publication date: 2024
https://doi.org/10.22367/uekat.9788378758884
Uusitalo SSalovaara AJokela TSalmimaa M(2024)”Clay to Play With”: Generative AI Tools in UX and Industrial Design PracticeProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661624(1566-1578)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661624
Peng XKoch JMackay W(2024)DesignPrompt: Using Multimodal Interaction for Design Exploration with Generative AIProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661588(804-818)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661588
Show More Cited By

Recommendations

Using human factors standards to support user experience and agile design
UAHCI'13: Proceedings of the 7th international conference on Universal Access in Human-Computer Interaction: design methods, tools, and interaction techniques for eInclusion - Volume Part I

The ISO 9241-210 standard provides a framework for human-centred design (HCD) activities comprising the four stages: context of use, specification of user and organisational requirements, design solutions, and evaluation against requirements. Other ...
Task Allocation Between UX Specialists and Developers in Agile Software Development Projects
Human-Computer Interaction – INTERACT 2015
Abstract
Synchronizing efforts between developers and user experience (UX) specialists is one of the major challenges in agile UX work. In this paper, we report results of a study conducted over a release cycle of six agile software development projects in ...
Usability and UX practices in small enterprises: lessons from a survey of the Brazilian context
SIGDOC '16: Proceedings of the 34th ACM International Conference on the Design of Communication

The role of small and medium enterprises (SME), especially small businesses, has significantly increased in the developing of interactive applications. Given the relevance of these companies in the market, the adoption of usability and user experience (...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI EA '22: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems

April 2022

3066 pages

ISBN:9781450391566

DOI:10.1145/3491101

Editors:
Simone Barbosa
PUC-Rio, Brazil
,
Cliff Lampe
University of Michigan, USA
,
Caroline Appert
Université Paris-Saclay, France
,
David A. Shamma
Toyota Research Institute, USA

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 April 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Poster
Research
Refereed limited

Funding Sources

Asia Research Collaboration Grant from Notre Dame International
Google Cloud Research Credit Grant

Conference

CHI '22

Sponsor:

SIGCHI

CHI '22: CHI Conference on Human Factors in Computing Systems

April 29 - May 5, 2022

LA, New Orleans, USA

Acceptance Rates

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
1,565
Total Downloads

Downloads (Last 12 months)838
Downloads (Last 6 weeks)89

Reflects downloads up to 10 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

(2024)Nowe media i technologie w komunikacji marketingowej – wybrane obszary aplikacji10.22367/uekat.9788378758884Online publication date: 2024
https://doi.org/10.22367/uekat.9788378758884
Uusitalo SSalovaara AJokela TSalmimaa M(2024)”Clay to Play With”: Generative AI Tools in UX and Industrial Design PracticeProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661624(1566-1578)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661624
Peng XKoch JMackay W(2024)DesignPrompt: Using Multimodal Interaction for Design Exploration with Generative AIProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661588(804-818)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661588
Takaffoli MLi SMäkelä V(2024)Generative AI in User Experience Design and Research: How Do UX Practitioners, Teams, and Companies Use GenAI in Industry?Proceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660720(1579-1593)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3660720
Wang ZShen LKuang EZhang SFan M(2024)Exploring the Impact of Artificial Intelligence-Generated Content (AIGC) Tools on Social Dynamics in UX CollaborationProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660703(1594-1606)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3660703
Andrew SBishop CTigwell G(2024)Light and Dark ModeProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435398:1(1-23)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643539
Chen JYoon D(2024)Exploring the Diminishing Allure of Paper and Low-Fidelity Prototyping Among Designers in the Software Industry: Impacts of Hybrid Work, Digital Tools, and Corporate CultureProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642774(1-14)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642774
Xia CMin TZhang DWang C(2024)Understanding the Needs of Novice Developers in Creating Self-Powered IoTProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642576(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642576
Suh SChen MMin BLi TXia H(2024)Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-CreationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642400(1-26)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642400
Li JCao HLin LHou YZhu REl Ali A(2024)User Experience Design Professionals’ Perceptions of Generative Artificial IntelligenceProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642114(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642114
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents