research-article

Large-scale Text-to-Image Generation Models for Visual Artists’ Creative Works

Authors:

Jinwook SeoAuthors Info & Claims

IUI '23: Proceedings of the 28th International Conference on Intelligent User Interfaces

Pages 919 - 933

https://doi.org/10.1145/3581641.3584078

Published: 27 March 2023 Publication History

Abstract

Large-scale Text-to-image Generation Models (LTGMs) (e.g., DALL-E), self-supervised deep learning models trained on a huge dataset, have demonstrated the capacity for generating high-quality open-domain images from multi-modal input. Although they can even produce anthropomorphized versions of objects and animals, combine irrelevant concepts in reasonable ways, and give variation to any user-provided images, we witnessed such rapid technological advancement left many visual artists disoriented in leveraging LTGMs more actively in their creative works. Our goal in this work is to understand how visual artists would adopt LTGMs to support their creative works. To this end, we conducted an interview study as well as a systematic literature review of 72 system/application papers for a thorough examination. A total of 28 visual artists covering 35 distinct visual art domains acknowledged LTGMs’ versatile roles with high usability to support creative works in automating the creation process (i.e., automation), expanding their ideas (i.e., exploration), and facilitating or arbitrating in communication (i.e., mediation). We conclude by providing four design guidelines that future researchers can refer to in making intelligent user interfaces using LTGMs.

Supplementary Material

PDF File (LTGMs_IUI2023_presentation.pdf)

IUI submission

Download
7.36 MB

MP4 File (LTGMs_IUI2023_video.mp4)

IUI submission

Download
243.79 MB

References

[1]

Rinat Abdrashitov, Fanny Chevalier, and Karan Singh. 2020. Interactive Exploration and Refinement of Facial Expression Using Manifold Learning. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology(Virtual Event, USA) (UIST ’20). Association for Computing Machinery, New York, NY, USA, 778–790. https://doi.org/10.1145/3379337.3415877

Digital Library

[2]

Safinah Ali, Daniella DiPaola, Irene Lee, Jenna Hong, and Cynthia Breazeal. 2021. Exploring Generative Models with Middle School Students. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 678, 13 pages. https://doi.org/10.1145/3411764.3445226

Digital Library

[3]

Gary Ang and Ee Peng Lim. 2021. Learning Network-Based Multi-Modal Mobile User Interface Embeddings. In 26th International Conference on Intelligent User Interfaces (College Station, TX, USA) (IUI ’21). Association for Computing Machinery, New York, NY, USA, 366–376. https://doi.org/10.1145/3397481.3450693

Digital Library

[4]

Toshiki Aoki, Rintaro Chujo, Katsufumi Matsui, Saemi Choi, and Ari Hautasaari. 2022. EmoBalloon-Conveying Emotional Arousal in Text Chats with Speech Balloons. In CHI Conference on Human Factors in Computing Systems. 1–16.

[5]

Gökçe Elif Baykal, Maarten Van Mechelen, and Eva Eriksson. 2020. Collaborative technologies for children with special needs: A systematic literature review. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1–13.

Digital Library

[6]

Mehmet Aydin Baytas, Damla Çay, Yuchong Zhang, Mohammad Obaid, Asim Evren Yantaç, and Morten Fjeld. 2019. The design of social drones: A review of studies on autonomous flyers in inhabited environments. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–13.

Digital Library

[7]

Cynthia L Bennett, Cole Gleason, Morgan Klaus Scheuerman, Jeffrey P Bigham, Anhong Guo, and Alexandra To. 2021. “It’s Complicated”: Negotiating Accessibility and (Mis) Representation in Image Descriptions of Race, Gender, and Disability. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–19.

Digital Library

[8]

Oloff C. Biermann, Ning F. Ma, and Dongwook Yoon. 2022. From Tool to Companion: Storywriters Want AI Writers to Respect Their Personal Values and Writing Strategies. In Designing Interactive Systems Conference (Virtual Event, Australia) (DIS ’22). Association for Computing Machinery, New York, NY, USA, 1209–1227. https://doi.org/10.1145/3532106.3533506

Digital Library

[9]

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258(2021).

[10]

Renaud Bougueng Tchemeube, Jeffrey John Ens, and Philippe Pasquier. 2022. Calliope: A Co-creative Interface for Multi-Track Music Generation. In Creativity and Cognition. 608–611.

[11]

Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77–101.

[12]

Daniel Buschek, Martin Zürn, and Malin Eiband. 2021. The impact of multiple parallel phrase suggestions on email input and composition behaviour of native and non-native english writers. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–13.

Digital Library

[13]

Jaemin Cho, Abhay Zala, and Mohit Bansal. 2022. DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers. arXiv preprint arXiv:2202.04053(2022).

[14]

John Joon Young Chung, Shiqing He, and Eytan Adar. 2021. The intersection of users, roles, interactions, and technologies in creativity support tools. In Designing Interactive Systems Conference 2021. 1817–1833.

Digital Library

[15]

John Joon Young Chung, Shiqing He, and Eytan Adar. 2022. Artist Support Networks: Implications for Future Creativity Support Tools. In Designing Interactive Systems Conference (Virtual Event, Australia) (DIS ’22). Association for Computing Machinery, New York, NY, USA, 232–246. https://doi.org/10.1145/3532106.3533505

Digital Library

[16]

John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. 2022. TaleBrush: Sketching Stories with Generative Pretrained Language Models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 209, 19 pages. https://doi.org/10.1145/3491102.3501819

Digital Library

[17]

Felipe Costa, Sixun Ouyang, Peter Dolog, and Aonghus Lawlor. 2018. Automatic generation of natural language explanations. In Proceedings of the 23rd international conference on intelligent user interfaces companion. 1–2.

Digital Library

[18]

Giulia Di Fede, Davide Rocchesso, Steven P Dow, and Salvatore Andolina. 2022. The Idea Machine: LLM-based Expansion, Rewriting, Combination, and Suggestion of Ideas. In Creativity and Cognition. 623–627.

[19]

Carl DiSalvo, Phoebe Sengers, and Hrönn Brynjarsdóttir. 2010. Mapping the landscape of sustainable HCI. In Proceedings of the SIGCHI conference on human factors in computing systems. 1975–1984.

Digital Library

[20]

Dovetail. 2022. Dovetail. Retrieved October 5, 2022 from https://dovetailapp.com

[21]

Judith E Fan, Monica Dinculescu, and David Ha. 2019. Collabdraw: an environment for collaborative sketching with an artificial agent. In Proceedings of the 2019 on Creativity and Cognition. 556–561.

Digital Library

[22]

Jonas Frich, Lindsay MacDonald Vermeulen, Christian Remy, Michael Mose Biskjaer, and Peter Dalsgaard. 2019. Mapping the Landscape of Creativity Support Tools in HCI. Association for Computing Machinery, New York, NY, USA, 1–18. https://doi.org/10.1145/3290605.3300619

Digital Library

[23]

Emma Frid, Celso Gomes, and Zeyu Jin. 2020. Music creation by example. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1–13.

Digital Library

[24]

Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. 2022. Make-a-scene: Scene-based text-to-image generation with human priors. arXiv preprint arXiv:2203.13131(2022).

[25]

Nishit Gajjar, Vinoth Pandian Sermuga Pandian, Sarah Suleri, and Matthias Jarke. 2021. Akin: Generating ui wireframes from ui design patterns using deep learning. In 26th International Conference on Intelligent User Interfaces-Companion. 40–42.

Digital Library

[26]

Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. 2022. An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion. https://doi.org/10.48550/ARXIV.2208.01618

[27]

Dilrukshi Gamage, Piyush Ghasiya, Vamshi Bonagiri, Mark E Whiting, and Kazutoshi Sasahara. 2022. Are Deepfakes Concerning? Analyzing Conversations of Deepfakes on Reddit and Exploring Societal Implications. In CHI Conference on Human Factors in Computing Systems. 1–19.

[28]

Katy Ilonka Gero and Lydia B. Chilton. 2019. Metaphoria: An Algorithmic Companion for Metaphor Creation. Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300526

Digital Library

[29]

Katy Ilonka Gero, Vivian Liu, and Lydia Chilton. 2022. Sparks: Inspiration for science writing using language models. In Designing Interactive Systems Conference. 1002–1019.

Digital Library

[30]

Leo A Goodman. 1961. Snowball sampling. The annals of mathematical statistics(1961), 148–170.

[31]

Shunan Guo, Zhuochen Jin, Fuling Sun, Jingwen Li, Zhaorui Li, Yang Shi, and Nan Cao. 2021. Vinci: an intelligent graphic design system for generating advertising posters. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–17.

Digital Library

[32]

Megan Hofmann, Kelly Mack, Jessica Birchfield, Jerry Cao, Autumn G Hughes, Shriya Kurpad, Kathryn J Lum, Emily Warnock, Anat Caspi, Scott E Hudson, 2022. Maptimizer: Using Optimization to Tailor Tactile Maps to Users Needs. In CHI Conference on Human Factors in Computing Systems. 1–15.

Digital Library

[33]

Silas Hsu, Tiffany Wenting Li, Zhilin Zhang, Max Fowler, Craig Zilles, and Karrie Karahalios. 2021. Attitudes surrounding an imperfect AI autograder. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–15.

Digital Library

[34]

Tianran Hu, Anbang Xu, Zhe Liu, Quanzeng You, Yufan Guo, Vibha Sinha, Jiebo Luo, and Rama Akkiraju. 2018. Touch your heart: A tone-aware chatbot for customer care on social media. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–12.

Digital Library

[35]

Bernd Huber, Daniel McDuff, Chris Brockett, Michel Galley, and Bill Dolan. 2018. Emotional dialogue generation using image-grounded language models. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–12.

Digital Library

[36]

WOMBO Inc.2022. Dream by WOMBO. Retrieved October 5, 2022 from https://www.wombo.art

[37]

Ishani Janveja, Akshay Nambi, Shruthi Bannur, Sanchit Gupta, and Venkat Padmanabhan. 2020. Insight: monitoring the state of the driver in low-light using smartphones. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 3 (2020), 1–29.

Digital Library

[38]

Ellen Jiang, Edwin Toh, Alejandra Molina, Kristen Olson, Claire Kayacik, Aaron Donsbach, Carrie J Cai, and Michael Terry. 2022. Discovering the Syntax and Strategies of Natural Language Programming with Generative Language Models. In CHI Conference on Human Factors in Computing Systems. 1–19.

[39]

Martin Jonsson and Jakob Tholander. 2022. Cracking the code: Co-coding with AI in creative programming education. In Creativity and Cognition. 5–14.

[40]

Pegah Karimi, Jeba Rezwana, Safat Siddiqui, Mary Lou Maher, and Nasrin Dehbozorgi. 2020. Creative sketching partner: an analysis of human-AI co-creativity. In Proceedings of the 25th International Conference on Intelligent User Interfaces. 221–230.

Digital Library

[41]

Pegah Karimi, Jeba Rezwana, Safat Siddiqui, Mary Lou Maher, and Nasrin Dehbozorgi. 2020. Creative Sketching Partner: An Analysis of Human-AI Co-Creativity. In Proceedings of the 25th International Conference on Intelligent User Interfaces (Cagliari, Italy) (IUI ’20). Association for Computing Machinery, New York, NY, USA, 221–230. https://doi.org/10.1145/3377325.3377522

Digital Library

[42]

Mario Klingemann. 2022. Tweet. Retrieved October 5, 2022 from https://twitter.com/quasimondo/status/1512769106717593610

[43]

Hyung-Kwon Ko, Subin An, Gwanmo Park, Seung Kwon Kim, Daesik Kim, Bohyoung Kim, Jaemin Jo, and Jinwook Seo. 2022. We-toon: A Communication Support System between Writers and Artists in Collaborative Webtoon Sketch Revision. In The 35th Annual ACM Symposium on User Interface Software and Technology. 1–14.

Digital Library

[44]

Philippe Laban, Elicia Ye, Srujay Korlakunta, John Canny, and Marti Hearst. 2022. NewsPod: Automatic and Interactive News Podcasts. In 27th International Conference on Intelligent User Interfaces. 691–706.

[45]

SeungHun Lee, KangHee Lee, and Hyun-chul Kim. 2018. Content-based success prediction of crowdfunding campaigns: A deep learning approach. In Companion of the 2018 ACM conference on computer supported cooperative work and social computing. 193–196.

Digital Library

[46]

Lexica. 2022. Lexica. Retrieved October 5, 2022 from https://lexica.art

[47]

Jingyi Li, Sonia Hashim, and Jennifer Jacobs. 2021. What We Can Learn From Visual Artists About Software Development. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–14.

Digital Library

[48]

Toby Jia-Jun Li, Lindsay Popowski, Tom Mitchell, and Brad A Myers. 2021. Screen2vec: Semantic embedding of gui screens and gui components. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–15.

[49]

Xinyi Li, Liqiong Chang, Fangfang Song, Ju Wang, Xiaojiang Chen, Zhanyong Tang, and Zheng Wang. 2021. Crossgr: accurate and low-cost cross-target gesture recognition using Wi-Fi. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 1 (2021), 1–23.

Digital Library

[50]

Zach Lieberman. 2022. Tweet. Retrieved October 5, 2022 from https://twitter.com/zachlieberman/status/1512579367968423941

[51]

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision. Springer, 740–755.

[52]

Ruibo Liu, Chenyan Jia, and Soroush Vosoughi. 2021. A transformer-based framework for neutralizing and reversing the political polarity of news articles. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1(2021), 1–26.

Digital Library

[53]

Thomas F Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar. 2018. Learning design semantics for mobile apps. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. 569–579.

Digital Library

[54]

Vivian Liu and Lydia B Chilton. 2022. Design Guidelines for Prompt Engineering Text-to-Image Generative Models. In CHI Conference on Human Factors in Computing Systems. 1–23.

[55]

Vivian Liu, Han Qiao, and Lydia Chilton. 2022. Opal: Multimodal Image Generation for News Illustration. In The 35th Annual ACM Symposium on User Interface Software and Technology. 1–17.

[56]

Yihe Liu, Anushk Mittal, Diyi Yang, and Amy Bruckman. 2022. Will AI Console Me when I Lose my Pet? Understanding Perceptions of AI-Mediated Email Writing. In CHI Conference on Human Factors in Computing Systems. 1–13.

[57]

Ryan Louie, Andy Coenen, Cheng Zhi Huang, Michael Terry, and Carrie J Cai. 2020. Novice-AI music co-creation via AI-steering tools for deep generative models. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.

Digital Library

[58]

John Maeda. 2022. Tweet. Retrieved October 5, 2022 from https://twitter.com/johnmaeda/status/1513197684735328272

[59]

Mehran Maghoumi, Eugene Matthew Taranta, and Joseph LaViola. 2021. DeepNAG: Deep non-adversarial gesture generation. In 26th International Conference on Intelligent User Interfaces. 213–223.

Digital Library

[60]

Justin Matejka, Michael Glueck, Erin Bradner, Ali Hashemi, Tovi Grossman, and George Fitzmaurice. 2018. Dream lens: Exploration and visualization of large-scale generative design datasets. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–12.

Digital Library

[61]

Midjourney. 2022. Midjourney. Retrieved October 5, 2022 from https://www.midjourney.com

[62]

Kristina Milanovic and Jeremy Pitt. 2021. Misattribution of error origination: The impact of preconceived expectations in co-operative online games. In Designing Interactive Systems Conference 2021. 707–717.

[63]

Paritosh Mittal, Kunal Aggarwal, Pragya Paramita Sahu, Vishal Vatsalya, Soumyajit Mitra, Vikrant Singh, Viswanath Veera, and Shankar M Venkatesan. 2020. Photo-realistic emoticon generation using multi-modal input. In Proceedings of the 25th International Conference on Intelligent User Interfaces. 254–258.

Digital Library

[64]

Studio Morphogen. 2022. Artbreeder. Retrieved October 5, 2022 from https://www.artbreeder.com/

[65]

Mohammad Amin Mozaffari, Xinyuan Zhang, Jinghui Cheng, and Jin LC Guo. 2022. GANSpiration: Balancing Targeted and Serendipitous Inspiration in User Interface Design with Style-Based Generative Adversarial Network. In CHI Conference on Human Factors in Computing Systems. 1–15.

[66]

NovelAI. 2022. Anlatan. Retrieved October 5, 2022 from https://novelai.net

[67]

Changhoon Oh, Jinhan Choi, Sungwoo Lee, SoHyun Park, Daeryong Kim, Jungwoo Song, Dongwhan Kim, Joonhwan Lee, and Bongwon Suh. 2020. Understanding user perception of automated news generation system. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.

Digital Library

[68]

Changhoon Oh, Jungwoo Song, Jinhan Choi, Seonghyeon Kim, Sungwoo Lee, and Bongwon Suh. 2018. I lead, you help but only with enough details: Understanding user experience of co-creation with artificial intelligence. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–13.

Digital Library

[69]

Aadarsh Padiyath and Brian Magerko. 2021. desAIner: Exploring the Use of” Bad” Generative Adversarial Networks in the Ideation Process of Fashion Design. In Creativity and Cognition. 1–3.

[70]

PromptBase. 2022. PromptBase. Retrieved October 5, 2022 from https://promptbase.com

[71]

Felix Putze, Mazen Salous, and Tanja Schultz. 2018. Detecting memory-based interaction obstacles with a recurrent neural model of user behavior. In 23rd International Conference on Intelligent User Interfaces. 205–209.

Digital Library

[72]

Han Qiao, Vivian Liu, and Lydia Chilton. 2022. Initial Images: Using Image Prompts to Improve Subject Representation in Multimodal AI Generated Art. In Creativity and Cognition (Venice, Italy) (C&C ’22). Association for Computing Machinery, New York, NY, USA, 15–28. https://doi.org/10.1145/3527927.3532792

Digital Library

[73]

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125(2022).

[74]

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-shot text-to-image generation. In International Conference on Machine Learning. PMLR, 8821–8831.

[75]

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2021. High-Resolution Image Synthesis with Latent Diffusion Models. arxiv:2112.10752 [cs.CV]

[76]

Andrew Ross, Nina Chen, Elisa Zhao Hang, Elena L Glassman, and Finale Doshi-Velez. 2021. Evaluating the interpretability of generative models by interactive reconstruction. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–15.

Digital Library

[77]

Margarete Sandelowski. 1995. Sample size in qualitative research. Research in nursing & health 18, 2 (1995), 179–183.

[78]

Oliver Schmitt and Daniel Buschek. 2021. Characterchat: Supporting the creation of fictional characters through conversation and progressive manifestation with a chatbot. In Creativity and Cognition. 1–10.

[79]

Christoph Schuhmann, Romain Beaumont, Cade W Gordon, Ross Wightman, Theo Coombes, Aarush Katta, Clayton Mullis, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, 2022. LAION-5B: An open large-scale dataset for training next generation image-text models. (2022).

[80]

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 2021. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114(2021).

[81]

Sabrina Scuri, Marta Ferreira, Nuno Jardim Nunes, Valentina Nisi, and Cathy Mulligan. 2022. Hitting the Triple Bottom Line: Widening the HCI Approach to Sustainability. In CHI Conference on Human Factors in Computing Systems. 1–19.

Digital Library

[82]

Ben Shneiderman. 2020. Human-centered artificial intelligence: Reliable, safe & trustworthy. International Journal of Human–Computer Interaction 36, 6(2020), 495–504.

[83]

Ben Shneiderman. 2022. Human-Centered AI. Oxford University Press.

[84]

Ben Shneiderman and Catherine Plaisant. 2010. Designing the user interface: Strategies for effective human-computer interaction. Pearson Education India.

[85]

Maximilian Speicher, Brian D Hall, and Michael Nebeling. 2019. What is mixed reality?. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1–15.

Digital Library

[86]

Sangho Suh and Pengcheng An. 2022. Leveraging Generative Conversational AI to Develop a Creative Learning Environment for Computational Thinking. In 27th International Conference on Intelligent User Interfaces. 73–76.

Digital Library

[87]

Rashid Tahir, Brishna Batool, Hira Jamshed, Mahnoor Jameel, Mubashir Anwar, Faizan Ahmed, Muhammad Adeel Zaffar, and Muhammad Fareed Zaffar. 2021. Seeing is believing: Exploring perceptual differences in deepfake videos. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–16.

Digital Library

[88]

Ming Tao, Hao Tang, Fei Wu, Xiao-Yuan Jing, Bing-Kun Bao, and Changsheng Xu. 2022. DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16515–16525.

[89]

Michihiko Ueno and Shin’ichi Satoh. 2021. Continuous and Gradual Style Changes of Graphic Designs with Generative Model. In 26th International Conference on Intelligent User Interfaces. 280–289.

[90]

Josh Urban Davis, Fraser Anderson, Merten Stroetzel, Tovi Grossman, and George Fitzmaurice. 2021. Designing co-creative ai for virtual environments. In Creativity and Cognition. 1–11.

[91]

Fenna Van Nes, Tineke Abma, Hans Jonsson, and Dorly Deeg. 2010. Language differences in qualitative research: is meaning lost in translation?European journal of ageing 7, 4 (2010), 313–316.

[92]

Maarten Van Someren, Yvonne F Barnard, and J Sandberg. 1994. The think aloud method: a practical approach to modelling cognitive. London: AcademicPress 11(1994).

[93]

Richard Vogl, Hamid Eghbal-Zadeh, and Peter Knees. 2019. An automatic drum machine with touch UI based on a generative neural network. In Proceedings of the 24th International Conference on Intelligent User Interfaces: Companion. 91–92.

Digital Library

[94]

Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. 2011. The caltech-ucsd birds-200-2011 dataset. (2011).

[95]

Benedikte Wallace, Charles P Martin, Jim Tørresen, and Kristian Nymoen. 2021. Learning Embodied Sound-Motion Mappings: Evaluating AI-Generated Dance Improvisation. In Creativity and Cognition. 1–9.

[96]

Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. 2021. Screen2words: Automatic mobile UI summarization with multimodal learning. In The 34th Annual ACM Symposium on User Interface Software and Technology. 498–510.

Digital Library

[97]

Chuyu Wang, Lei Xie, Yuancan Lin, Wei Wang, Yingying Chen, Yanling Bu, Kai Zhang, and Sanglu Lu. 2021. Thru-the-wall Eavesdropping on Loudspeakers via RFID by Capturing Sub-mm Level Vibration. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 4 (2021), 1–25.

Digital Library

[98]

Jiewen Wang, Olivia Hsieh, Jack Gale, Sienna Xin Sun, and Sang-won Leigh. 2020. Latent Sheep Dreaming: Machine for Extrapolated Visual Inception. In Companion Publication of the 2020 ACM Designing Interactive Systems Conference. 481–484.

Digital Library

[99]

Thomas Weber, Heinrich Hußmann, Zhiwei Han, Stefan Matthes, and Yuanting Liu. 2020. Draw with me: Human-in-the-loop for image restoration. In Proceedings of the 25th International Conference on Intelligent User Interfaces. 243–253.

Digital Library

[100]

Justin D Weisz, Michael Muller, Stephanie Houde, John Richards, Steven I Ross, Fernando Martinez, Mayank Agarwal, and Kartik Talamadupula. 2021. Perfection not required? Human-AI partnerships in code translation. In 26th International Conference on Intelligent User Interfaces. 402–412.

Digital Library

[101]

Justin D Weisz, Michael Muller, Steven I Ross, Fernando Martinez, Stephanie Houde, Mayank Agarwal, Kartik Talamadupula, and John T Richards. 2022. Better together? an evaluation of ai-supported code translation. In 27th International Conference on Intelligent User Interfaces. 369–391.

Digital Library

[102]

Chenfei Wu, Jian Liang, Xiaowei Hu, Zhe Gan, Jianfeng Wang, Lijuan Wang, Zicheng Liu, Yuejian Fang, and Nan Duan. 2022. NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesis. arXiv preprint arXiv:2207.09814(2022).

[103]

Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. 2021. N\" uwa: Visual synthesis pre-training for neural visual world creation. arXiv preprint arXiv:2111.12417(2021).

[104]

Youngwoo Yoon, Keunwoo Park, Minsu Jang, Jaehong Kim, and Geehyuk Lee. 2021. Sgtoolkit: An interactive gesture authoring toolkit for embodied conversational agents. In The 34th Annual ACM Symposium on User Interface Software and Technology. 826–840.

Digital Library

[105]

Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, 2022. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789(2022).

[106]

Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022. Wordcraft: Story Writing With Large Language Models. In 27th International Conference on Intelligent User Interfaces. 841–852.

[107]

Beste F Yuksel, Pooyan Fazli, Umang Mathur, Vaishali Bisht, Soo Jung Kim, Joshua Junhee Lee, Seung Jung Jin, Yue-Ting Siu, Joshua A Miele, and Ilmi Yoon. 2020. Human-in-the-Loop Machine Learning to Increase Video Accessibility for Visually Impaired and Blind Users. In Proceedings of the 2020 ACM Designing Interactive Systems Conference. 47–60.

Digital Library

[108]

Paulina Yurman and Anuradha Venugopal Reddy. 2022. Drawing Conversations Mediated by AI. In Creativity and Cognition. 56–70.

[109]

Chao Zhang, Cheng Yao, Jiayi Wu, Weijia Lin, Lijuan Liu, Ge Yan, and Fangtian Ying. 2022. StoryDrawer: A Child–AI Collaborative Drawing System to Support Children’s Creative Visual Storytelling. In CHI Conference on Human Factors in Computing Systems. 1–15.

[110]

Enhao Zhang and Nikola Banovic. 2021. Method for Exploring Generative Adversarial Networks (GANs) via Automatically Generated Image Galleries. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–15.

Digital Library

[111]

Yijun Zhou, Yuki Koyama, Masataka Goto, and Takeo Igarashi. 2021. Interactive Exploration-Exploitation Balancing for Generative Melody Composition. In 26th International Conference on Intelligent User Interfaces (College Station, TX, USA) (IUI ’21). Association for Computing Machinery, New York, NY, USA, 43–47. https://doi.org/10.1145/3397481.3450663

Digital Library

Cited By

Alcaide-Marzal JDiego-Mas J(2025)Computers as co-creative assistants. A comparative study on the use of text-to-image AI models for computer aided conceptual designComputers in Industry10.1016/j.compind.2024.104168164(104168)Online publication date: Jan-2025
https://doi.org/10.1016/j.compind.2024.104168
Yang NHan EChoi J(2024)Explainability of Image Generative AI for Novice and Expert Users: A Comparative Study of Static and Dynamic ExplanationsJournal of Digital Contents Society10.9728/dcs.2024.25.8.226125:8(2261-2272)Online publication date: 31-Aug-2024
https://doi.org/10.9728/dcs.2024.25.8.2261
Alimjanovna Y(2024)The Important Significance of Introducing Students to Artists' Creations in Circle Classes (Ural Tansikbaev)Emergent Journal of Educational Discoveries and Lifelong Learning (EJEDL)10.47134/emergent.v3i1.413:1(9)Online publication date: 21-Feb-2024
https://doi.org/10.47134/emergent.v3i1.41
Show More Cited By

Index Terms

Large-scale Text-to-Image Generation Models for Visual Artists’ Creative Works
1. Computing methodologies
  1. Artificial intelligence
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Empirical studies in HCI

Recommendations

3DALL-E: Integrating Text-to-Image AI in 3D Design Workflows
DIS '23: Proceedings of the 2023 ACM Designing Interactive Systems Conference

Text-to-image AI are capable of generating novel images for inspiration, but their applications for 3D design workflows and how designers can build 3D models using AI-provided inspiration have not yet been explored. To investigate this, we integrated ...
Visual Task Performance and Spatial Abilities: An Investigation of Artists and Mathematicians
CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

This study builds on past research to present a domain-specific empirical investigation of artists and math & computer scientists on their respective relationships to, perceptions of, and interactions with data visualization. We conducted a three-phase ...
Intelligent Systems to Support Large-Scale Collective Creative Idea Generation
C&C '15: Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition

In recent years, it has become possible for large groups of people to collaborate and generate ideas together in ways that were not possible before. However, the large number of ideas and participants in this setting also pose new challenges in helping ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

IUI '23: Proceedings of the 28th International Conference on Intelligent User Interfaces

March 2023

972 pages

ISBN:9798400701061

DOI:10.1145/3581641

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 March 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Research Foundation of Korea
Information & communications Technology Planning & Evaluation(IITP)

Conference

IUI '23

Sponsor:

IUI '23: 28th International Conference on Intelligent User Interfaces

March 27 - 31, 2023

NSW, Sydney, Australia

Acceptance Rates

Overall Acceptance Rate 746 of 2,811 submissions, 27%

Upcoming Conference

IUI '25

Sponsor:
sigai
sigai

30th International Conference on Intelligent User Interfaces

March 24 - 27, 2025

Cagliari , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

49
Total Citations
View Citations
2,203
Total Downloads

Downloads (Last 12 months)1,413
Downloads (Last 6 weeks)184

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Alcaide-Marzal JDiego-Mas J(2025)Computers as co-creative assistants. A comparative study on the use of text-to-image AI models for computer aided conceptual designComputers in Industry10.1016/j.compind.2024.104168164(104168)Online publication date: Jan-2025
https://doi.org/10.1016/j.compind.2024.104168
Yang NHan EChoi J(2024)Explainability of Image Generative AI for Novice and Expert Users: A Comparative Study of Static and Dynamic ExplanationsJournal of Digital Contents Society10.9728/dcs.2024.25.8.226125:8(2261-2272)Online publication date: 31-Aug-2024
https://doi.org/10.9728/dcs.2024.25.8.2261
Alimjanovna Y(2024)The Important Significance of Introducing Students to Artists' Creations in Circle Classes (Ural Tansikbaev)Emergent Journal of Educational Discoveries and Lifelong Learning (EJEDL)10.47134/emergent.v3i1.413:1(9)Online publication date: 21-Feb-2024
https://doi.org/10.47134/emergent.v3i1.41
Hai-Jew S(2024)Remote Virtual SanctuaryMaking Art With Generative AI Tools10.4018/979-8-3693-1950-5.ch009(150-178)Online publication date: 19-Apr-2024
https://doi.org/10.4018/979-8-3693-1950-5.ch009
Gentet PCoffin MGentet YLee S(2024)From Text to Hologram: Creation of High-Quality Holographic Stereograms Using Artificial IntelligencePhotonics10.3390/photonics1109078711:9(787)Online publication date: 23-Aug-2024
https://doi.org/10.3390/photonics11090787
Ewuzie POwoade PAjayi OElamin MOmer MBiyabani A(2024)Generating African Artistic Styles Using Textual Inversion2024 IST-Africa Conference (IST-Africa)10.23919/IST-Africa63983.2024.10569305(1-9)Online publication date: 20-May-2024
https://doi.org/10.23919/IST-Africa63983.2024.10569305
Park HEirich JLuckow ASedlmair M(2024)"We Are Visual Thinkers, Not Verbal Thinkers!": A Thematic Analysis of How Professional Designers Use Generative AI Image Generation ToolsProceedings of the 13th Nordic Conference on Human-Computer Interaction10.1145/3679318.3685370(1-14)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3679318.3685370
Ling LChen XWen RLi TLC R(2024)Sketchar: Supporting Character Design and Illustration Prototyping Using Generative AIProceedings of the ACM on Human-Computer Interaction10.1145/36771028:CHI PLAY(1-28)Online publication date: 14-Oct-2024
https://dl.acm.org/doi/10.1145/3677102
Panchanadikar RFreeman G(2024)"I'm a Solo Developer but AI is My New Ill-Informed Co-Worker": Envisioning and Designing Generative AI to Support Indie Game DevelopmentProceedings of the ACM on Human-Computer Interaction10.1145/36770828:CHI PLAY(1-26)Online publication date: 14-Oct-2024
https://dl.acm.org/doi/10.1145/3677082
Lu QFang JYao ZYang YLyu SMi HYao L(2024)Large Language Model Agents Enabled Generative Design of Fluidic Computation InterfacesAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686351(1-3)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3672539.3686351
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents