research-article

When is a Tool a Tool? User Perceptions of System Agency in Human–AI Co-Creative Drawing

Authors:

Francisco J IbarrolaAuthors Info & Claims

DIS '23: Proceedings of the 2023 ACM Designing Interactive Systems Conference

Pages 1978 - 1996

https://doi.org/10.1145/3563657.3595977

Published: 10 July 2023 Publication History

Abstract

This paper presents an analysis of the user experience of Reframer, a novel human-AI drawing interface designed with the iterative and reflective nature of creativity in mind. Collaboration with Reframer occurs in real time, with the user and the system drawing together concurrently. This approach is inspired by theories of creativity as being more problem-framing than problem-solving, and contrasts with the automated one-shot end-to-end workflows of most generative AI models. A 12-participant qualitative exploratory study of the capabilities of our prototype is detailed, as well as a thematic analysis of user attitudes towards drawing with it. The paper then describes two modified prototypes and a second 32-participant comparative study revealing how interface variations evoke differences in user attitudes and experiences. It concludes by proposing a model that characterises the conditions under which users experience co-creative AI as a collaborator, rather than a non-agentive tool.

References

[1]

Zirui An, Jingbo Yu, Runtao Liu, Chuang Wang, and Qian Yu. 2023. SketchInverter: Multi-Class Sketch-Based Image Generation via GAN Inversion. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 4319–4329.

[2]

Mandy M Archibald. 2016. Investigator triangulation: A collaborative strategy with potential for mixed methods research. Journal of mixed methods research 10, 3 (2016), 228–250.

[3]

Jennifer Attride-Stirling. 2001. Thematic networks: an analytic tool for qualitative research. Qualitative research 1, 3 (2001), 385–405.

[4]

Oliver Bown, Kazjon Grace, Liam Bray, and Dan Ventura. 2020. A Speculative Exploration of the Role of Dialogue in Human-ComputerCo-creation. In ICCC. 25–32.

[5]

Virginia Braun, Victoria Clarke, and Nikki Hayfield. 2022. ‘A starting point for your journey, not a map’: Nikki Hayfield in conversation with Virginia Braun and Victoria Clarke about thematic analysis. Qualitative Research in Psychology 19, 2 (2022), 424–445.

[6]

Linda Candy and Ernest Edmonds. 2002. Modeling co-creativity in art and technology. In Proceedings of the 4th conference on Creativity & cognition. 134–141.

Digital Library

[7]

Simon Colton, Geraint A Wiggins, 2012. Computational creativity: The final frontier?. In Ecai, Vol. 12. Montpelier, 21–26.

[8]

Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff. 2022. Vqgan-clip: Open domain image generation and editing with natural language guidance. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXVII. Springer, 88–105.

[9]

Mihaly Csikszentmihalyi. 1988. Society, culture and person: A system view of creativity. The Nature of Creativity (New York: Cambridge University Press, 1988) (1988), 325–339.

[10]

Mihaly Csikszentmihalyi. 1997. Flow and creativity.Namta Journal 22, 2 (1997), 60–97.

[11]

Nicholas Davis, Chih-PIn Hsiao, Kunwar Yashraj Singh, Lisa Li, Sanat Moningi, and Brian Magerko. 2015. Drawing apprentice: An enactive co-creative agent for artistic collaboration. In Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition. 185–186.

Digital Library

[12]

Nicholas Mark Davis. 2013. Human-computer co-creativity: Blending human and computational creativity. In Ninth Artificial Intelligence and Interactive Digital Entertainment Conference.

[13]

Sebastian Deterding, Jonathan Hook, Rebecca Fiebrink, Marco Gillies, Jeremy Gow, Memo Akten, Gillian Smith, Antonios Liapis, and Kate Compton. 2017. Mixed-initiative creative interfaces. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. 628–635.

Digital Library

[14]

Kees Dorst. 2006. Design problems and design paradoxes. Design issues 22, 3 (2006), 4–17.

[15]

Mathias Eitz, Kristian Hildebrand, Tamy Boubekeur, and Marc Alexa. 2010. Sketch-based image retrieval: Benchmark and bag-of-features descriptors. IEEE transactions on visualization and computer graphics 17, 11 (2010), 1624–1636.

[16]

Jonathan Ericson. 2022. Reimagining the Role of Friction in Experience Design. Journal of User Experience Vol 17, 4 (2022).

[17]

Catherine Flick and Kyle Worrall. 2022. The Ethics of Creative AI. In The Language of Creative AI: Practices, Aesthetics and Structures. Springer, 73–91.

[18]

Kevin Frans, LB Soros, and Olaf Witkowski. 2021. Clipdraw: Exploring text-to-drawing synthesis through language-image encoders. arXiv preprint arXiv:2106.14843 (2021).

[19]

Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2016. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2414–2423.

[20]

Gabriela Goldschmidt. 1991. The dialectics of sketching. Creativity research journal 4, 2 (1991), 123–143.

[21]

Graham Harman. 2011. Tool-being: Heidegger and the metaphysics of objects. Open Court.

[22]

James Hollan, Edwin Hutchins, and David Kirsh. 2000. Distributed cognition: toward a new foundation for human-computer interaction research. ACM Transactions on Computer-Human Interaction (TOCHI) 7, 2 (2000), 174–196.

Digital Library

[23]

Edwin L Hutchins, James D Hollan, and Donald A Norman. 1985. Direct manipulation interfaces. Human–computer interaction 1, 4 (1985), 311–338.

Digital Library

[24]

Francisco Ibarrola, Tomas Lawton, and Kazjon Grace. 2022. A collaborative, interactive and context-aware drawing agent for co-creative design. arXiv preprint arXiv:2209.12588 (2022).

[25]

Syed Muhammad Israr and Feng Zhao. 2022. Customizing GAN Using Few-shot Sketches. In Proceedings of the 30th ACM International Conference on Multimedia. 2229–2238.

Digital Library

[26]

Kenneth M Kahn. 1978. Ani: am example of computational creativity. In Proceedings of the 1978 AISB/GI Conference on Artificial Intelligence. 164–168.

[27]

Anna Kantosalo, Prashanth Thattai Ravikumar, Kazjon Grace, and Tapio Takala. 2020. Modalities, styles and strategies: An interaction framework for human-computer co-creativity. In ICCC. 57–64.

[28]

Victor Kaptelinin and Bonnie Nardi. 2012. Activity theory in HCI: Fundamentals and reflections. Synthesis Lectures Human-Centered Informatics 5, 1 (2012), 1–105.

[29]

Pegah Karimi, Kazjon Grace, Nicholas Davis, and Mary Lou Maher. 2018. Creative sketching apprentice: Supporting conceptual shifts in sketch ideation. In International Conference on-Design Computing and Cognition. Springer, 721–738.

[30]

Pegah Karimi, Kazjon Grace, Mary Lou Maher, and Nicholas Davis. 2018. Evaluating creativity in computational co-creative systems. arXiv preprint arXiv:1807.09886 (2018).

[31]

Veronika Köbberling and Peter P Wakker. 2005. An index of loss aversion. Journal of Economic Theory 122, 1 (2005), 119–131.

[32]

Robert Komaniecki. 2017. Analyzing collaborative flow in rap music. Music Theory Online 23, 4 (2017).

[33]

Glenn E Krasner, Stephen T Pope, 1988. A description of the model-view-controller user interface paradigm in the smalltalk-80 system. Journal of object oriented programming 1, 3 (1988), 26–49.

Digital Library

[34]

Max Kreminski, Isaac Karth, Michael Mateas, and Noah Wardrip-Fruin. 2022. Evaluating mixed-initiative creative interfaces via expressive range coverage analysis. In IUI Workshops. 34–45.

[35]

Tomas Lawton, Francisco J Ibarrola, Dan Ventura, and Kazjon Grace. 2023. Drawing with Reframer: Emergence and Control in Co-Creative AI. In Proceedings of the 28th International Conference on Intelligent User Interfaces. 264–277.

Digital Library

[36]

Tzu-Mao Li, Michal Lukáč, Michaël Gharbi, and Jonathan Ragan-Kelley. 2020. Differentiable vector graphics rasterization for editing and learning. ACM Transactions on Graphics 39, 6 (2020), 1–15.

Digital Library

[37]

Pattie Maes and Robyn Kozierok. 1993. Learning interface agents. In AAAI, Vol. 93. 459–465.

[38]

Mary Lou Maher and Josiah Poon. 1996. Modeling design exploration as co-evolution. Computer-Aided Civil and Infrastructure Engineering 11, 3 (1996), 195–209.

Digital Library

[39]

Susan Mohammed, Gül Okudan, and Madara Ogot. 2006. Tolerance for ambiguity: An investigation on its effect on student design performance. In 2006 Annual Conference & Exposition. 11–1343.

[40]

Allen Newell, J Clifford Shaw, and Herbert A Simon. 1962. The processes of creative thinking. In Contemporary Approaches to Creative Thinking,. Atherton Press.

[41]

Lior Noy, Nava Levit-Binun, and Yulia Golland. 2015. Being in the zone: physiological markers of togetherness in joint improvisation. Frontiers in human neuroscience 9 (2015), 187.

[42]

Rich Page, Tim Ash, and Maura Ginty. 2012. Landing page optimization: The definitive guide to testing and tuning for conversions. John Wiley & Sons.

[43]

Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. 2019. GauGAN: semantic image synthesis with spatially adaptive normalization. In ACM SIGGRAPH 2019 Real-Time Live!1–1.

Digital Library

[44]

Paul B Paulus and Bernard A Nijstad. 2003. Group creativity: Innovation through collaboration. Oxford University Press.

[45]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. PMLR, 8748–8763.

[46]

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125 (2022).

[47]

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10684–10695.

[48]

Benjamin Saunders, Julius Sim, Tom Kingstone, Shula Baker, Jackie Waterfield, Bernadette Bartlam, Heather Burroughs, and Clare Jinks. 2018. Saturation in qualitative research: exploring its conceptualization and operationalization. Quality & quantity 52 (2018), 1893–1907.

[49]

R Keith Sawyer. 2006. Group creativity: Musical performance and collaboration. Psychology of music 34, 2 (2006), 148–165.

[50]

Donald A Schon and Glenn Wiggins. 1992. Kinds of seeing and their functions in designing. Design studies 13, 2 (1992), 135–156.

[51]

Ben Shneiderman. 2000. Creating creativity: User interfaces for supporting innovation. ACM Transactions on Computer-Human Interaction (TOCHI) 7, 1 (2000), 114–138.

Digital Library

[52]

Jonathan A Smith. 2015. Qualitative psychology: A practical guide to research methods. Qualitative psychology (2015), 1–312.

[53]

Masaki Suwa, John Gero, and Terry Purcell. 2000. Unexpected discoveries and S-invention of design requirements: Important vehicles for a design process. Design Studies 21, 6 (2000), 539–567.

[54]

Edward R Tufte. 2006. Beautiful evidence. Graphis Pr.

[55]

AM Turing. 1950. Computing Machinery and Intelligence. (1950).

[56]

Barbara Tversky. 2002. What do sketches say about thinking. In 2002 AAAI Spring Symposium, Sketch Understanding Workshop, Stanford University, AAAI Technical Report SS-02-08, Vol. 148. 151.

[57]

Yael Vinker, Ehsan Pajouheshgar, Jessica Y Bo, Roman Christian Bachmann, Amit Haim Bermano, Daniel Cohen-Or, Amir Zamir, and Ariel Shamir. 2022. CLIPasso: Semantically-aware object sketching. arXiv preprint arXiv:2202.05822 (2022).

[58]

Sheng-Yu Wang, David Bau, and Jun-Yan Zhu. 2021. Sketch your own GAN. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 14050–14060.

[59]

Qiong Wu and Kathryn Cormican. 2016. Shared leadership and team creativity: A social network analysis in engineering design teams. Journal of technology management & innovation 11, 2 (2016), 2–12.

[60]

Georgios N Yannakakis, Antonios Liapis, and Constantine Alexopoulos. 2014. Mixed-initiative co-creativity. (2014).

[61]

John Zimmerman, Jodi Forlizzi, and Shelley Evenson. 2007. Research through design as a method for interaction design research in HCI. In Proceedings of the SIGCHI conference on Human factors in computing systems. 493–502.

Digital Library

Cited By

Jin YYoon JSelf JLee K(2024)Understanding Fashion Designers’ Behavior Using Generative AI for Early-Stage Concept Ideation and RevisionArchives of Design Research10.15187/adr.2024.07.37.3.2537:3(25-45)Online publication date: 31-Jul-2024
https://doi.org/10.15187/adr.2024.07.37.3.25
Bremers AJu W(2024)Can Machines Tell What People Want? Bringing Situated Intelligence to Generative AIProceedings of the Halfway to the Future Symposium10.1145/3686169.3686172(1-6)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3686169.3686172
Antony VHuang C(2024)ID.8: Co-Creating Visual Stories with Generative AIACM Transactions on Interactive Intelligent Systems10.1145/367227714:3(1-29)Online publication date: 2-Aug-2024
https://dl.acm.org/doi/10.1145/3672277
Show More Cited By

Index Terms

When is a Tool a Tool? User Perceptions of System Agency in Human–AI Co-Creative Drawing
1. Computer systems organization
  1. Dependable and fault-tolerant systems and networks
    1. Redundancy
  2. Embedded and cyber-physical systems
    1. Embedded systems
    2. Robotics
2. Networks
  1. Network properties
    1. Network reliability

Recommendations

Drawing Tool
TEI '12: Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction

Drawing Tool is a conceptual art piece intended to provoke discussion on the subject of authorship and originality through the means of an interactive experience, by taking the familiar experience of photography and translating it to the ...
Understanding User Perceptions, Collaborative Experience and User Engagement in Different Human-AI Interaction Designs for Co-Creative Systems
C&C '22: Proceedings of the 14th Conference on Creativity and Cognition

Human-AI co-creativity involves humans and AI collaborating on a shared creative product as partners. In a creative collaboration, communication is an essential component among collaborators. In many existing co-creative systems, users can communicate ...
Creating Word Paintings Jointly Considering Semantics, Attention, and Aesthetics
In this article, we present a content-aware method for generating a word painting. Word painting is a composite artwork made from the assemblage of words extracted from a given text, which carries similar semantics and visual features to a given source ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

DIS '23: Proceedings of the 2023 ACM Designing Interactive Systems Conference

July 2023

2717 pages

ISBN:9781450398930

DOI:10.1145/3563657

Editors:
Daragh Byrne
Carnegie Mellon University
,
Nikolas Martelaro
Carnegie Mellon University
,
Andy Boucher
Northumbria University
,
David Chatting
Goldsmiths, University of London
,
Sarah Fdili Alaoui
LISN-Université Paris Saclay
,
Sarah Fox
Carnegie Mellon University
,
Iohanna Nicenboim
Delft University of Technology
,
Cayley MacArthur
University of Waterloo

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 July 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Best Paper

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Australian Research Council

Conference

DIS '23

Sponsor:

SIGCHI

DIS '23: Designing Interactive Systems Conference

July 10 - 14, 2023

PA, Pittsburgh, USA

Acceptance Rates

Overall Acceptance Rate 1,158 of 4,684 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
1,906
Total Downloads

Downloads (Last 12 months)1,262
Downloads (Last 6 weeks)155

Reflects downloads up to 23 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Jin YYoon JSelf JLee K(2024)Understanding Fashion Designers’ Behavior Using Generative AI for Early-Stage Concept Ideation and RevisionArchives of Design Research10.15187/adr.2024.07.37.3.2537:3(25-45)Online publication date: 31-Jul-2024
https://doi.org/10.15187/adr.2024.07.37.3.25
Bremers AJu W(2024)Can Machines Tell What People Want? Bringing Situated Intelligence to Generative AIProceedings of the Halfway to the Future Symposium10.1145/3686169.3686172(1-6)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3686169.3686172
Antony VHuang C(2024)ID.8: Co-Creating Visual Stories with Generative AIACM Transactions on Interactive Intelligent Systems10.1145/367227714:3(1-29)Online publication date: 2-Aug-2024
https://dl.acm.org/doi/10.1145/3672277
Xu YCheng MKuzminykh A(2024)What Makes It Mine? Exploring Psychological Ownership over Human-AI Co-CreationsProceedings of the 50th Graphics Interface Conference10.1145/3670947.3670974(1-8)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3670947.3670974
Liu YSra M(2024)DanceGen: Supporting Choreography Ideation and Prototyping with Generative AIProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661594(920-938)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661594
Sivertsen CLøvlie A(2024)Exploring Aesthetic Qualities of Deep Generative Models through Technological (Art) MediationProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661498(2738-2752)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661498
Lim HCho JKim TPark JShin HChoi SPark SLee KKim JLee MHong H(2024)Co-Creating Question-and-Answer Style Articles with Large Language Models for Research PromotionProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660705(975-994)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3660705
Deshpande MPark JPait SMagerko B(2024)Perceptions of Interaction Dynamics in Co-Creative AI: A Comparative Study of Interaction Modalities in DrawctoProceedings of the 16th Conference on Creativity & Cognition10.1145/3635636.3656202(102-116)Online publication date: 23-Jun-2024
https://dl.acm.org/doi/10.1145/3635636.3656202
Moruzzi CMargarido S(2024)A User-centered Framework for Human-AI Co-creativityExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650929(1-9)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650929
Almeda SZamfirescu-Pereira JKim KMani Rathnam PHartmann B(2024)Prompting for Discovery: Flexible Sense-Making for AI Art-Making with DreamsheetsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642858(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642858
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents