research-article

Designing Co-Creative AI for Virtual Environments

Authors:

Josh Urban Davis,

Fraser Anderson,

Merten Stroetzel,

George FitzmauriceAuthors Info & Claims

C&C '21: Proceedings of the 13th Conference on Creativity and Cognition

Article No.: 26, Pages 1 - 11

https://doi.org/10.1145/3450741.3465260

Published: 22 June 2021 Publication History

Abstract

Co-creative AI tools provide a method of creative collaboration between a user and machine. One form of co-creative AI called generative design requires the user to input design parameters and wait substantial periods of time while the system computes design solutions. We explore this interaction dynamic by providing an embodied experience in VR. Calliope is a virtual reality (VR) system that enables users to explore and manipulate generative design solutions in real time. Calliope accounts for the typical idle times in the generative design process by using a virtual environment to encourage parallelized and embodied data-exploration and synthesis, while maintaining a tight human-in-the-loop collaboration with the underlying algorithms. In this paper we discuss design considerations informed by formative studies with generative designers and artists and provide design guidelines to aid others in the development of co-creative AI systems in virtual environments.

References

[1]

Grégoire Allaire, François Jouve, and Anca-Maria Toader. 2002. A level-set method for shape optimization. Comptes Rendus Mathematique 334, 12: 1125–1130. http://doi.org/10.1016/S1631-073X(02)02412-3

[2]

Mayra Barrera Machuca, Paul Asente, Jingwan Lu, Byungmoon Kim, and Wolfgang Stuerzlinger. 2017. Multiplanes: Assisted Freehand VR Drawing. 1–3. http://doi.org/10.1145/3131785.3131794

Digital Library

[3]

Martin Bendsøe. 1989. Bendsoe, M.P.: Optimal Shape Design as a Material Distribution Problem. Structural Optimization 1, 193-202. Structural Optimization 1: 193–202. http://doi.org/10.1007/BF01650949

[4]

Martin Philip Bendsoe and Ole Sigmund. 2004. Topology Optimization: Theory, Methods, and Applications. Springer-Verlag, Berlin Heidelberg. http://doi.org/10.1007/978-3-662-05086-6

[5]

Jorge Loise Borges. (1944b). Las ruinas circulares. In Ficciones, Jorge Luis Borges: obras completas. Harmondsworth : Penguin, pp. 72–79. (1982)

[6]

Andrew Brock, Jeff Donahue, and Karen Simonyan. 2019. Large Scale GAN Training for High Fidelity Natural Image Synthesis. arXiv:1809.11096 [cs, stat]. Retrieved May 5, 2020 from http://arxiv.org/abs/1809.11096

[7]

Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, 2015. ShapeNet: An Information-Rich 3D Model Repository. arXiv:1512.03012 [cs]. Retrieved May 5, 2020 from http://arxiv.org/abs/1512.03012

[8]

Anthony Chen, Ye Tao, Guanyun Wang, 2018. Forte: User-Driven Generative Design. 1–12. http://doi.org/10.1145/3173574.3174070

Digital Library

[9]

Tao Du, Adriana Schulz, Bo Zhu, Bernd Bickel, and Wojciech Matusik. 2016. Computational multicopter design. ACM Transactions on Graphics 35, 6: 1–10. http://doi.org/10.1145/2980179.2982427

Digital Library

[10]

Upol Ehsan and Mark O Riedl. On Design and Evaluation of Human-centered Explainable AI systems. 6.

[11]

George Ferguson, James Allen, and Brad Miller. 1996. TRAINS-95: Towards a Mixed-Initiative Planning Assistant. 8.

[12]

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, 2014. Generative Adversarial Networks. arXiv:1406.2661 [cs, stat]. Retrieved May 5, 2020 from http://arxiv.org/abs/1406.2661

[13]

Saul Greenberg and Bill Buxton. 2008. Usability evaluation considered harmful (some of the time). Proceeding of the twenty-sixth annual CHI conference on Human factors in computing systems - CHI ’08, ACM Press, 111. http://doi.org/10.1145/1357054.1357074

Digital Library

[14]

Naama Hadad, Lior Wolf, and Moni Shahar. 2018. A Two-Step Disentanglement Method. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, 772–780. http://doi.org/10.1109/CVPR.2018.00087

[15]

Joshua Harman. 2001. Creating a memory palace using a computer. CHI ’01 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’01), Association for Computing Machinery, 407–408. http://doi.org/10.1145/634067.634306

Digital Library

[16]

William E. Lorensen and Harvey E. Cline. 1987. Marching cubes: A high resolution 3D surface construction algorithm. ACM SIGGRAPH Computer Graphics 21, 4: 163–169. http://doi.org/10.1145/37402.37422

Digital Library

[17]

Jonàs Martínez, Jérémie Dumas, Sylvain Lefebvre, and Li-Yi Wei. 2015. Structure and appearance optimization for controllable shape design. ACM Transactions on Graphics 34, 6: 1–11. http://doi.org/10.1145/2816795.2818101

Digital Library

[18]

Yoav I. H. Parish and Pascal Müller. 2001. Procedural modeling of cities. Proceedings of the 28th annual conference on Computer graphics and interactive techniques (SIGGRAPH ’01), Association for Computing Machinery, 301–308. http://doi.org/10.1145/383259.383292

Digital Library

[19]

Steven Schkolne, Michael Pruett, and Peter Schröder. Surface drawing: creating organic 3D shapes with the hand and tangible tools. Proc. CHI ’01, 261–268.

[20]

Kristina Shea, Robert Aish, and Marina Gourtovaia. 2005. Towards integrated performance-driven generative design tools. Automation in Construction 14, 2: 253–264. http://doi.org/10.1016/j.autcon.2004.07.002

[21]

Evgeny Stemasov, Tobias Wagner, Jan Gugenheimer, and Enrico Rukzio. 2020. Mix&Match: Towards Omitting Modelling Through In-Situ Alteration and Remixing of Model Repository Artifacts in Mixed Reality. arXiv:2003.09169 [cs]. http://doi.org/10.1145/3313831.3376839

Digital Library

[22]

Maryam Tohidi, Bill Buxton, Ron Baecker, and Abigail Sellen. 2006. Getting the right design and the design right: Testing many is better than one. Retrieved May 5, 2020 from https://www.microsoft.com/en-us/research/publication/getting-the-design-right-and-the-right-design-testing-many-is-better-than-one/

[23]

Steve Tsang, Ravin Balakrishnan, Karan Singh, and Abhishek Ranjan. 2004. A suggestive interface for image guided 3D sketching. 591–598. http://doi.org/10.1145/985692.985767

Digital Library

[24]

Nurcan Gecer Ulu and Levent Burak Kara. 2015. DMS2015-33: Generative interface structure design for supporting existing objects. Journal of Visual Languages & Computing 31: 171–183. http://doi.org/10.1016/j.jvlc.2015.10.016

Digital Library

[25]

Nobuyuki Umetani, Takeo Igarashi, and Niloy J Mitra. Guided Exploration of Physically Valid Shapes for Furniture Design. 11.

[26]

Danding Wang, Qian Yang, Ashraf Abdul, and Brian Y. Lim. 2019. Designing Theory-Driven User-Centric Explainable AI. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19, ACM Press, 1–15. http://doi.org/10.1145/3290605.3300831

Digital Library

[27]

Michael Yu Wang, Xiaoming Wang, and Dongming Guo. 2003. A level set method for structural topology optimization. Computer Methods in Applied Mechanics and Engineering 192, 1: 227–246. http://doi.org/10.1016/S0045-7825(02)00559-5

[28]

Christian Weichel, Manfred Lau, David Kim, Nicolas Villar, and Hans W. Gellersen. 2014. MixFab: a mixed-reality environment for personal fabrication. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’14), Association for Computing Machinery, 3855–3864. http://doi.org/10.1145/2556288.2557090

Digital Library

[29]

Gerold Wesche and Hans-Peter Seidel. 2001. FreeDrawer: a free-form sketching system on the responsive workbench. Proceedings of the ACM symposium on Virtual reality software and technology (VRST ’01), Association for Computing Machinery, 167–174. http://doi.org/10.1145/505008.505041

Digital Library

[30]

E. Wiese, J. H. Israel, A. Meyer, and S. Bongartz. 2010. Investigating the learnability of immersive free-hand sketching. Proceedings of the Seventh Sketch-Based Interfaces and Modeling Symposium (SBIM ’10), Eurographics Association, 135–142.

[31]

Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. 2016. Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling. In Advances in Neural Information Processing Systems 29, D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon and R. Garnett (eds.). Curran Associates, Inc., 82–90. Retrieved May 5, 2020 from http://papers.nips.cc/paper/6096-learning-a-probabilistic-latent-space-of-object-shapes-via-3d-generative-adversarial-modeling.pdf

[32]

Physics-based Generative Design | Autodesk Research. Retrieved May 5, 2020 from https://www.autodeskresearch.com/publications/gendesign

[33]

ChronoFab: Fabricating Motion | Autodesk Research. Retrieved May 5, 2020 from https://www.autodeskresearch.com/publications/chronofab-fabricating-motion

[34]

Experimental Evaluation of Sketching on Surfaces in VR | Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. Retrieved May 5, 2020 from https://dl.acm.org/doi/10.1145/3025453.3025474

[35]

DreamRooms: Prototyping Rooms in Collaboration with a Generative Process • Graphics Interface. Graphics Interface. Retrieved May 5, 2020 from https://graphicsinterface.org/proceedings/gi2019/gi2019-19/

[36]

DreamSketch: Early Stage 3D Design Explorations with Sketching and Generative Design | Autodesk Research. Retrieved May 5, 2020 from https://www.autodeskresearch.com/publications/dreamsketch-early-stage-3d-design-explorations-sketching-and-generative-design

[37]

SymbiosisSketch | Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Retrieved May 5, 2020 from https://dl.acm.org/doi/10.1145/3173574.3173759

[38]

Generative Design at Airbus | Customer Stories | Autodesk. Retrieved May 5, 2020 from https://www.autodesk.com/customer-stories/airbus

[39]

Creative Computers? Here's What You Get When You Design a Chair Using Algorithms. Futurism. Retrieved May 5, 2020 from https://futurism.com/creative-computers-heres-what-you-get-when-you-design-a-chair-using-algorithms

[40]

Generative Design Applied on Buildings. BIM Toolbox. Retrieved May 5, 2020 from https://autodesk.typepad.com/bimtoolbox/2017/06/generative-design-applied-on-buildings.html

[41]

Dream Lens: Exploration and Visualization of Large-Scale Generative Design Datasets | Autodesk Research. Retrieved May 5, 2020 from https://www.autodeskresearch.com/publications/dreamlens

[42]

A suggestive interface for 3D drawing | Proceedings of the 14th annual ACM symposium on User interface software and technology. Retrieved May 5, 2020 from https://dl.acm.org/doi/10.1145/502348.502379

[43]

Sketching User Experiences: Getting the Design Right and the Right Design - 1st Edition. Retrieved May 5, 2020 from https://www.elsevier.com/books/sketching-user-experiences-getting-the-design-right-and-the-right-design/buxton/978-0-12-374037-3

[44]

TutoriVR: A Video-Based Tutorial System for Design Applications in Virtual Reality. Adobe Research. Retrieved May 5, 2020 from https://research.adobe.com/publication/tutorivr-a-video-based-tutorial-system-for-design-applications-in-virtual-reality

[45]

Tilt Brush by Google. Retrieved May 5, 2020 from https://www.tiltbrush.com/

[46]

Christopher Andrews. 2019. An Embodied Approach to AI Art Collaboration. In Proceedings of the 2019 on Creativity and Cognition (C&C'19). Association for Computing Machinery, New York, NY, USA, 156–162.

Digital Library

[47]

Duri Long, Mikhail Jacob, and Brian Magerko. 2019. Designing Co-Creative AI for Public Spaces. In Proceedings of the 2019 on Creativity and Cognition (C&C'19). Association for Computing Machinery, New York, NY, USA, 271–284.

Digital Library

[48]

Liang Men and Nick Bryan-Kinns. 2019. LeMo: Exploring Virtual Space for Collaborative Creativity. In Proceedings of the 2019 on Creativity and Cognition (C&C'19). Association for Computing Machinery, New York, NY, USA, 71–82.

Digital Library

[49]

Elinda Ai-Lim Lee, Kok Wai Wong. 2014 Learning with desktop virtual reality: Low spatial ability learners are more positively affected. Computers & Education, Volume 79, 2014, Pages 49-58, ISSN 0360-1315 https://doi.org/10.1016/j.compedu.2014.07.010.

Digital Library

[50]

Firas Khatib, Seth Cooper, Michael D. Tyka, Kefan Xu, Ilya Makedon, Zoran Popović, David Baker, Foldit Players. Algorithm discovery by protein folding game players. Proceedings of the National Academy of Sciences Nov 2011, 108 (47) 18949-18953;

Cited By

Kumar DDhalwal RChaudhary A(2024)Exploring the Ethical Implications of Generative AI in HealthcareThe Ethical Frontier of AI and Data Analysis10.4018/979-8-3693-2964-1.ch011(180-195)Online publication date: 12-Apr-2024
https://doi.org/10.4018/979-8-3693-2964-1.ch011
Serbanescu ANack F(2024)Towards an analytical framework for AI-powered creative support systems in interactive digital narrativesJournal of Entrepreneurial Researchers10.29073/jer.v2i1.202:1(097-115)Online publication date: 9-Jul-2024
https://doi.org/10.29073/jer.v2i1.20
Fukaya KDaylamani-Zad DAgius H(2024)Intelligent Generation of Graphical Game Assets: A Conceptual Framework and Systematic Review of the State of the ArtACM Computing Surveys10.1145/3708499Online publication date: 18-Dec-2024
https://doi.org/10.1145/3708499
Show More Cited By

Recommendations

The Creative Proteus Effect: How Self-Similarity, Embodiment, and Priming of Creative Stereotypes with Avatars Influences Creative Ideation
C&C '17: Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition

Creative ideation can be enhanced in 3D virtual environments by manipulating the appearance of a user's avatar so that it primes a creative stereotype. However, not much is known about the factors that influence the effectiveness of using avatars to ...
Flow with the Beat! Human-Centered Design of Virtual Environments for Musical Creativity Support in VR
C&C '22: Proceedings of the 14th Conference on Creativity and Cognition

As previous studies have shown, the environment of creative people can have a significant impact on their creative process and thus on their creations. However, with the advent of digital tools such as virtual instruments and digital audio workstations,...
Creative Virtual Environments: Effect of Supraliminal Priming on Team Brainstorming
HICSS '13: Proceedings of the 2013 46th Hawaii International Conference on System Sciences

Much of human behavior involves non-conscious cognition. Research has shown that it is possible to manipulate non-conscious cognition and behavior through "priming", a well-known phenomenon from cognitive psychology. 3-D Virtual Environments (VEs) have ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

C&C '21: Proceedings of the 13th Conference on Creativity and Cognition

June 2021

581 pages

ISBN:9781450383769

DOI:10.1145/3450741

General Chairs:
Corina Sas
Lancaster University, UK
,
Neil Maiden
City, University of London, UK

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 June 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

C&C '21

Sponsor:

SIGCHI

C&C '21: Creativity and Cognition

June 22 - 23, 2021

Virtual Event, Italy

Acceptance Rates

Overall Acceptance Rate 108 of 371 submissions, 29%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

29
Total Citations
View Citations
1,282
Total Downloads

Downloads (Last 12 months)393
Downloads (Last 6 weeks)14

Reflects downloads up to 25 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kumar DDhalwal RChaudhary A(2024)Exploring the Ethical Implications of Generative AI in HealthcareThe Ethical Frontier of AI and Data Analysis10.4018/979-8-3693-2964-1.ch011(180-195)Online publication date: 12-Apr-2024
https://doi.org/10.4018/979-8-3693-2964-1.ch011
Serbanescu ANack F(2024)Towards an analytical framework for AI-powered creative support systems in interactive digital narrativesJournal of Entrepreneurial Researchers10.29073/jer.v2i1.202:1(097-115)Online publication date: 9-Jul-2024
https://doi.org/10.29073/jer.v2i1.20
Fukaya KDaylamani-Zad DAgius H(2024)Intelligent Generation of Graphical Game Assets: A Conceptual Framework and Systematic Review of the State of the ArtACM Computing Surveys10.1145/3708499Online publication date: 18-Dec-2024
https://doi.org/10.1145/3708499
Drogemuller AMatthews BCunningham AYu RGu NThomas B(2024)Hey Building! Novel Interfaces for Parametric Design Manipulations in Virtual RealityProceedings of the ACM on Human-Computer Interaction10.1145/36981408:ISS(330-355)Online publication date: 24-Oct-2024
https://doi.org/10.1145/3698140
Vachha CKang YDive ZChidambaram AGupta AJun EHartmann B(2024)Dreamcrafter: Immersive Editing of 3D Radiance Fields Through Flexible, Generative Inputs and OutputsAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686328(1-3)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3672539.3686328
Yoshida HItoh S(2024)Modeling by Clipped Furniture Parts: Design with Text-Image Model with Stability UnderstandingProceedings of the 2024 ACM International Conference on Interactive Media Experiences10.1145/3639701.3656315(294-299)Online publication date: 7-Jun-2024
https://dl.acm.org/doi/10.1145/3639701.3656315
Liu PKitson APicard-Deland CCarr MLiu SLc RZhu-Tian C(2024)Virtual Dream Reliving: Exploring Generative AI in Immersive Environment for Dream Re-experiencingExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3644054(1-11)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3644054
Mahdavi Goloujeh ASullivan AMagerko B(2024)Is It AI or Is It Me? Understanding Users’ Prompt Journey with Text-to-Image Generative AI ToolsProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642861(1-13)Online publication date: 11-May-2024
https://doi.org/10.1145/3613904.3642861
Weisz JHe JMuller MHoefer GMiles RGeyer W(2024)Design Principles for Generative AI ApplicationsProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642466(1-22)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642466
Qin HJin SGao ZFan MHui P(2024)CharacterMeet: Supporting Creative Writers' Entire Story Character Construction Processes Through Conversation with LLM-Powered Chatbot AvatarsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642105(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642105
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten