research-article

InteractML: Making machine learning accessible for creative practitioners working with movement interaction in immersive media

Authors:

Clarice Hilton,

Carlos González Díaz,

Bruno Martelli,

Michael Zbyszynski,

Rebecca Fiebrink,

Marco GilliesAuthors Info & Claims

VRST '21: Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology

Article No.: 23, Pages 1 - 10

https://doi.org/10.1145/3489849.3489879

Published: 08 December 2021 Publication History

Abstract

Interactive Machine Learning offers a method for designing movement interaction that supports creators in implementing even complex movement designs in their immersive applications by simply performing them with their bodies. We introduce a new tool, InteractML, and an accompanying ideation method, which makes movement interaction design faster, adaptable and accessible to creators of varying experience and backgrounds, such as artists, dancers and independent game developers. The tool is specifically tailored to non-experts as creators configure and train machine learning models via a node-based graph and VR interface, requiring minimal programming. We aim to democratise machine learning for movement interaction to be used in the development of a range of creative and immersive applications.

Supplementary Material

VTT File (vrst21a-sub2948-cam.vtt)

Download
7.79 KB

MP4 File (vrst21a-sub2948-cam.mp4)

Supplemental video

Download
106.68 MB

References

[1]

Sarah Fdili Alaoui. 2019. Making an Interactive Dance Piece. Proceedings of the 2019 on Designing Interactive Systems Conference. https://doi.org/10.1145/3322276.3322289

Digital Library

[2]

Joel Brandt, Philip J. Guo, Joel Lewenstein, Mira Dontcheva, and Scott R. Klemmer. 2009. Two studies of opportunistic programming. (2009), 1589–1598. https://doi.org/10.1145/1518701.1518944

Digital Library

[3]

V. Braun and V. Clarke. 2006. Using thematic analysis in psychology. 3 (2006), 77–101. https://doi.org/10.1191/1478088706qp063oa

[4]

Scott Andrew Brown, Michael Gratton, and Bronwen Williams. 2019. A Visual Programming Tool for Creative Practice Pedagogy in Embodied Interaction and Media Arts. Proceedings of the 31st Australian Conference on Human-Computer-Interaction. https://doi.org/10.1145/3369457.3369495

Digital Library

[5]

Jamie Bullock and Ali Momeni. 2015. Ml.Lib: Robust, Cross-Platform, Open-Source Machine Learning for Max and Pure Data. In Proceedings of the International Conference on New Interfaces for Musical Expression(Baton Rouge, Louisiana, USA) (NIME 2015). The School of Music and the Center for Computation and Technology (CCT), Louisiana State University, Baton Rouge, Louisiana, USA, 265–270. https://doi.org/10.5555/2993778.2993845

Digital Library

[6]

Baptiste Caramiaux, Alessandro Altavilla, Scott G. Pobiner, and Atau Tanaka. 2015. Form Follows Sound. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. https://doi.org/10.1145/2702123.2702515

Digital Library

[7]

Carlos Gonzalez Diaz, Phoenix Perry, and Rebecca Fiebrink. 2019. Interactive Machine Learning for More Expressive Game Interactions. 2019 IEEE Conference on Games (CoG). https://doi.org/10.1109/CIG.2019.8848007

Digital Library

[8]

Jerry Alan Fails and Dan R. Olsen. 2003. Interactive machine learning. Proceedings of the 8th international conference on Intelligent user interfaces - IUI ’03. https://doi.org/10.1145/604045.604056

Digital Library

[9]

Rebecca Fiebrink, Perry R. Cook, and Dan Trueman. 2011. Human model evaluation in interactive supervised learning. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/1978942.1978965

Digital Library

[10]

Rebecca Fiebrink, Dan Trueman, and Perry. R. Cook. 2009. A Meta-Instrument for Interactive, On-the-fly Machine Learning. New Interfaces for Musical Expression (NIME), 280–285.

[11]

John C. Flanagan. 1954. The critical incident technique.Psychological Bulletin 51 (1954). Issue 4. https://doi.org/10.1037/h0061470

[12]

Nicholas Gillian and Joseph. A. Paradiso. 2014. The gesture recognition toolkit. The Journal of Machine Learning Research 15 (2014), 3483–3487. Issue 1.

Digital Library

[13]

Marco Gillies. 2016. What is Movement Interaction in Virtual Reality for?. In Proceedings of the 3rd International Symposium on Movement and Computing - MOCO ’16. ACM Press, New York, New York, USA, 1–4. https://doi.org/10.1145/2948910.2948951

Digital Library

[14]

Marco Gillies. 2019. Understanding the Role of Interactive Machine Learning in Movement Interaction Design. ACM Transactions on Computer-Human Interaction 26 (2 2019). Issue 1. https://doi.org/10.1145/3287307

Digital Library

[15]

Marco Gillies and Andrea Kleinsmith. 2013. Interactive Machine Learning for Virtual Agents. Intelligent Virtual Agents: 13th International Conference, IVA, 29–31.

[16]

Clarice M Hilton, Carlos Gonzalez, Ruth Gibson, Michael Zbyszyński, Phoenix Perry, Rebecca Fiebrink, Nicola Plant, Bruno Martelli, and Marco Gillies. 2021. InteractML : Node Based Tool to Empower Artists and Dancers in using Interactive Machine Learning for Designing Movement Interaction. (2021), 1–5.

[17]

Caroline Hummels, Kees C. J. Overbeeke, and Sietske Klooster. 2007. Move to get moved: a search for methods, tools and knowledge to design for expressive and rich movement-based interaction. Personal and Ubiquitous Computing 11 (10 2007). Issue 8. https://doi.org/10.1007/s00779-006-0135-y

Digital Library

[18]

Kristina Höök. 2018. Designing with the Body: Somaesthetic Interaction Design. MIT Press.

[19]

Robert Jacob, Audrey Girouard, Leanne Hirshfield, Michael Horn, Orit Shaer, Erin Treacy Solovey, and Jamie Zigelbaum. 2008. Reality-Based Interaction: A Framework for Post-WIMP Interfaces. SIGCHI conference on Human factors in computing systems.

Digital Library

[20]

David Kirsh. 2013. Embodied cognition and the magical future of interaction design. ACM Transactions on Computer-Human Interaction 20 (3 2013). Issue 1. https://doi.org/10.1145/2442106.2442109

Digital Library

[21]

William Latham, Stephen Todd, Peter Todd, and Lance Putnam. 2021. Exhibiting Mutator VR: Procedural Art Evolves to Virtual Reality. Leonardo 54 (6 2021). Issue 3. https://doi.org/10.1162/leon_a_01857

[22]

Lian Loke and Toni Robertson. 2010. Studies of Dancers: Moving from Experience to Interaction Design. International Journal of Design 4 (2010). Issue 2.

[23]

Raul Masu, Nuno N. Correia, Stephan Jurgens, Ivana Druzetic, and William Primett. 2019. How do Dancers Want to Use Interactive Technology?Proceedings of the 9th International Conference on Digital and Interactive Arts. https://doi.org/10.1145/3359852.3359869

Digital Library

[24]

D. A. Norman. 2010. Natural user interfaces are not natural. interactions 17(2010), 6–10. Issue 3.

[25]

Claudia Núñez-Pacheco and Lian Loke. 2018. Towards a technique for articulating aesthetic experiences in design using Focusing and the Felt Sense. The Design Journal 21 (7 2018). Issue 4. https://doi.org/10.1080/14606925.2018.1467680

[26]

Kayur Patel, James Fogarty, James A. Landay, and Beverly Harrison. 2008. Investigating statistical machine learning as a tool for software development. Proceeding of the twenty-sixth annual CHI conference on Human factors in computing systems - CHI ’08. https://doi.org/10.1145/1357054.1357160

Digital Library

[27]

N. Plant, R. Gibson, C.G. Diaz, B. Martelli, M. Zbyszyński, R. Fiebrink, M. Gillies, C. Hilton, and P. Perry. 2020. Movement interaction design for immersive media using interactive machine learning. In ACM International Conference Proceeding Series. https://doi.org/10.1145/3401956.3404252

Digital Library

[28]

Nicola Plant, Clarice Hilton Goldsmiths, Marco Gillies Goldsmiths, Michael Zbyszyński, Rebecca Fiebrink, Phoenix Perry, Carlos Gonzalez Diaz, Ruth Gibson, Studio Gibson, and / Martelli. 2020. Programming by Moving: Interactive Machine Learning for Embodied Interaction Design. Workshop ”The UX of Interactive Machine Learning” at NordiCHI ’20 (2020), 1–4.

[29]

Nicola Plant, Clarice Hilton, Marco Gillies, Rebecca Fiebrink, Phoenix Perry, Carlos González DÍaz, Ruth Gibson, Bruno Martelli, and Michael Zbyszynski. 2021. Interactive Machine Learning for Embodied Interaction Design: A tool and methodology. TEI 2021 - Proceedings of the 15th International Conference on Tangible, Embedded, and Embodied Interaction(2021). https://doi.org/10.1145/3430524.3442703

Digital Library

[30]

Anna Rizzo, Katerina Eh Raheb, Sarah Whatley, Rosa Maria Cisneros, Antonio Camurri, Vladimir Viro, Jean-Marc Mato, Stefano Piana, Michele Buccoli, Amalia Markatzi, Pablo Palacio, Oshri Even, Augusto Sarti, Yannis Ioannidis, and Edwin-Morley Fletcher. 2018. WhoLoDancE: Whole-body Interaction Learning for Dance Education. Workshop on Cultural Informatics, 41–50.

[31]

Elena Márquez Segura, Laia Turmo Vidal, Asreen Rostami, and Annika Waern. 2016. Embodied Sketching. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/2858036.2858486

Digital Library

[32]

Richard Shusterman. 2012. Thinking Through the Body: Essays in Somaesthetics. Cambridge University Press.

[33]

Mel Slater. 2009. Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philosophical Transactions of the Royal Society B: Biological Sciences 364 (12 2009). Issue 1535. https://doi.org/10.1098/rstb.2009.0138

[34]

Robert Wechsler, Peter Dowling, and Frieder Weiß. 2004. EyeCon–A motion sensing tool for creating interactive dance, music, and video projections. SSAISB Convention.

[35]

Danielle Wilde, Anna Vallgårda, and Oscar Tomico. 2017. Embodied Design Ideation Methods. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3025453.3025873

Digital Library

[36]

Michael Zbyszynski, Mick Grierson, and Matthew Yee-king. 2017. Rapid Prototyping of New Instruments with CodeCircle. Proceedings of the International Conference on New Interfaces for Musical Expression (2017), 227–230.

Cited By

Grech AMehnen JWodehouse A(2023)An Extended AI-Experience: Industry 5.0 in Creative Product InnovationSensors10.3390/s2306300923:6(3009)Online publication date: 10-Mar-2023
https://doi.org/10.3390/s23063009
Shi YGao TJiao XCao N(2023)Understanding Design Collaboration Between Designers and Artificial Intelligence: A Systematic Literature ReviewProceedings of the ACM on Human-Computer Interaction10.1145/36102177:CSCW2(1-35)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3610217
Hilton CPan XKoeck RYu H(2023)Simulating Location-Based Experiences in VR2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW58643.2023.00030(119-122)Online publication date: Mar-2023
https://doi.org/10.1109/VRW58643.2023.00030
Show More Cited By

Recommendations

Interactive Machine Learning for Embodied Interaction Design: A tool and methodology
TEI '21: Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction

As immersive technologies are increasingly being adopted by artists, dancers and developers in their creative work, there is a demand for tools and methods to design compelling ways of embodied interaction within virtual environments. Interactive ...
Machine Learning Education for Artists, Musicians, and Other Creative Practitioners
Special Section on ML Education and Regular Articles

This article aims to lay a foundation for the research and practice of machine learning education for creative practitioners. It begins by arguing that it is important to teach machine learning to creative practitioners and to conduct research about ...
Movement interaction design for immersive media using interactive machine learning
MOCO '20: Proceedings of the 7th International Conference on Movement and Computing

Interactive Machine Learning is a promising approach for designing movement interaction because it allows developers to capture complex movements by simply performing them. We introduce a new tool being developed to make embodied interaction design ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

VRST '21: Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology

December 2021

563 pages

ISBN:9781450390927

DOI:10.1145/3489849

Editors:
Yuichi Itoh
Aoyama Gakuin University, Japan
,
Kazuki Takashima
Tohoku University, Japan
,
Parinya Punpongsanon
Osaka University, Japan
,
Misha Sra
University of California, Santa Barbara, USA
,
Kazuyuki Fujita
Tohoku University, Japan
,
Shigeo Yoshida
The University of Tokyo, Japan
,
Tham Piumsomboon
University of Canterbury, New Zealand

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 December 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Engineering and Physical Sciences Research Council, UK Research and Innovation

Conference

VRST '21

Sponsor:

VRST '21: 27th ACM Symposium on Virtual Reality Software and Technology

December 8 - 10, 2021

Osaka, Japan

Acceptance Rates

Overall Acceptance Rate 66 of 254 submissions, 26%

Upcoming Conference

VRST '24

Sponsor:
sigchi
sigchi

30th ACM Symposium on Virtual Reality Software and Technology

October 9 - 11, 2024

Trier , Germany

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
309
Total Downloads

Downloads (Last 12 months)84
Downloads (Last 6 weeks)7

Reflects downloads up to 18 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Grech AMehnen JWodehouse A(2023)An Extended AI-Experience: Industry 5.0 in Creative Product InnovationSensors10.3390/s2306300923:6(3009)Online publication date: 10-Mar-2023
https://doi.org/10.3390/s23063009
Shi YGao TJiao XCao N(2023)Understanding Design Collaboration Between Designers and Artificial Intelligence: A Systematic Literature ReviewProceedings of the ACM on Human-Computer Interaction10.1145/36102177:CSCW2(1-35)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3610217
Hilton CPan XKoeck RYu H(2023)Simulating Location-Based Experiences in VR2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW58643.2023.00030(119-122)Online publication date: Mar-2023
https://doi.org/10.1109/VRW58643.2023.00030
Escamilla AMelenchón JMonzo CMoran J(2023)User Clustering Visualization and Its Impact on Motion-Based Interaction DesignHuman-Computer Interaction10.1007/978-3-031-35596-7_4(47-63)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1007/978-3-031-35596-7_4

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents