research-article

Intelligent Agents and Networked Buttons Improve Free-Improvised Ensemble Music-Making on Touch-Screens

Authors:

Charles Martin,

Michael MartinAuthors Info & Claims

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

Pages 2295 - 2306

https://doi.org/10.1145/2858036.2858269

Published: 07 May 2016 Publication History

Abstract

We present the results of two controlled studies of free-improvised ensemble music-making on touch-screens. In our system, updates to an interface of harmonically-selected pitches are broadcast to every touch-screen in response to either a performer pressing a GUI button, or to interventions from an intelligent agent. In our first study, analysis of survey results and performance data indicated significant effects of the button on performer preference, but of the agent on performance length. In the second follow-up study, a mixed-initiative interface, where the presence of the button was interlaced with agent interventions, was developed to leverage both approaches. Comparison of this mixed-initiative interface with the always-on button-plus-agent condition of the first study demonstrated significant preferences for the former. The different approaches were found to shape the creative interactions that take place. Overall, this research offers evidence that an intelligent agent and a networked GUI both improve aspects of improvised ensemble music-making.

Supplementary Material

suppl.mov (pn1242-file3.mp4)

Supplemental video

Download
91.83 MB

MP4 File (p2295-martin.mp4)

Download
275.76 MB

References

[1]

Derek Bailey. 1993. Improvisation: It's Nature and Practice in Music. Da Capo, Cambridge, MA.

[2]

David Borgo. 2005. Sync or Swarm: Improvising Music in a Complex Age. Continuum, NY, NY, USA.

[3]

James V. Bradley. 1958. Complete Counterbalancing of Immediate Sequential Effects in a Latin Square Design. J. Amer. Statist. Assoc. 53, 282 (1958), 525--528.

[4]

Nick Bryan-Kinns. 2004. Daisyphone: The Design and Impact of a Novel Environment for Remote Group Music Improvisation. In Proceedings of the 5th Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques (DIS '04). ACM, NY, NY, USA, 135--144.

Digital Library

[5]

Nick Bryan-Kinns and Fraser Hamilton. 2012. Identifying Mutual Engagement. Behaviour & Information Technology 31, 2 (2012), 101--125.

[6]

William L. Cahn. 2005. Creative Music Making. Routledge, NY, NY.

[7]

Arne Eigenfeldt and Ajay Kapur. 2008. An Agent-based System for Robotic Musical Performance. In Proceedings of the International Conference on New Interfaces for Musical Expression. Genoa, Italy, 144--149. http://www.nime.org/proceedings/2008/nime2008_144.pdf

[8]

Jacob Eisenberg and William Forde Thompson. 2003. A Matter of Taste: Evaluating Improvised Music. Creativity Research Journal 15, 2--3 (2003), 287--296.

[9]

Rebecca Fiebrink, Perry R. Cook, and Dan Trueman. 2011. Human Model Evaluation in Interactive Supervised Learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, NY, NY, USA, 147--156.

Digital Library

[10]

Lalya Gaye, Lars Erik Holmquist, Frauke Behrendt, and Atau Tanaka. 2006. Mobile Music Technology: Report on an Emerging Community. In Proceedings of the 2006 Conference on New Interfaces for Musical Expression (NIME '06). IRCAM & Centre Pompidou, Paris, France, 22--25. http://dl.acm.org/citation.cfm?id=1142215.1142219

Digital Library

[11]

Michael Gurevich, Adnan Marquez-Borbon, and Paul Stapleton. 2012. Playing with Constraints: Stylistic Variation with a Simple Electronic Instrument. Computer Music Journal 36, 1 (2012), 23--41.

Digital Library

[12]

Eric Horvitz. 1999. Principles of Mixed-Initiative User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '99). ACM, NY, NY, USA, 159--166.

Digital Library

[13]

Mark Jenkins. 2012. iPad Music: In the Studio and on Stage. Taylor & Francis, Burlington, MA, USA.

[14]

Sergi Jordà. 2010. The Reactable: Tangible and Tabletop Music Performance. In CHI '10 Extended Abstracts on Human Factors in Computing Systems (CHI EA '10). ACM, NY, NY, USA, 2989--2994.

Digital Library

[15]

Matthew Kay and Jacob O. Wobbrock. 2015. ARTool: Aligned Rank Transform for Nonparametric Factorial ANOVAs. R package version 0.9.5. Available online at https://github.com/mjskay/ARTool. (2015).

[16]

Korg Inc. 2011. Korg WIST SDK: Wireless Sync-Start Technology for iOS Music Apps. https://code.google.com/p/korg-wist-sdk/. (2011).

[17]

Sang Won Lee, Jason Freeman, and Andrew Collela. 2012. Real-Time Music Notation, Collaborative Improvisation, and Laptop Ensembles. In Proceedings of the International Conference on New Interfaces for Musical Expression. University of MI, Ann Arbor, MI. http://www.nime.org/proceedings/2012/nime2012_62.pdf

[18]

Aengus Martin, Craig T. Jin, and Oliver Bown. 2011. A Toolkit for Designing Interactive Musical Agents. In Proceedings of the 23rd Australian Computer-Human Interaction Conference (OzCHI '11). ACM, NY, NY, USA, 194--197.

Digital Library

[19]

Charles Martin, Henry Gardner, and Ben Swift. 2014. Exploring Percussive Gesture on iPads with Ensemble Metatone. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, NY, NY, USA, 1025--1028.

Digital Library

[20]

Charles Martin, Henry Gardner, and Ben Swift. 2015a. Tracking Ensemble Performance on Touch-Screens with Gesture Classification and Transition Matrices. In Proceedings of the International Conference on New Interfaces for Musical Expression, Edgar Berdahl and Jesse Allison (Eds.). Louisiana State University, Baton Rouge, Louisiana, USA, 359--364. http://www.nime.org/proceedings/2015/nime2015_242.pdf

Digital Library

[21]

Charles Martin, Henry Gardner, Ben Swift, and Michael Martin. 2015b. Music of 18 Performances: Evaluating Apps and Agents with Free Improvisation. In Proceedings of the 2015 Conference of the Australasian Computer Music Association, Ian Stevenson (Ed.). Australasian Computer Music Association, Fitzroy, Australia, 85--94. http://hdl.handle.net/1885/95205

[22]

Guerino Mazzola and Paul B. Cherlin. 2009. Flow, Gesture, and Spaces in Free Jazz. Springer, Berlin, Germany.

Digital Library

[23]

MIDI Manufacturers Association. 1996. The Complete MIDI 1.0 Detailed Specification: Incorporating All Recommended Practices. MIDI Manufacturers Association, La Habra, California, USA.

[24]

Jieun Oh, Jorge Herrera, Nicholas J. Bryan, Luke Dahl, and Ge Wang. 2010. Evolving the Mobile Phone Orchestra. In Proceedings of the International Conference on New Interfaces for Musical Expression. University of Technology Sydney, Sydney, Australia, 82--87. http://www.nime.org/proceedings/2010/nime2010_082.pdf

[25]

Sile O'Modhrain. 2011. A Framework for the Evaluation of Digital Musical Instruments. Computer Music Journal 35, 1 (2011), 28--42.

Digital Library

[26]

Ivan Poupyrev, Michael J. Lyons, Sidney Fels, and Tina Blaine (Bean). 2001. New Interfaces for Musical Expression. In CHI '01 Extended Abstracts on Human Factors in Computing Systems (CHI EA '01). ACM, NY, NY, USA, 491--492.

Digital Library

[27]

Jeff Pressing. 1988. Improvisation: Methods and Models. In Generative Processes in Music, John Sloboda (Ed.). Oxford University Press, Oxford, UK.

[28]

Jeff Pressing. 1990. Cybernetic Issues in Interactive Performance Systems. Computer Music Journal 14, 1 (1990), pp. 12--25.

[29]

Muhammad Hafiz Wan Rosli, Karl Yerkes, Matthew Wright, Timothy Wood, Hannah Wolfe, Charlie Roberts, Anis Haron, and Fernando Rincon Estrada. 2015. Ensemble Feedback Instruments. In Proceedings of the International Conference on New Interfaces for Musical Expression, Edgar Berdahl and Jesse Allison (Eds.). Louisiana State University, Baton Rouge, Louisiana, USA, 144--149. http://www.nime.org/proceedings/2015/nime2015_329.pdf

Digital Library

[30]

R. Keith Sawyer. 2006. Group Creativity: Musical Performance and Collaboration. Psychology of Music 34, 2 (2006), 148--165.

[31]

Ben Shneiderman. 2007. Creativity Support Tools: Accelerating Discovery and Innovation. Commun. ACM 50, 12 (Dec. 2007), 20--32.

Digital Library

[32]

Ben Shneiderman and Pattie Maes. 1997. Direct Manipulation vs. Interface Agents. Interactions 4, 6 (Nov. 1997), 42--61.

Digital Library

[33]

Scott Smallwood, Dan Trueman, Perry R. Cook, and Ge Wang. 2008. Composing for Laptop Orchestra. Computer Music Journal 32, 1 (2008), pp. 9--25.

Digital Library

[34]

Derek T. Smith. 2009. Development and Validation of a Rating Scale for Wind Jazz Improvisation Performance. Journal of Research in Music Education 57, 3 (2009), 217--235.

[35]

Harald Stenström. 2009. Free Ensemble Improvisation. Number 13 in ArtMonitor. Konstnärliga fakultetskansliet, University of Gothenburg, Gothenburg, Sweden. http://hdl.handle.net/2077/20293

[36]

Dan Stowell, Andrew Robertson, Nick Bryan-Kinns, and Mark D. Plumbley. 2009. Evaluation of Live Human-computer Music-Making: Quantitative and Qualitative Approaches. Int. J. Hum.-Comput. Stud. 67, 11 (Nov. 2009), 960--975.

Digital Library

[37]

Atau Tanaka. 2010. Mapping Out Instruments, Affordances, and Mobiles. In Proceedings of the International Conference on New Interfaces for Musical Expression, Kirsty Beilharz, Andrew Johnston, Sam Ferguson, and Amy Yi-Chun Chen (Eds.). University of Technology Sydney, Sydney, Australia, 88--93. http://www.nime.org/proceedings/2010/nime2010_088.pdf

[38]

Dan Trueman. 2007. Why a Laptop Orchestra? Organised Sound 12 (2007), 171--179. Issue 2.

Digital Library

[39]

Sebastian Trump and Jamie Bullock. 2014. Orphion: A Gestural Multi-Touch Instrument for the iPad. In Proceedings of the International Conference on New Interfaces for Musical Expression. Goldsmiths, University of London, London, United Kingdom, 159--162. http://www.nime.org/proceedings/2014/nime2014_277.pdf

[40]

Marcelo Mortensen Wanderley and Nicola Orio. 2002. Evaluation of Input Devices for Musical Expression: Borrowing Tools from HCI. Computer Music Journal 26, 3 (2002), 62--76.

Digital Library

[41]

Ge Wang. 2014. Ocarina: Designing the iPhone's Magic Flute. Computer Music Journal 38, 2 (2014), 8--21.

Digital Library

[42]

Ge Wang, Georg Essl, and Henri Penttinen. 2008. Do Mobile Phones Dream of Electric Orchestras?. In Routes/Roots: Proceedings of the International Computer Music Conference. MPublishing, University of MI Library, Ann Arbor, MI. http://hdl.handle.net/2027/spo.bbp2372.2008.039

[43]

Ge Wang, Jieun Oh, and Tom Lieber. 2011. Designing for the iPad : Magic Fiddle. In Proceedings of the International Conference on New Interfaces for Musical Expression. Oslo, Norway, 197--202. http://www.nime.org/proceedings/2011/nime2011_197.pdf

[44]

David A. Williams. 2014. Another Perspective: The iPad Is a REAL Musical Instrument. Music Educators Journal 101, 1 (2014), 93--98.

[45]

Jacob O. Wobbrock, Leah Findlater, Darren Gergle, and James J. Higgins. 2011. The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only ANOVA Procedures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, NY, NY, USA, 143--146.

Digital Library

[46]

Anna Xambó, Eva Hornecker, Paul Marshall, Sergi Jordà, Chris Dobbyn, and Robin Laney. 2013. Let's Jam the Reactable: Peer Learning During Musical Improvisation with a Tabletop Tangible Interface. ACM Trans. Comput.-Hum. Interact. 20, 6, Article 36 (Dec. 2013), 34 pages.

Digital Library

Cited By

Kopecka HSuch JLuck M(2024)Preferences for AI Explanations Based on Cognitive Style and Socio-Cultural FactorsProceedings of the ACM on Human-Computer Interaction10.1145/36373868:CSCW1(1-32)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637386
Hagen MCruzes DJaccheri LFails J(2024)Evaluating digital creativity support for childrenInternational Journal of Child-Computer Interaction10.1016/j.ijcci.2023.10060338:COnline publication date: 27-Feb-2024
https://dl.acm.org/doi/10.1016/j.ijcci.2023.100603
Rezwana JMaher M(2023)Designing Creative AI Partners with COFI: A Framework for Modeling Interaction in Human-AI Co-Creative SystemsACM Transactions on Computer-Human Interaction10.1145/351902630:5(1-28)Online publication date: 23-Sep-2023
https://dl.acm.org/doi/10.1145/3519026
Show More Cited By

Index Terms

Intelligent Agents and Networked Buttons Improve Free-Improvised Ensemble Music-Making on Touch-Screens
1. Applied computing
  1. Arts and humanities
    1. Sound and music computing
2. Information systems
  1. Information retrieval
    1. Specialized information retrieval
      1. Multimedia and multimodal retrieval
        Music retrieval

Recommendations

Deep Models for Ensemble Touch-Screen Improvisation
AM '17: Proceedings of the 12th International Audio Mostly Conference on Augmented and Participatory Sound and Music Experiences

For many, the pursuit and enjoyment of musical performance goes hand-in-hand with collaborative creativity, whether in a choir, jazz combo, orchestra, or rock band. However, few musical interfaces use the affordances of computers to create or enhance ...
Touch screen ensemble music: collaborative interaction for older people with dementia
OzCHI '13: Proceedings of the 25th Australian Computer-Human Interaction Conference: Augmentation, Application, Innovation, Collaboration

This paper presents new touch-screen collaborative interaction models for people with dementia. The authors argue that dementia technology has yet to focus on group musical interactions. The project aims to contribute to dementia care while addressing a ...
Hands on music: physical approach to interaction with digital music
MobileHCI '09: Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services

Mobile users listen to large digital music libraries with thousands of songs. Browsing such libraries in mobile contexts is difficult due to constraints of the context and devices. We explore the usage of physical interaction with digital music to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

May 2016

6108 pages

ISBN:9781450333627

DOI:10.1145/2858036

General Chairs:
Jofish Kaye
Yahoo
,
Allison Druin
University of Maryland / National Park Service
,
Program Chairs:
Cliff Lampe
University of Michigan
,
Dan Morris
Microsoft
,
Juan Pablo Hourcade
University of Iowa

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI'16

Sponsor:

SIGCHI

CHI'16: CHI Conference on Human Factors in Computing Systems

May 7 - 12, 2016

California, San Jose, USA

Acceptance Rates

CHI '16 Paper Acceptance Rate 565 of 2,435 submissions, 23%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
410
Total Downloads

Downloads (Last 12 months)33
Downloads (Last 6 weeks)5

Reflects downloads up to 25 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kopecka HSuch JLuck M(2024)Preferences for AI Explanations Based on Cognitive Style and Socio-Cultural FactorsProceedings of the ACM on Human-Computer Interaction10.1145/36373868:CSCW1(1-32)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637386
Hagen MCruzes DJaccheri LFails J(2024)Evaluating digital creativity support for childrenInternational Journal of Child-Computer Interaction10.1016/j.ijcci.2023.10060338:COnline publication date: 27-Feb-2024
https://dl.acm.org/doi/10.1016/j.ijcci.2023.100603
Rezwana JMaher M(2023)Designing Creative AI Partners with COFI: A Framework for Modeling Interaction in Human-AI Co-Creative SystemsACM Transactions on Computer-Human Interaction10.1145/351902630:5(1-28)Online publication date: 23-Sep-2023
https://dl.acm.org/doi/10.1145/3519026
Zhou YKoyama YGoto MIgarashi THammond TVerbert KParra DKnijnenburg BO'Donovan JTeale P(2021)Interactive Exploration-Exploitation Balancing for Generative Melody CompositionProceedings of the 26th International Conference on Intelligent User Interfaces10.1145/3397481.3450663(43-47)Online publication date: 14-Apr-2021
https://dl.acm.org/doi/10.1145/3397481.3450663
Martin CTorresen J(2020)Data-Driven Analysis of Tiny Touchscreen Performance with MicroJamComputer Music Journal10.1162/comj_a_0053643:4(41-57)Online publication date: 1-Oct-2020
https://dl.acm.org/doi/10.1162/comj_a_00536
Remy CMacDonald Vermeulen LFrich JBiskjaer MDalsgaard PWakkary RAndersen KOdom WDesjardins APetersen M(2020)Evaluating Creativity Support Tools in HCI ResearchProceedings of the 2020 ACM Designing Interactive Systems Conference10.1145/3357236.3395474(457-476)Online publication date: 3-Jul-2020
https://dl.acm.org/doi/10.1145/3357236.3395474
Attanayake USwift BGardner HSorensen A(2020)Disruption and creativity in live coding2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)10.1109/VL/HCC50065.2020.9127204(1-5)Online publication date: Aug-2020
https://doi.org/10.1109/VL/HCC50065.2020.9127204
Frich JMacDonald Vermeulen LRemy CBiskjaer MDalsgaard PBrewster SFitzpatrick GCox AKostakos V(2019)Mapping the Landscape of Creativity Support Tools in HCIProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300619(1-18)Online publication date: 2-May-2019
https://dl.acm.org/doi/10.1145/3290605.3300619
Martin CGardner H(2019)Free-Improvised Rehearsal-as-Research for Musical HCINew Directions in Music and Human-Computer Interaction10.1007/978-3-319-92069-6_17(269-284)Online publication date: 7-Feb-2019
https://doi.org/10.1007/978-3-319-92069-6_17

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten