Spatial Orchestra: Locomotion Music Instruments through Spatial Exploration
Authors: 
You-Jin Kim, 
Myungin Lee, Marko Peljhan, JoAnn Kuchera-Morin, Tobias HöllererAuthors Info & Claims
You-Jin Kim, Myungin Lee, Marko Peljhan, JoAnn Kuchera-Morin, Tobias HöllererAuthors Info & ClaimsArticle No.: 420, Pages 1 - 5
Abstract
Spatial Orchestra demonstrates how easy it is to play musical instruments using basic input like natural locomotion, which is accessible to most. Unlike many musical instruments, our work allows individuals of all skill levels to effortlessly create music by walking into virtual bubbles. Our Augmented Reality experience involves interacting with ever-shifting sound bubbles that the user engages with by stepping into color-coded bubbles within the assigned area using a standalone AR headset. Each bubble corresponds to a cello note, and omits sound from the center of the bubble, and lets the user hear and express in spatial audio, effectively transforming participants into musicians. This interactive element enables users to explore the intersection of spatial awareness, musical rhythm that extends to bodily expression through playful movements and dance-like gestures within the bubble-filled environment. This unique experience illuminates the intricate relationship between spatial awareness and the art of musical performance.
Supplemental Material
- Download
- 119.25 MB
- Transcript
- Download
- 24.29 MB
- Transcript
References
[1]
Sam Bilbow. 2022. Evaluating polaris - An Audiovisual Augmented Reality Experience Built on Open-Source Hardware and Software. New Interfaces for Musical Expression. https://doi.org/10.21428/92fbeb44.8abb9ce6
[2]
Alex Buck, Radford Parker, and Matt Wojciakowski. 2022. Azure Spatial Anchors overview. https://learn.microsoft.com/en-us/azure/spatial-anchors/overview. Accessed: October 2022.
[3]
Baptiste Caramiaux, Alessandro Altavilla, Scott G. Pobiner, and Atau Tanaka. 2015. Form Follows Sound: Designing Interactions from Sonic Memories. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 3943–3952. https://doi.org/10.1145/2702123.2702515
[4]
A.D. Cheok, Wang Weihua, Xubo Yang, S. Prince, Fong Siew Wan, M. Billinghurst, and H. Kato. 2002. Interactive theatre experience in embodied + wearable mixed reality space. In Proceedings. International Symposium on Mixed and Augmented Reality. 59–317. https://doi.org/10.1109/ISMAR.2002.1115073
[5]
Mário Escarce Junior, Georgia Rossmann Martins, Leandro Soriano Marcolino, and Elisa Rubegni. 2021. A Meta-Interactive Compositional Approach That Fosters Musical Emergence through Ludic Expressivity, Vol. 5. Association for Computing Machinery, New York, NY, USA, 32. https://doi.org/10.1145/3474689
[6]
Rolf Gehlhaar. 1991. SOUND = SPACE: an interactive musical environment. Contemporary Music Review 6, 1. https://doi.org/10.1080/07494469100640061
[7]
Balazs Andras Ivanyi, Truls Bendik Tjemsland, Christian Vasileios Tsalidis de Zabala, Lilla Julia Toth, Marcus Alexander Dyrholm, Scott James Naylor, Ann Paradiso, Dwayne Lamb, Jarnail Chudge, Ali Adjorlu, and Stefania Serafin. 2023. DuoRhythmo: Design and Remote User Experience Evaluation (UXE) of a Collaborative Accessible Digital Musical Interface (CADMI) for People with ALS (PALS). In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 56. https://doi.org/10.1145/3544548.3581285
[8]
You-Jin Kim, Radha Kumaran, Ehsan Sayyad, Anne Milner, Tom Bullock, Barry Giesbrecht, and Tobias Hollerer. 2022. Investigating Search Among Physical and Virtual Objects Under Different Lighting Conditions. IEEE Transactions on Visualization and Computer Graphics (2022), 1–11. https://doi.org/10.1109/TVCG.2022.3203093
[9]
You-Jin Kim, Joshua Lu, and Tobias Höllerer. 2023. Dynamic Theater: Location-Based Immersive Dance Theater, Investigating User Guidance and Experience. In Proceedings of the 29th ACM Symposium on Virtual Reality Software and Technology (Christchurch, New Zealand) (VRST ’23). Association for Computing Machinery, New York, NY, USA, Article 27. https://doi.org/10.1145/3611659.3615705
[10]
You-Jin Kim, Andrew D. Wilson, Jennifer Jacobs, and Tobias Höllerer. 2023. Reality Distortion Room: A Study of User Locomotion Responses to Spatial Augmented Reality Effects. In 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 1201–1210. https://doi.org/10.1109/ISMAR59233.2023.00137
[11]
Peter Kurucz, Nilufar Baghaei, Stefania Serafin, and Eve Klein. 2023. Enhancing Auditory Immersion in Interactive Virtual Reality Environments. In 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). 789–792. https://doi.org/10.1109/ISMAR-Adjunct60411.2023.00174
[12]
Myungin Lee. 2021. Entangled: A Multi-Modal, Multi-User Interactive Instrument in Virtual 3D Space Using the Smartphone for Gesture Control. New Interfaces for Musical Expression. https://doi.org/10.21428/92fbeb44.eae7c23f
[13]
Pieter-Jan Maes. 2016. Sensorimotor grounding of musical embodiment and the role of prediction: A review. Frontiers in psychology 7 (2016), 308. https://doi.org/10.3389/fpsyg.2016.00308
[14]
Pieter-Jan Maes, Marc Leman, Caroline Palmer, and Marcelo M Wanderley. 2014. Action-based effects on music perception. Frontiers in psychology 4 (2014), 1008. https://doi.org/10.3389/fpsyg.2013.01008
[15]
Fabio Morreale, Antonella De Angeli, Raul Masu, Paolo Rota, and Nicola Conci. 2014. Collaborative creativity: The Music Room. Personal and Ubiquitous Computing 18, 1187–1199. https://doi.org/10.1007/s00779-013-0728-1
[16]
Nicolas Rasamimanana, Frédéric Bevilacqua, Julien Bloit, Norbert Schnell, Emmanuel Fléty, Andrea Cera, Uros Petrevski, and Jean-Louis Frechin. 2012. The Urban Musical Game: Using Sport Balls as Musical Interfaces. CHI ’12 Extended Abstracts, 1027–1030. https://doi.org/10.1145/2212776.2212377
[17]
Anders Riddershom Bargum, Oddur Ingi Kristjánsson, Péter Babó, Rasmus Eske Waage Nielsen, Simon Rostami Mosen, and Stefania Serafin. 2023. Spatial Audio Mixing in Virtual Reality. Springer International Publishing, Cham, 269–302. https://doi.org/10.1007/978-3-031-04021-4_9
[18]
Ruben Schlagowski, Dariia Nazarenko, Yekta Can, Kunal Gupta, Silvan Mertes, Mark Billinghurst, and Elisabeth André. 2023. Wish You Were Here: Mental and Physiological Effects of Remote Music Collaboration in Mixed Reality. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 102. https://doi.org/10.1145/3544548.3581162
[19]
Stefania Serafin and Ali Adjorlu. 2023. Earnormous: An Educational VR Game about How Humans Hear. In Proceedings of the 29th ACM Symposium on Virtual Reality Software and Technology (Christchurch, New Zealand) (VRST ’23). Association for Computing Machinery, New York, NY, USA, Article 52. https://doi.org/10.1145/3611659.3616887
[20]
Luca Turchet, Rob Hamilton Hamilton, and Anil Çamci. 2021. Music in Extended Realities. IEEE Access 9, 15810–15832. https://doi.org/10.1109/ACCESS.2021.3052931
[21]
Yichen Wang and Charles Martin. 2022. Cubing Sound: Designing a NIME for Head-mounted Augmented Reality. New Interfaces for Musical Expression. https://doi.org/10.21428/92fbeb44.b540aa59
[22]
Robert Winter. 2006. Interactive music: Compositional techniques for communicating different emotional qualities. Master’s dissertation, University of York (2006).
Index Terms
- Spatial Orchestra: Locomotion Music Instruments through Spatial Exploration
Recommendations
Transition of instruments in The SINE WAVE ORCHESTRA
SPECIAL ISSUE: Media Arts (Part II)This article discusses the transition of instruments in The SINE WAVE ORCHESTRA (SWO), which is a participatory sound performance project. Its basic concept is that each participant plays a sine wave, to collectively create a sound representation. ...
A Spatial Music Listening Experience in Augmented Reality
UIST '21 Adjunct: Adjunct Proceedings of the 34th Annual ACM Symposium on User Interface Software and TechnologyLive music provides a more immersive and social experience that recorded music cannot replicate. In a live music setting, listeners perceive sounds differently based on their position with respect to the musicians and can enjoy the experience with ...
Pitch-related identification of instruments in classical music recordings
NFMCP'14: Proceedings of the 3rd International Conference on New Frontiers in Mining Complex PatternsIdentification of particular voices in polyphonic and polytimbral music is a task often performed by musicians in their everyday life. However, the automation of this task is very challenging, because of high complexity of audio data. Usually additional ...
Comments
Information & Contributors
Information
Published In
May 2024
4761 pages
ISBN:9798400703317
DOI:10.1145/3613905
Copyright © 2024 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 May 2024
Check for updates
Author Tags
Qualifiers
- Demonstration
- Research
- Refereed limited
Funding Sources
Conference
Acceptance Rates
Overall Acceptance Rate 6,164 of 23,696 submissions, 26%
Upcoming Conference
CHI 2025
- Sponsor:
- sigchi
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 402Total Downloads
- Downloads (Last 12 months)402
- Downloads (Last 6 weeks)90
Reflects downloads up to 01 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format