DualStage: Enhancing Emotional Connections through Music Visualization for Synchronizing Live and Virtual Performances
Pages 771 - 775
Abstract
The pandemic has accelerated the adoption of virtual concerts, bridging the physical and virtual worlds for performers and audiences. However, the emotional connection of live concerts remains irreplaceable. To address this issue, we propose DualStage, a real-time emotional visualization system for synchronizing live and virtual performances. The system will visualize the collected data, encompassing various aspects such as the performers' movements, the audience's interactions, and the tempo, rhythm, and intensity of the music. By integrating these elements, the system aims to create a comprehensive and immersive visual experience that enhances the emotional connection between performers and both onsite and online audiences. We conducted an exploratory study with 22 participants to refine the system and enhance cross-reality engagement. The results showed that most participants recognized the feasibility and effectiveness of the system.
References
[1]
Valentin Bazarevsky, Ivan Grishchenko, Karthik Raveendran, Tyler Zhu, Fan Zhang, and Matthias Grundmann. 2020. Blazepose: On-device real-time body pose tracking. arXiv preprint arXiv:2006.10204 (2020).
[2]
Alejandro Beacco, Ramon Oliva, Carlos Cabreira, Jaime Gallego, and Mel Slater. 2021. Disturbance and plausibility in a virtual rock concert: A pilot study. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR). IEEE, 538--545.
[3]
Steven Caldwell Brown and Don Knox. 2017. Why go to pop concerts? The motivations behind live music attendance. Musicae Scientiae 21, 3 (2017), 233 249.
[4]
Ryota Horie, Minami Wada, and Eri Watanabe. 2018. Participation in a virtual reality concert via brainwave and heartbeat. In Advances in Affective and Pleasur able Design: Proceedings of the AHFE 2017 International Conference on Affective and Pleasurable Design, July 17--21, 2017, The Westin Bonaventure Hotel, Los Angeles, California, USA 8. Springer, 276--284.
[5]
Takashi Kasuya, Manabu Tsukada, Yu Komohara, Shigeki Takasaka, Takuhiro Mizuno, Yoshitaka Nomura, Yuta Ueda, and Hiroshi Esaki. 2019. LiVRation: Remote VR live platform with interactive 3D audio-visual service. In 2019 IEEE Games, Entertainment, Media Conference (GEM). IEEE, 1--7.
[6]
Alexandra Lamont. 2011. University students? strong experiences of music: Pleasure, engagement, and meaning. Musicae Scientiae 15, 2 (2011), 229--249.
[7]
Ke-Fan Lin, Yu-Chih Chou, Yu-Hsiang Weng, Yvone Tsai Chen, Zin-Yin Lim, Chi-Po Lin, Ping-Hsuan Han, and Tse-Yu Pan. 2023. Actualities: Seamless Live Performance with the Physical and Virtual Audiences in Multiverse. In ACM SIGGRAPH 2023 Immersive Pavilion. 1--2.
[8]
Root Motion. 2024. Final IK. http://root-motion.com/. Unity Asset.
[9]
Kelsey E Onderdijk, Lies Bouckaert, Edith Van Dyck, and Pieter-Jan Maes. 2023. Concert experiences in virtual reality environments. Virtual Reality 27, 3 (2023), 2383--2396.
[10]
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition. 779--788.
[11]
Luca Turchet, Rob Hamilton, and Anil Çamci. 2021. Music in extended realities. IEEE Access 9 (2021), 15810--15832.
[12]
Roel Vergauwen. 2021. Worden games de nieuwe concerttempels.
[13]
Monu Verma, Santosh Kumar Vipparthi, and Girdhari Singh. 2020. Affectivenet: Affective-motion feature learning for microexpression recognition. IEEE Multi Media 28, 1 (2020), 17--27.
[14]
Hiromu Yakura and Masataka Goto. 2020. Enhancing participation experience in vr live concerts by improving motions of virtual audience avatars. In 2020 IEEE international symposium on mixed and augmented reality (ISMAR). IEEE, 555--565.
[15]
Shuo Yan, Xuning Yan, and Xukun Shen. 2020. Exploring social interactions for live performance in virtual reality. In SIGGRAPH Asia 2020 Posters. 1--2.
[16]
Jing Yang, Yue Bai, and Jundong Cho. 2017. Smart light stick: an interactive system for pop concert. In 2017 5th International Conference on Mechatronics, Materials, Chemistry and Computer Engineering (ICMMCCE 2017). Atlantis Press, 596--600
Index Terms
- DualStage: Enhancing Emotional Connections through Music Visualization for Synchronizing Live and Virtual Performances
Recommendations
A Survey of Music Visualization Techniques
Music Information Research (MIR) comprises all the research topics involved in modeling and understanding music. Visualizations are frequently adopted to convey better understandings about music pieces, and the association of music with visual elements ...
TelEmoScatter: Enabling Remote Interaction and Emotional Connections in Virtual and Physical Music Performance
MMAsia '23: Proceedings of the 5th ACM International Conference on Multimedia in AsiaTo enrich the emotional experiences of virtual reality (VR) online audiences in music performances, we developed TelEmoScatter, a system that facilitates remote interaction between music performers and onsite audiences. Our system also fosters emotional ...
Exploring music rankings with interactive visualization
SAC '17: Proceedings of the Symposium on Applied ComputingMusic rankings are mainly aimed at marketing purposes but also help users in discovering new music as well as comparing songs, artists, albums, etc. This work presents an interactive way to visualize, find and compare music rankings using different ...
Comments
Information & Contributors
Information
Published In
October 2024
1032 pages
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 05 October 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
UbiComp '24: The 2024 ACM International Joint Conference on Pervasive and Ubiquitous Computing
October 5 - 9, 2024
Melbourne VIC, Australia
Acceptance Rates
Overall Acceptance Rate 764 of 2,912 submissions, 26%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 77Total Downloads
- Downloads (Last 12 months)77
- Downloads (Last 6 weeks)16
Reflects downloads up to 08 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in