Virtual Reality Dance Tracks from Skeletal Animations
Pages 248 - 253
Abstract
This paper presents a novel approach for automatically generating Virtual Reality (VR) dance tracks, focusing on translating the movement of animated 3D models directly derived from music. Our method capitalizes on the recent advances in automated synthesis of animated 3D models from music, using this data to bridge the gap to fully automatic VR dance track creation. We introduce a novel plugin for the Unity game engine, facilitating the conversion of dance animations from the Choreomaster dataset into dance tracks for the VR game Synth Riders. This approach aims to guide dance while supporting creative movement interpretation and minimizing movement restriction. The paper offers a comprehensive review of the current state of VR dance experiences and music-to-animation synthesis, and an in-depth explanation of our plugin and its key components. Our approach has potential to enhance the creation of dance experiences in VR, reducing resource dependency and increasing versatility in dance styles. This research is a step towards fully automated VR dance track creation, potentially impacting realms of music, dance, and fitness in VR.
Supplementary Material
Results video.
- Download
- 46.52 MB
References
[1]
Omid Alemi, Jules Françoise, and Philippe Pasquier. 2017. GrooveNet: Real-time music-driven dance movement generation using artificial neural networks. networks 8, 17 (2017), 26.
[2]
Dimitrios S Alexiadis, Philip Kelly, Petros Daras, Noel E O’Connor, Tamy Boubekeur, and Maher Ben Moussa. 2011. Evaluating a dancer’s performance using kinect-based skeleton tracking. In Proceedings of the 19th ACM international conference on Multimedia. 659–662.
[3]
Andreas Aristidou, Nefeli Andreou, Loukas Charalambous, Anastasios Yiannakidis, Yiorgos Chrysanthou, V Hulusic, and A Chalmers. 2021. Virtual Dance Museum: The case of greek/cypriot folk dancing. In Proceedings of the Eurographics Workshop on Graphics and Cultural Heritage (Aire-la-Ville, Switzerland, Switzerland, 2021), Hulusic V., Chalmers A.,(Eds.), GCH, Vol. 21.
[4]
Rachele Bellini, Yanir Kleiman, and Daniel Cohen-Or. 2018. Dance to the beat: Synchronizing motion to audio. In Computational Visual Media. Springer, 197–208.
[5]
Jacky CP Chan, Howard Leung, Jeff KT Tang, and Taku Komura. 2010. A virtual reality dance training system using motion capture technology. IEEE transactions on learning technologies 4, 2 (2010), 187–195.
[6]
Kang Chen, Zhipeng Tan, Jin Lei, Song-Hai Zhang, Yuan-Chen Guo, Weidong Zhang, and Shi-Min Hu. 2021. Choreomaster: choreography-oriented music-driven dance synthesis. ACM Transactions on Graphics (TOG) 40, 4 (2021), 1–13.
[7]
Augusto Dias Pereira Dos Santos, Kalina Yacef, and Roberto Martinez-Maldonado. 2017. Let’s dance: how to build a user model for dance students using wearable technology. In Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization. 183–191.
[8]
Katerina El Raheb, Akrivi Katifori, and Yannis Ioannidis. 2016. HCI challenges in Dance Education. EAI Endorsed Transactions on Ambient Systems 3, 9 (2016).
[9]
Rukun Fan, Songhua Xu, and Weidong Geng. 2011. Example-based automatic music-driven conventional dance motion synthesis. IEEE transactions on visualization and computer graphics 18, 3 (2011), 501–515.
[10]
Joao P Ferreira, Thiago M Coutinho, Thiago L Gomes, José F Neto, Rafael Azevedo, Renato Martins, and Erickson R Nascimento. 2021. Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audio. Computers & Graphics 94 (2021), 11–21.
[11]
Diego Felipe Paez Granados, Jun Kinugawa, Yasuhisa Hirata, and Kazuhiro Kosuge. 2016. Guiding human motions in physical human-robot interaction through COM motion control of a dance teaching robot. In 2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids). IEEE, 279–285.
[12]
John K Haas. 2014. A history of the unity game engine. (2014).
[13]
Khoi Hoang Dinh, Ozgur S Oguz, Mariam Elsayed, and Dirk Wollherr. 2019. Adaptation and transfer of robot motion policies for close proximity human-robot interaction. Frontiers in Robotics and AI 6 (2019), 69.
[14]
Daniel Holden, Jun Saito, and Taku Komura. 2016. A deep learning framework for character motion synthesis and editing. ACM Transactions on Graphics (TOG) 35, 4 (2016), 1–11.
[15]
Javid Iqbal and Manjit Singh Sidhu. 2022. Acceptance of dance training system based on augmented reality and technology acceptance model (TAM). Virtual Reality 26, 1 (2022), 33–54.
[16]
Manish Joshi and Sangeeta Chakrabarty. 2021. An extensive review of computational dance automation techniques and applications. Proceedings of the Royal Society A 477, 2251 (2021), 20210071.
[17]
Tae-hoon Kim, Sang Il Park, and Sung Yong Shin. 2003. Rhythmic-motion synthesis based on motion-beat analysis. ACM Transactions on Graphics (TOG) 22, 3 (2003), 392–401.
[18]
Matthew Kyan, Guoyu Sun, Haiyan Li, Ling Zhong, Paisarn Muneesawang, Nan Dong, Bruce Elder, and Ling Guan. 2015. An approach to ballet dance training through ms kinect and visualization in a cave virtual reality environment. ACM Transactions on Intelligent Systems and Technology (TIST) 6, 2 (2015), 1–37.
[19]
Jehee Lee, Jinxiang Chai, Paul SA Reitsma, Jessica K Hodgins, and Nancy S Pollard. 2002. Interactive control of avatars animated with human motion data. In Proceedings of the 29th annual conference on Computer graphics and interactive techniques. 491–500.
[20]
Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. 2021. Ai choreographer: Music conditioned 3d dance generation with aist++. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 13401–13412.
[21]
Dylan P Losey, Andrea Bajcsy, Marcia K O’Malley, and Anca D Dragan. 2022. Physical interaction as communication: Learning robot objectives online from human corrections. The International Journal of Robotics Research 41, 1 (2022), 20–44.
[22]
Adriano Manfrè, Ignazio Infantino, Filippo Vella, and Salvatore Gaglio. 2016. An automatic system for humanoid dance creation. Biologically Inspired Cognitive Architectures 15 (2016), 1–9.
[23]
Christos Mousas. 2018. Performance-driven dance motion control of a virtual partner character. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 57–64.
[24]
Ferda Ofli, Yasemin Demir, Yücel Yemez, Engin Erzin, A Murat Tekalp, Koray Balcı, İdil Kızoğlu, Lale Akarun, Cristian Canton-Ferrer, Joëlle Tilmanne, 2008. An audio-driven dancing avatar. Journal on Multimodal User Interfaces 2 (2008), 93–103.
[25]
Ferda Ofli, Engin Erzin, Yücel Yemez, and A Murat Tekalp. 2011. Learn2dance: Learning statistical music-to-dance mappings for choreography synthesis. IEEE Transactions on Multimedia 14, 3 (2011), 747–759.
[26]
Roosa Piitulainen, Perttu Hämäläinen, and Elisa D Mekler. 2022. Vibing together: Dance experiences in social virtual reality. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–18.
[27]
Xuanchi Ren, Haoran Li, Zijian Huang, and Qifeng Chen. 2020. Self-supervised dance video synthesis conditioned on music. In Proceedings of the 28th ACM International Conference on Multimedia. 46–54.
[28]
Adriana Schulz, Wojciech Matusik, and Luiz Velho. 2013. Choreographics: An authoring tool for dance shows. Journal of Graphics Tools 17, 4 (2013), 159–176.
[29]
Simon Senecal, Niels A Nijdam, Andreas Aristidou, and Nadia Magnenat-Thalmann. 2020. Salsa dance learning evaluation and motion analysis in gamified virtual reality environment. Multimedia Tools and Applications 79 (2020), 24621–24643.
[30]
Taoran Tang, Jia Jia, and Hanyang Mao. 2018. Dance with Melody: An LSTM-autoencoder Approach to Music-oriented Dance Synthesis. In Proceedings of the 26th ACM international conference on Multimedia. 1598–1606.
[31]
Jonathan Tseng, Rodrigo Castellon, and Karen Liu. 2023. Edge: Editable dance generation from music. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 448–458.
[32]
Sijie Yan, Zhizhong Li, Yuanjun Xiong, Huahan Yan, and Dahua Lin. 2019. Convolutional sequence generation for skeleton-based action synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 4394–4402.
[33]
Zijie Ye, Haozhe Wu, Jia Jia, Yaohua Bu, Wei Chen, Fanbo Meng, and Yanfeng Wang. 2020. Choreonet: Towards music to dance synthesis with choreographic action unit. In Proceedings of the 28th ACM International Conference on Multimedia. 744–752.
[34]
Yong Zhao. 2022. Teaching traditional Yao dance in the digital environment: Forms of managing subcultural forms of cultural capital in the practice of local creative industries. Technology in Society 69 (2022), 101943.
Index Terms
- Virtual Reality Dance Tracks from Skeletal Animations
Recommendations
Virtual Dance Mirror: A Functional Approach to Avatar Representation through Movement in Immersive VR
MOCO '22: Proceedings of the 8th International Conference on Movement and ComputingImmersive Virtual Reality (VR) technologies offer new possibilities for studying embodied interaction with different sets of constraints and affordances for action-taking while using one’s physical body. In this study, we designed and prototyped a VR ...
LMA driven Dynamic Audiovisuals in a Virtual Reality Live Dance Performance: Ghostdance
MOCO '24: Proceedings of the 9th International Conference on Movement and ComputingGhostdance is an evolving generative art project in the field of dance and virtual reality (VR). It mixes visual, auditory and immersive experiences, making use of generative algorithms to create a dynamic audiovisual landscape with continuously changing ...
HEROES: a duet in mixed reality
SIGGRAPH '17: ACM SIGGRAPH 2017 VR VillageUsing 2-D, virtual reality, and augmented reality headsets, Heroes crafts a multidimensional experience that lets you chase dancers around a pool and on stage, scale through a labyrinth of stunning architecture, and even reach out and hold the dancers ...
Comments
Information & Contributors
Information
Published In
November 2023
315 pages
ISBN:9798400709432
DOI:10.1145/3625008
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 06 January 2024
Check for updates
Author Tags
Qualifiers
- Short-paper
- Research
- Refereed limited
Conference
SVR '23
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 47Total Downloads
- Downloads (Last 12 months)47
- Downloads (Last 6 weeks)8
Reflects downloads up to 25 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format