abstract

HoloKeys: Interactive Piano Education Using Augmented Reality and IoT

Authors:

Austin Jerald Stanbury,

Ines Said,

Hyo Jeong KangAuthors Info & Claims

VRST '21: Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology

Article No.: 76, Pages 1 - 3

https://doi.org/10.1145/3489849.3489921

Published: 08 December 2021 Publication History

Get Access

Abstract

The rise of online learning poses unique challenges in music education, where live demonstration and musical synchronization are critical for student success. We present HoloKeys, a music education interface which allows instructors to play remotely located pianos using an augmented reality headset and wifi-enabled microcontrollers. This approach allows students to receive distance education which is more direct, immersive, and comprehensive than conventional video conferencing allows for. HoloKeys enables remote students to observe live instructional demonstration on a physical keyboard in their immediate environment just as they would in traditional settings. HoloKeys consists of two separate components: an augmented reality user interface and a piano playing apparatus. Our system aims to extend online music education beyond desktop platforms into the physical world, thereby addressing crucial obstacles encountered by educators and students transitioning into online education.

Supplementary Material

AVI File (HoloKeysSubmissionVideo.avi)

Supplemental video

Download
224.13 MB

References

[1]

Jonathan Chow, Haoyang Feng, Robert Amor, and Burkhard C Wünsche. 2013. Music education using augmented reality with a head mounted display. In Proceedings of the Fourteenth Australasian User Interface Conference-Volume 139. 73–79. https://doi.org/10.5555/2525493.2525501

Digital Library

Google Scholar

[2]

Shantanu Das, Seth Glickman, Fu Yen Hsiao, and Byunghwan Lee. 2017. Music Everywhere–Augmented Reality Piano Improvisation Learning System. Entertainment Technology Center, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA (2017).

Google Scholar

[3]

Phillip M Hash. 2020. Remote Learning in School Bands During the COVID-19 Shutdown. Journal of Research in Music Education(2020), 0022429420967008. https://doi.org/10.1177/0022429420967008

Crossref

Google Scholar

[4]

F. Huang, Y. Zhou, Yao Yu, Z. Wang, and Sidan Du. 2011. Piano AR: A Markerless Augmented Reality Based Piano Teaching System. 2011 Third International Conference on Intelligent Human-Machine Systems and Cybernetics 2(2011), 47–52. https://doi.org/10.1109/IHMSC.2011.82

Digital Library

Google Scholar

[5]

Estelle Ruth Jorgensen. 2003. Transforming music education. Indiana University Press.

Google Scholar

[6]

D Molero, S Schez-Sobrino, D Vallejo, C Glez-Morcillo, and J Albusac. 2020. A novel approach to learning music and piano based on mixed reality and gamification. Multimedia Tools and Applications(2020), 1–22. https://doi.org/10.1007/s11042-020-09678-9

Digital Library

Google Scholar

[7]

M Ostanin, R Yagfarov, D Devitt, A Akhmetzyanov, and A Klimchik. 2020. Multi robots interactive control using mixed reality. International Journal of Production Research (2020), 1–13.

Google Scholar

[8]

Pamela D Pike. 2017. Improving music teaching and learning through online service: A case study of a synchronous online teaching internship. International Journal of Music Education 35, 1 (2017), 107–117. https://doi.org/10.1177/0255761415613534

Crossref

Google Scholar

[9]

Xizi Lucy Wang. 2020. PPVR: Teaching and Learning to Play the Piano in Virtual Reality. Master’s thesis. University of Michigan.

Google Scholar

Cited By

View all

Matzek EYankee TKohler OLipke-Perry TBanerjee NBanerjee S(2024)VRmonic: A VR Piano Playing Form Trainer2024 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR)10.1109/AIxVR59861.2024.00056(330-334)Online publication date: 17-Jan-2024
https://doi.org/10.1109/AIxVR59861.2024.00056
Liu YAbidin SYang SWang J(2023)Design of an Interactive Educational Information System Based on the SECI ModelProceedings of the 2023 4th International Conference on Education, Knowledge and Information Management (ICEKIM 2023)10.2991/978-94-6463-172-2_207(1869-1880)Online publication date: 30-Jun-2023
https://doi.org/10.2991/978-94-6463-172-2_207
Liu RWu ELiao CNishioka HFuruya SKoike H(2023)PianoSyncAR: Enhancing Piano Learning through Visualizing Synchronized Hand Pose Discrepancies in Augmented Reality2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR59233.2023.00101(859-868)Online publication date: 16-Oct-2023
https://doi.org/10.1109/ISMAR59233.2023.00101
Show More Cited By

Recommendations

Virtual reality and augmented reality for education: panel
SA '16: SIGGRAPH ASIA 2016 Symposium on Education: Talks

Since the last few decades, virtual reality (VR) and augmented-reality (AR) interfaces have shown the potential to enhance teaching and learning, by combining physical and virtual worlds and leveraging the advantages of both [Yannier et al. 2015]. ...
Interactive piano training using augmented reality and the Internet of Things
Abstract
The technology of augmented reality in education is a significant attempt to reform the training mode and develop education in general. Piano training usually includes using well-known computer means of teaching music on-screen and hardware. In ...
Creating interactive physics education books with augmented reality
OzCHI '12: Proceedings of the 24th Australian Computer-Human Interaction Conference

Augmented Books show three-dimensional animated educational content and provide a means for students to interact with this content in an engaging learning experience. In this paper we present a framework for creating educational Augmented Reality (AR) ...

Comments

Information & Contributors

Information

Published In

VRST '21: Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology

December 2021

563 pages

ISBN:9781450390927

DOI:10.1145/3489849

Editors:
Yuichi Itoh
Aoyama Gakuin University, Japan
,
Kazuki Takashima
Tohoku University, Japan
,
Parinya Punpongsanon
Osaka University, Japan
,
Misha Sra
University of California, Santa Barbara, USA
,
Kazuyuki Fujita
Tohoku University, Japan
,
Shigeo Yoshida
The University of Tokyo, Japan
,
Tham Piumsomboon
University of Canterbury, New Zealand

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 December 2021

Check for updates

Qualifiers

Abstract
Research
Refereed limited

Conference

VRST '21

Sponsor:

VRST '21: 27th ACM Symposium on Virtual Reality Software and Technology

December 8 - 10, 2021

Osaka, Japan

Acceptance Rates

Overall Acceptance Rate 66 of 254 submissions, 26%

Upcoming Conference

VRST '24

Sponsor:
sigchi
sigchi

30th ACM Symposium on Virtual Reality Software and Technology

October 9 - 11, 2024

Trier , Germany

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
299
Total Downloads

Downloads (Last 12 months)70
Downloads (Last 6 weeks)5

Reflects downloads up to 18 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Matzek EYankee TKohler OLipke-Perry TBanerjee NBanerjee S(2024)VRmonic: A VR Piano Playing Form Trainer2024 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR)10.1109/AIxVR59861.2024.00056(330-334)Online publication date: 17-Jan-2024
https://doi.org/10.1109/AIxVR59861.2024.00056
Liu YAbidin SYang SWang J(2023)Design of an Interactive Educational Information System Based on the SECI ModelProceedings of the 2023 4th International Conference on Education, Knowledge and Information Management (ICEKIM 2023)10.2991/978-94-6463-172-2_207(1869-1880)Online publication date: 30-Jun-2023
https://doi.org/10.2991/978-94-6463-172-2_207
Liu RWu ELiao CNishioka HFuruya SKoike H(2023)PianoSyncAR: Enhancing Piano Learning through Visualizing Synchronized Hand Pose Discrepancies in Augmented Reality2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR59233.2023.00101(859-868)Online publication date: 16-Oct-2023
https://doi.org/10.1109/ISMAR59233.2023.00101
Deja JMayer SČopič Pucihar KKljun M(2022)A Survey of Augmented Piano Prototypes: Has Augmentation Improved Learning Experiences?Proceedings of the ACM on Human-Computer Interaction10.1145/35677196:ISS(226-253)Online publication date: 14-Nov-2022
https://dl.acm.org/doi/10.1145/3567719

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Abstract

Supplementary Material

References

Cited By

Recommendations

Virtual reality and augmented reality for education: panel

Interactive piano training using augmented reality and the Internet of Things

Creating interactive physics education books with augmented reality

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

HTML Format

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations