research-article

3d augmented mirror: a multimodal interface for string instrument learning and teaching with gesture support

Authors:

Kerstin Neubarth,

Thijs Koerselman,

Bee OngAuthors Info & Claims

ICMI '07: Proceedings of the 9th international conference on Multimodal interfaces

Pages 339 - 345

https://doi.org/10.1145/1322192.1322252

Published: 12 November 2007 Publication History

Abstract

Multimodal interfaces can open up new possibilities for music education, where the traditional model of teaching is based predominantly on verbal feedback. This paper explores the development and use of multimodal interfaces in novel tools to support music practice training. The design of multimodal interfaces for music education presents a challenge in several respects. One is the integration of multimodal technology into the music learning process. The other is the technological development, where we present a solution that aims to support string practice training with visual and auditory feedback. Building on the traditional function of a physical mirror as a teaching aid, we describe the concept and development of an "augmented mirror" using 3D motion capture technology.

References

[1]

Ng, K., Ong, B., Khan, A., Bellini, P., Nesi, P., Mitolo, N. Technology Enhanced Learning with Interactive Multimedia Environment for Music Edutainment. In Proceedings of the EDEN 2006 Annual Conference -- E-Competences for Life, Employment and Innovation, "E" is more! E-learning Enabling Education in Evolving Europe, European Distance and E-Learning Network, Vienna, Austria, 2006, 182--187.

[2]

Ng, K. Creative Multimedia Interfaces and Collaborative Technology Enhanced Environment for Music Playing and Learning. In Proceedings of The Reflective Conservatoire: Sorcerers and Apprentices Conference, London, UK, 2006

[3]

West, T., and Rostvall, A-L. A Study of Interaction and Learning in Instrumental Teaching. International Journal of Music Education. 40,1, 2003 16--27.

[4]

Bevilaqua, F., Guedy, F., Schnell, N., Flety, E., Leyroy, N. Wireless Sensor Interface and Gesture-Follower for Music Pedagogy. In Proceedings of the International Conference on New Interfaces for Musical Expression 2007 (NIME'07). New York, USA. 2007.

Digital Library

[5]

Kramer, G. (ed.), Auditory Display: Sonification, Audification and Auditory Interfaces. Proceedings of the First International Conference on Auditory Display (ICAD) 1992

[6]

Styns, F., van Noorden, L., Leman, M. Some basic observations on how people move on music and how they relate music to movement. Second International Conference on Music and Gesture, Manchester, UK. 2006.

[7]

Effenberg, A. Zum Potential Komplexer Akustischer Bewegungsinformationen für die Technikansteuerung, In Leistungssport 5, 19--25.

[8]

Hermann, T., Höner, O., Ritter, H. AcouMotion -- An Interactive Sonification System for Acoustic Motion Control. In Gibet, S., Courty, N., and Kamp, J.-F., (Eds.):Gesture In Human-Computer Interaction and Simulation -- 6th International Gesture Workshop, (GW'2005), Berder Island, France, 2005, Series LNCS/LNAI 3881, Springer, 312--323.

Digital Library

[9]

Väljamäe, A., Larsson, P., Västfjäl, D., Kleiner, M. Travelling Without Moving: Auditory Scene Cues for Translational Self-Motion. In Proceedings of the International Conference on Auditory Display (ICAD'05). Limerick, Ireland, 2005.

[10]

Mora, J., Won-Sook, L., Comeau, G., Shirmohammadi, S., El Saddik., A. Assisted Piano Pedagogy through 3D Visualization of Piano Playing. In Proceedings of the IEEE International Workshop on Haptic Audio Visual Environments and Their Applications (HAVE' 2006), Ottawa, Canada, 2006.

[11]

Sturm, H., Rabbath F. The Art of the Bow DVD, 2006

[12]

Shan, G. Visentin, P. A Quantitative Three-Dimensional Analysis of Arm Kinematics in Violin Performance. In Medical Problems of Performing Artists, 18, 1. 2003, 3--10.

[13]

Ferguson, S. Learning Musical Instrument Skills through Interactive Sonification. In Proceedings of the International Conference on New Interfaces for Musical Expression 2006 (NIME'06), Paris, France, 2006

Digital Library

[14]

Verfaille, V., Quek., O, Wanderley M. M. Sonification of Musicians' Ancillary Gestures, in Proceedings of the International Conference on Auditory Display (ICAD'06), London, UK, 2006

[15]

Bevilaqua, F., Ridenour, J. and Cuccia. D. 3D Motion Capture Data: Motion Analysis and Mapping to Music. In Proceedings of the Symposium on Sensing and Input for Media Centric Systems (SIMS). Santa Barbara, USA. 2002

[16]

Rolland, P. The Teaching of Action in String Playing. Illinois String Research Associates. 1974

[17]

Gerle, R. The Art of Bowing Practice: The Expressive Bow Technique. Stainer & Bell, 1991

[18]

Winstein, C. Knowledge of Results and Motor Learning - Implications for Physical Therapy. Physical Therapy 71, 2. 1991, 140--149.

[19]

Bellini, P., Nesi, P., and Zoia, G. Symbolic music representation in MPEG, IEEE Multimedia, 12, 4, 2005, 42--49.

Digital Library

Cited By

Provenzale CDi Tommaso FDi Stefano NFormica DTaffoni F(2024)Real-Time Visual Feedback Based on MIMUs Technology Reduces Bowing Errors in Beginner Violin StudentsSensors10.3390/s2412396124:12(3961)Online publication date: 19-Jun-2024
https://doi.org/10.3390/s24123961
Wilson WGranieri NAli-MacLachlan I(2023)Ubiquitous Multimodality as a Tool in Violin Performance Classification2023 4th International Symposium on the Internet of Sounds10.1109/IEEECONF59510.2023.10335435(1-8)Online publication date: 26-Oct-2023
https://doi.org/10.1109/IEEECONF59510.2023.10335435
Mandanici MSpagnol SLudovico LBaratè AAvanzini FMandanici MSpagnol SLudovico LBaratè AAvanzini F(2023)A Taxonomy of Digital Music Learning ResourcesDigital Music Learning Resources10.1007/978-981-99-4206-0_4(53-66)Online publication date: 27-Jul-2023
https://doi.org/10.1007/978-981-99-4206-0_4
Show More Cited By

Index Terms

3d augmented mirror: a multimodal interface for string instrument learning and teaching with gesture support

Recommendations

Gesture in performance with traditional musical instruments and electronics: Use of embodied music cognition and multimodal motion capture to design gestural mapping strategies
MOCO '14: Proceedings of the 2014 International Workshop on Movement and Computing

This paper describes the implementation of gestural mapping strategies for performance with a traditional musical instrument and electronics. The approach adopted is informed by embodied music cognition and functional categories of musical gestures. ...
Interactive feedbacks with visualisation and sonification for technology-enhanced learning for music performance
SIGDOC '08: Proceedings of the 26th annual ACM international conference on Design of communication

In this poster, we present a brief overview of the i-Maestro project and focus on the design and applications of interactive multimodal feedbacks using visualisation and sonification to communicate live analysis of the playing gesture. Interactive ...
Sonification of a juggling performance using spatial audio

We developed a new musical instrument by equipping juggling balls with accelerometers, gyroscopes, and WiFi sensors. The system measures acceleration and rotation, allowing us to distinguish between ball movement and static states. We can determine the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '07: Proceedings of the 9th international conference on Multimodal interfaces

November 2007

402 pages

ISBN:9781595938176

DOI:10.1145/1322192

General Chairs:
Kenji Mase
Nagoya University, Japan
,
Dominic Massaro
UC Santa Cruz, USA
,
Program Chairs:
Kazuya Takeda
Nagoya University, Japan
,
Deb Roy
MIT, USA
,
Alexandros Potamianos
Technical University of Crete, Greece

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 November 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICMI07

Sponsor:

ICMI07: International Conference on Multimodal Interface

November 12 - 15, 2007

Aichi, Nagoya, Japan

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

35
Total Citations
View Citations
749
Total Downloads

Downloads (Last 12 months)19
Downloads (Last 6 weeks)1

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Provenzale CDi Tommaso FDi Stefano NFormica DTaffoni F(2024)Real-Time Visual Feedback Based on MIMUs Technology Reduces Bowing Errors in Beginner Violin StudentsSensors10.3390/s2412396124:12(3961)Online publication date: 19-Jun-2024
https://doi.org/10.3390/s24123961
Wilson WGranieri NAli-MacLachlan I(2023)Ubiquitous Multimodality as a Tool in Violin Performance Classification2023 4th International Symposium on the Internet of Sounds10.1109/IEEECONF59510.2023.10335435(1-8)Online publication date: 26-Oct-2023
https://doi.org/10.1109/IEEECONF59510.2023.10335435
Mandanici MSpagnol SLudovico LBaratè AAvanzini FMandanici MSpagnol SLudovico LBaratè AAvanzini F(2023)A Taxonomy of Digital Music Learning ResourcesDigital Music Learning Resources10.1007/978-981-99-4206-0_4(53-66)Online publication date: 27-Jul-2023
https://doi.org/10.1007/978-981-99-4206-0_4
Mandanici MSpagnol SLudovico LBaratè AAvanzini FMandanici MSpagnol SLudovico LBaratè AAvanzini F(2023)Learning Theories and Technology-Based Learning ApproachesDigital Music Learning Resources10.1007/978-981-99-4206-0_2(19-37)Online publication date: 27-Jul-2023
https://doi.org/10.1007/978-981-99-4206-0_2
Zhou QIrlitti AYu DGoncalves JVelloso E(2022)Movement Guidance using a Mixed Reality MirrorProceedings of the 2022 ACM Designing Interactive Systems Conference10.1145/3532106.3533466(821-834)Online publication date: 13-Jun-2022
https://dl.acm.org/doi/10.1145/3532106.3533466
Ramirez-Melendez RWaddell G(2022)Technology-Enhanced Learning of PerformanceThe Oxford Handbook of Music Performance, Volume 210.1093/oxfordhb/9780190058869.013.24(528-C24.P144)Online publication date: 18-Mar-2022
https://doi.org/10.1093/oxfordhb/9780190058869.013.24
Zhang FChen LMiao WSun L(2020)Research on Accuracy of Augmented Reality Surgical Navigation System Based on Multi-View Virtual and Real Registration TechnologyIEEE Access10.1109/ACCESS.2020.30071788(122511-122528)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.3007178
Park HLee GSeo BBillinghurst M(2020)User experience design for a smart-mirror-based personalized training systemMultimedia Tools and Applications10.1007/s11042-020-10148-5Online publication date: 17-Nov-2020
https://doi.org/10.1007/s11042-020-10148-5
Lee GPark HKim SBillinghurst M(2020)Improving User Experience in Augmented Reality Mirrors with 3D DisplaysAdvances in Visual Computing10.1007/978-3-030-64556-4_50(642-653)Online publication date: 7-Dec-2020
https://doi.org/10.1007/978-3-030-64556-4_50
Hohagen JWöllner C(2019)Bewegungssonifikation: Psychologische Grundlagen und Auswirkungen der Verklanglichung menschlicher Handlungen in der Rehabilitation, im Sport und bei MusikaufführungenJahrbuch Musikpsychologie10.5964/jbdgm.2018v28.3628Online publication date: 28-May-2019
https://doi.org/10.5964/jbdgm.2018v28.36
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents