poster

A multimodal probabilistic model for gesture--based control of sound synthesis

Authors:

Jules Françoise,

Norbert Schnell,

Frédéric BevilacquaAuthors Info & Claims

MM '13: Proceedings of the 21st ACM international conference on Multimedia

Pages 705 - 708

https://doi.org/10.1145/2502081.2502184

Published: 21 October 2013 Publication History

Get Access

Abstract

In this paper, we propose a multimodal approach to create the mapping between gesture and sound in interactive music systems. Specifically, we propose to use a multimodal HMM to conjointly model the gesture and sound parameters. Our approach is compatible with a learning method that allows users to define the gesture--sound relationships interactively. We describe an implementation of this method for the control of physical modeling sound synthesis. Our model is promising to capture expressive gesture variations while guaranteeing a consistent relationship between gesture and sound.

References

[1]

F. Bevilacqua, N. Schnell, N. Rasamimanana, B. Zamborlin, and F. Guédy. Online Gesture Analysis and Control of Audio Processing. In Musical Robots and Interactive Multimodal Systems, pages 127--142. Springer, 2011.

Crossref

Google Scholar

[2]

M. Brand. Voice puppetry. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pages 21--28. 1999.

Digital Library

Google Scholar

[3]

K. Choi, Y. Luo, and J.-n. Hwang. Hidden Markov model inversion for audio-to-visual conversion in an MPEG-4 facial animation system. The Journal of VLSI Signal Processing, 29(1):51--61, 2001.

Digital Library

Google Scholar

[4]

R. Fiebrink, P. R. Cook, and D. Trueman. Play-along mapping of musical controllers. In Proceedings of the International Computer Music Conference, 2009.

Google Scholar

[5]

R. Fiebrink, P. R. Cook, and D. Trueman. Human model evaluation in interactive supervised learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 147--156. 2011.

Digital Library

Google Scholar

[6]

S. Fu, R. Gutierrez-Osuna, A. Esposito, P. Kakumanu, and O. Garcia. Audio/visual mapping with cross-modal hidden Markov models. Multimedia, IEEE Transactions on, 7(2):243--252, Apr. 2005.

Digital Library

Google Scholar

[7]

N. Gillian and R. Knapp. A Machine Learning Toolbox For Musician Computer Interaction. In proceedings ot the International Conference on New Interfaces for Musical Expression, Oslo, Norway, 2011.

Google Scholar

[8]

G. Hofer. Speech-driven animation using multi-modal hidden Markov models. Phd dissertation, University of Edimburgh, 2009.

Google Scholar

[9]

L. R. Rabiner. A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. Proceedings of the IEEE, 77(2):257--286, 1989.

Digital Library

Google Scholar

[10]

N. Rasamimanana, F. Bevilacqua, J. Bloit, N. Schnell, E. Fléty, A. Cera, U. Petrevski, and J.-L. Frechin. The urban musical game: using sport balls as musical interfaces. In CHI '12 Extended Abstracts on Human Factors in Computing Systems, pages 1027--1030. 2012.

Digital Library

Google Scholar

[11]

D. Rocchesso, S. Serafin, F. Behrendt, N. Bernardini, R. Bresin, G. Eckel, K. Franinovic, T. Hermann, S. Pauletto, P. Susini, and Y. Visell. Sonic interaction design: sound, information and experience. In CHI '08 Extended Abstracts on Human Factors in Computing Systems, pages 3969--3972. 2008.

Digital Library

Google Scholar

[12]

D. Wessel and M. Wright. Problems and Prospects for Intimate Musical Control of Computers. Computer Music Journal, 26(3):11--22, Sept. 2002.

Digital Library

Google Scholar

[13]

E. Yamamoto, S. Nakamura, and K. Shikano. Speech-to-lip movement synthesis based on the EM algorithm using audio-visual HMMs. In Multimedia Signal Processing, 1998 IEEE Second Workshop on, pages 2--5, 1998.

Google Scholar

[14]

H. Zen, Y. Nankaku, and K. Tokuda. Continuous Stochastic Feature Mapping Based on Trajectory HMMs. Audio, Speech, and Language Processing, IEEE Transactions on, 19(2):417--430, 2011.

Digital Library

Google Scholar

Cited By

View all

Nabi SEsling PPeeters GBevilacqua F(2024)Embodied exploration of deep latent spaces in interactive dance-music performanceProceedings of the 9th International Conference on Movement and Computing10.1145/3658852.3659072(1-9)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3658852.3659072
Bernardo FZbyszyński MGrierson MFiebrink R(2020)Designing and Evaluating the Usability of a Machine Learning API for Rapid Prototyping Music TechnologyFrontiers in Artificial Intelligence10.3389/frai.2020.000133Online publication date: 3-Apr-2020
https://doi.org/10.3389/frai.2020.00013
Gillies M(2019)Understanding the Role of Interactive Machine Learning in Movement Interaction DesignACM Transactions on Computer-Human Interaction10.1145/328730726:1(1-34)Online publication date: 6-Feb-2019
https://dl.acm.org/doi/10.1145/3287307
Show More Cited By

Index Terms

A multimodal probabilistic model for gesture--based control of sound synthesis
1. Applied computing
  1. Arts and humanities
    1. Sound and music computing
2. Information systems
  1. Information retrieval
    1. Specialized information retrieval
      1. Multimedia and multimodal retrieval
        Music retrieval

Recommendations

Gesture-based control of physical modeling sound synthesis: a mapping-by-demonstration approach
MM '13: Proceedings of the 21st ACM international conference on Multimedia

We address the issue of mapping between gesture and sound for gesture-based control of physical modeling sound synthesis. We propose an approach called mapping by demonstration, allowing users to design the mapping by performing gestures while listening ...
Gesture--sound mapping by demonstration in interactive music systems
MM '13: Proceedings of the 21st ACM international conference on Multimedia

In this paper we address the issue of mapping between gesture and sound in interactive music systems. Our approach, we call mapping by demonstration, aims at learning the mapping from examples provided by users while interacting with the system. We ...
Gesture in performance with traditional musical instruments and electronics: Use of embodied music cognition and multimodal motion capture to design gestural mapping strategies
MOCO '14: Proceedings of the 2014 International Workshop on Movement and Computing

This paper describes the implementation of gestural mapping strategies for performance with a traditional musical instrument and electronics. The approach adopted is informed by embodied music cognition and functional categories of musical gestures. ...

Comments

Information & Contributors

Information

Published In

MM '13: Proceedings of the 21st ACM international conference on Multimedia

October 2013

1166 pages

ISBN:9781450324045

DOI:10.1145/2502081

General Chairs:
Alejandro (Alex) Jaimes
Yahoo!, Spain
,
Nicu Sebe
University of Trento, Italy
,
Nozha Boujemaa
INRIA, France
,
Program Chairs:
Daniel Gatica-Perez
IDIAP & EPFL, Switzerland
,
David A. Shamma
Yahoo!, USA
,
Marcel Worring
University of Amsterdam, The Netherlands
,
Roger Zimmermann
National University of Singapore, Singapore

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 October 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Poster

Conference

MM '13

Sponsor:

SIGMM

MM '13: ACM Multimedia Conference

October 21 - 25, 2013

Barcelona, Spain

Acceptance Rates

MM '13 Paper Acceptance Rate 47 of 235 submissions, 20%;

Overall Acceptance Rate 995 of 4,171 submissions, 24%

Upcoming Conference

MM '24

Sponsor:
sigmm

The 32nd ACM International Conference on Multimedia

October 28 - November 1, 2024

Melbourne , VIC , Australia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

20
Total Citations
View Citations
329
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)0

Reflects downloads up to 09 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Nabi SEsling PPeeters GBevilacqua F(2024)Embodied exploration of deep latent spaces in interactive dance-music performanceProceedings of the 9th International Conference on Movement and Computing10.1145/3658852.3659072(1-9)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3658852.3659072
Bernardo FZbyszyński MGrierson MFiebrink R(2020)Designing and Evaluating the Usability of a Machine Learning API for Rapid Prototyping Music TechnologyFrontiers in Artificial Intelligence10.3389/frai.2020.000133Online publication date: 3-Apr-2020
https://doi.org/10.3389/frai.2020.00013
Gillies M(2019)Understanding the Role of Interactive Machine Learning in Movement Interaction DesignACM Transactions on Computer-Human Interaction10.1145/328730726:1(1-34)Online publication date: 6-Feb-2019
https://dl.acm.org/doi/10.1145/3287307
Bernardo FGrierson MFiebrink R(2018)User-Centred Design Actions for Lightweight Evaluation of an Interactive Machine Learning ToolkitJournal of Science and Technology of the Arts10.7559/citarj.v10i2.50910:2(2)Online publication date: 11-Jul-2018
https://doi.org/10.7559/citarj.v10i2.509
Nakamura TNagai TTaniguchi T(2018)SERKET: An Architecture for Connecting Stochastic Models to Realize a Large-Scale Cognitive ModelFrontiers in Neurorobotics10.3389/fnbot.2018.0002512Online publication date: 26-Jun-2018
https://doi.org/10.3389/fnbot.2018.00025
Françoise JBevilacqua F(2018)Motion-Sound Mapping through InteractionACM Transactions on Interactive Intelligent Systems10.1145/32118268:2(1-30)Online publication date: 13-Jun-2018
https://dl.acm.org/doi/10.1145/3211826
Houix OMonache SLachambre HBevilacqua FRocchesso DLemaitre GFagerlönn J(2016)Innovative Tools for Sound Sketching Combining Vocalizations and GesturesProceedings of the Audio Mostly 201610.1145/2986416.2986442(12-19)Online publication date: 4-Oct-2016
https://dl.acm.org/doi/10.1145/2986416.2986442
Yang XMolchanov PKautz JHanjalic ASnoek CWorring MBulterman DHuet BKelliher AKompatsiaris YLi J(2016)Multilayer and Multimodal Fusion of Deep Neural Networks for Video ClassificationProceedings of the 24th ACM international conference on Multimedia10.1145/2964284.2964297(978-987)Online publication date: 1-Oct-2016
https://dl.acm.org/doi/10.1145/2964284.2964297
Françoise JChapuis OHanneton SBevilacqua FKaye JDruin ALampe CMorris DHourcade J(2016)SoundGuidesProceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems10.1145/2851581.2892420(2829-2836)Online publication date: 7-May-2016
https://dl.acm.org/doi/10.1145/2851581.2892420
Lee SKim SHan J(2016)Promoting Stretching Activity with Smartwatch - A Pilot StudyEntertainment Computing - ICEC 201610.1007/978-3-319-46100-7_19(211-216)Online publication date: 20-Sep-2016
https://doi.org/10.1007/978-3-319-46100-7_19
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Gesture-based control of physical modeling sound synthesis: a mapping-by-demonstration approach

Gesture--sound mapping by demonstration in interactive music systems

Gesture in performance with traditional musical instruments and electronics: Use of embodied music cognition and multimodal motion capture to design gestural mapping strategies