research-article

Spatial Sound and Multimodal Interaction in Immersive Environments

Authors:

Francesco Grani,

Steven Gelineck,

Georgios Triantafyllidis,

Stefania SerafinAuthors Info & Claims

AM '15: Proceedings of the Audio Mostly 2015 on Interaction With Sound

Article No.: 17, Pages 1 - 5

https://doi.org/10.1145/2814895.2814919

Published: 07 October 2015 Publication History

Abstract

Spatial sound and interactivity are key elements of investigation at the Sound And Music Computing master program at Aalborg University Copenhagen.

We present a collection of research directions and recent results from work in these areas, with the focus on our multifaceted approaches to two primary problem areas: 1) creation of interactive spatial audio experiences for immersive virtual and augmented reality scenarios, and 2) production and mixing of spatial audio for cinema, music, and other artistic contexts. Several ongoing research projects are described, wherein the latest developments are discussed.

These include elements in which we have provided sonic interaction in virtual environments, interactivity with volumetric sound sources using VBAP and Wave Field Synthesis (WFS), and binaural sound for virtual environments and spatial audio mixing. We show that the variety of approaches presented here are necessary in order to optimize interactivity with spatial audio for each particular type of task.

References

[1]

Jens Ahrens and Sascha Spors. Two physical models for spatially extended virtual sound sources. In Proc. AES Convention, New York, NY, USA, 2011.

[2]

V Ralph Algazi, Richard O Duda, Dennis M Thompson, and Carlos Avendano. The cipic hrtf database. In Applications of Signal Processing to Audio and Acoustics, 2001 IEEE Workshop on the, pages 99--102. IEEE, 2001.

[3]

P. Barattini, C. Morand, and N. M. Robertson. A proposed gesture set for the control of industrial collaborative robots. In RO-MAN, 2012 IEEE, pages 132--137, Sept 2012.

[4]

Durand R Begault et al. 3-D sound for virtual reality and multimedia, volume 955. Citeseer, 1994.

Digital Library

[5]

Jung-Woo Choi. Extension of perceived source width using sound field reproduction systems. In ICA 2013 Montreal, 2013.

[6]

Anthony Churnside, Chris Pike, and Max Leonard. Musical movements---gesture based audio interfaces. In Audio Engineering Society Convention 131, Oct 2011.

[7]

L. Cruz, D. Lucio, and L. Velho. Kinect and rgbd images: Challenges and applications. In Graphics, Patterns and Images Tutorials (SIBGRAPI-T), 2012 25th SIBGRAPI Conference on, pages 36--49, Aug 2012.

Digital Library

[8]

Romuald Deshayes and Tom Mens. Statechart modelling of interactive gesture-based applications. In Proc. First International Workshop on Combining Design and Engineering of Interactive Systems through Models and Tools (ComDeis-Moto),. Lisbon, Portugal (September 2011), iNTERACT, 2011.

[9]

Wolfgang Fohl and Malte Nogalski. A gesture control interface for a wave field synthesis system. In International Conference on New Interfaces for Musical Expression, 2013.

[10]

Federico Fontana and Yon Visell. Walking with the Senses: Perceptual Techniques for Walking in Simulated Environments. Logos-Verlag, 2012.

[11]

Steven Gelineck, Morten Büchert, and Jesper Andersen. Towards a more flexible and creative music mixing interface. In CHI'13 Extended Abstracts on Human Factors in Computing Systems, pages 733--738. ACM, 2013.

Digital Library

[12]

Steven Gelineck and Dannie Korsgaard. An exploratory evaluation of user interfaces for 3d audio mixing. In Audio Engineering Society Convention 138. Audio Engineering Society, 2015.

[13]

Steven Gelineck, Dan Overholt, Morten Büchert, and Jesper Andersen. Towards an interface for music mixing based on smart tangibles and multitouch. In Proc. of NIME, 2013.

[14]

Francesco Grani, Ferran Argelaguet, Valérie Gouranton, Marwan Badawi, Ronan Gaugne, Stefania Serafin, and Anatole Lecuyer. Design and evaluation of binaural auditory rendering for caves. In Virtual Reality (VR), 2014 iEEE, pages 73--74. IEEE, 2014.

[15]

Francesco Grani, S Serafin, F Argelaguet, V Gouranton, M Badawi, R Gaugne, and Anatole Lécuyer. Audio-visual attractors for capturing attention to the screens when walking in cave systems. In VR Workshop: Sonic Interaction in Virtual Environments (SIVE), 2014 IEEE, pages 3--6. IEEE, 2014.

[16]

Matti Gröhn, Tapio Lokki, and Tapio Takala. Localizing sound sources in a cave-like virtual environment with loudspeaker array reproduction. Presence: Teleoperators and Virtual Environments, 16(2):157--171, 2007.

Digital Library

[17]

Rose Johnson, Kenton O'Hara, Abigail Sellen, Claire Cousins, and Antonio Criminisi. Exploring the potential for touchless interaction in image-guided interventional radiology. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '11, pages 3323--3332, 2011.

Digital Library

[18]

Jean-Marc Jot. Real-time spatial processing of sounds for music, multimedia and interactive human-computer interfaces. Multimedia systems, 7(1):55--69, 1999.

Digital Library

[19]

Mikko-Ville Laitinen, Tapani Pihlajamäki, Cumhur Erkut, and Ville Pulkki. Parametric time-frequency representation of spatial sound in virtual worlds. ACM Transactions on Applied Perception, 2012.

Digital Library

[20]

Pontus Larsson, Daniel Vastfjall, and Mendel Kleiner. Better presence and performance in virtual environments by improved binaural sound rendering. In Audio Engineering Society Conference: 22nd International Conference: Virtual, Synthetic, and Entertainment Audio. Audio Engineering Society, 2002.

[21]

Daniel Liebling and Meredith Ringel Morris. Kinected browser: Depth camera interaction for the web. In Proceedings of the 2012 ACM International Conference on Interactive Tabletops and Surfaces, ITS '12, pages 105--108, 2012.

Digital Library

[22]

Shih-Yao Lin, Yun-Chien Lai, Li-Wei Chan, and Yi-Ping Hung. Real-time 3d model-based gesture tracking for multimedia control. In Pattern Recognition (ICPR), 2010 20th International Conference on, pages 3822--3825, Aug 2010.

Digital Library

[23]

Justin Mathew, Stéphane Huot, and Alan Blum. A morphological analysis of audio objects and their control methods for 3d audio. In Proc. of NIME, 2014.

[24]

Frank Melchior, S Brix, and D De Vries. Zur kombination von wellenfeldsynthese mit monoskopischer und stereoskopischer bildwiedergabe. FORTSCHRITTE DER AKUSTIK, 31(1):207, 2005.

[25]

Frank Melchior, Sandra Brix, Thomas Sporer, Thomas Roder, and Beate Klehs. Wave field syntheses in combination with 2d video projection. In Audio Engineering Society Conference: 24th International Conference: Multichannel Audio, The New Reality. Audio Engineering Society, 2003.

[26]

Frank Melchior, Tobias Laubach, and Diemer De Vries. Authoring and user interaction for the production of wave field synthesis content in an augmented reality system. In Proceedings of the 4th IEEE/ACM International Symposium on Mixed and Augmented Reality, pages 48--51. IEEE Computer Society, 2005.

Digital Library

[27]

Frank Melchior, Chris Pike, Matthew Brooks, and Stuart Grace. On the use of a haptic feedback device for sound source control in spatial audio systems. In Audio Engineering Society Convention 134. Audio Engineering Society, 2013.

[28]

Frank Melchior and Sascha Spors. Spatial audio reproduction: from theory to production. In tutorial, 129th Convention of the AES, 2010.

[29]

Alexander Müller and Rudolf Rabenstein. Physical Modeling for Spatial Sound Synthesis. In Proc. Intl. Conf. Digital Audio Effects (DAFx), 2009.

[30]

Jörg Müller, Matthias Geier, Christina Dicke, and Sascha Spors. The boomroom: mid-air direct interaction with virtual sound sources. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 247--256. ACM, 2014.

Digital Library

[31]

G. Odowichuk, S. Trail, P. Driessen, W. Nie, and W. Page. Sensor fusion: Towards a fully expressive 3d music control interface. In Communications, Computers and Signal Processing (PacRim), 2011 IEEE Pacific Rim Conference on, pages 836--841, Aug 2011.

[32]

Tabitha C Peck, Henry Fuchs, and Mary C Whitton. Evaluation of reorientation techniques and distractors for walking in large virtual environments. Visualization and Computer Graphics, IEEE Transactions on, 15(3):383--394, 2009.

Digital Library

[33]

Tabitha C Peck, Henry Fuchs, and Mary C Whitton. Improved redirection with distractors: A large-scale-real-walking locomotion interface and its effect on navigation in virtual environments. In Virtual Reality Conference (VR), 2010 IEEE, pages 35--38. IEEE, 2010.

Digital Library

[34]

Tapani Pihlajamäki, Olli Santala, and Ville Pulkki. Synthesis of Spatially Extended Virtual Source with Time-Frequency Decomposition of Mono Signals. Journal of the Audio Engineering Society, 62(7/8):467--484, July/August 2014.

[35]

E. Potetsianakis, E. Ksylakis, and G. Triantafyllidis. A kinect-based framework for better user experience in real-time audiovisual content manipulation. In Telecommunications and Multimedia (TEMU), 2014 International Conference on, pages 238--242, July 2014.

[36]

Ville Pulkki. Virtual sound source positioning using vector base amplitude panning. Journal of the Audio Engineering Society, 45(6):456--466, 1997.

[37]

Ville Pulkki, Mikko-Ville Laitinen, and Cumhur Erkut. Efficient spatial sound synthesis for virtual worlds. In Proc. AES Intl. Conf, London, UK, 2009.

[38]

Jan P Springer, Christoph Sladeczek, Martin Scheffler, Jan Hochstrate, Frank Melchior, and Bernd Fröhlich. Combining wave field synthesis and multi-viewer stereo displays. In Virtual Reality Conference, 2006, pages 237--240. IEEE, 2006.

Digital Library

[39]

M. Van den Bergh, D. Carton, R. de Nijs, N. Mitsou, C. Landsiedel, K. Kuehnlenz, D. Wollherr, L. Van Gool, and M. Buss. Real-time 3d hand gesture interaction with a robot for understanding directions from humans. In RO-MAN, 2011 IEEE, pages 357--362, July 2011.

[40]

N. Vidakis, M. Syntychakis, G. Triantafyllidis, and D. Akoumianakis. Multimodal natural user interaction for multiple applications: The gesture - voice example. In Telecommunications and Multimedia (TEMU), 2012 International Conference on, pages 208--213, July 2012.

[41]

Dan Xu, Yen-Lun Chen, Chuan Lin, Xin Kong, and Xinyu Wu. Real-time dynamic gesture recognition system based on depth perception for robot navigation. In Robotics and Biomimetics (ROBIO), 2012 IEEE International Conference on, pages 689--694, Dec 2012.

Cited By

Anne de Haas ELee L(2022)Deceiving Audio Design in Augmented Environments : A Systematic Review of Audio Effects in Augmented Reality2022 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct57072.2022.00018(36-43)Online publication date: Oct-2022
https://doi.org/10.1109/ISMAR-Adjunct57072.2022.00018
Lakka EPapadaki MBrutzman DPuk RMalamos A(2021)X3D Audio Graph for the consistent declarative representation of the W3C Audio APIProceedings of the 26th International Conference on 3D Web Technology10.1145/3485444.3487645(1-5)Online publication date: 8-Nov-2021
https://dl.acm.org/doi/10.1145/3485444.3487645
Salselas IPenha RBernardes G(2020)Sound design inducing attention in the context of audiovisual immersive environmentsPersonal and Ubiquitous Computing10.1007/s00779-020-01386-3Online publication date: 14-Apr-2020
https://doi.org/10.1007/s00779-020-01386-3
Show More Cited By

Index Terms

Spatial Sound and Multimodal Interaction in Immersive Environments
1. Applied computing
  1. Arts and humanities
    1. Sound and music computing
2. Information systems
  1. Information retrieval
    1. Specialized information retrieval
      1. Multimedia and multimodal retrieval
        Music retrieval
  2. Information systems applications
    1. Multimedia information systems

Recommendations

Spatial sound localization in an augmented reality environment
OZCHI '06: Proceedings of the 18th Australia conference on Computer-Human Interaction: Design: Activities, Artefacts and Environments

Augmented Reality (AR), the overlay of virtual images onto the real world, is an increasingly popular technique for developing new human-computer interfaces. As human navigation and orientation in different environments depend on both visual and ...
An experimental study of spatial sound usefulness in searching and navigating through AR environments

This paper presents an experimental study of spatial sound usefulness in searching and navigating through augmented reality environments. Participants were asked to find three objects hidden within no-sound and spatial sound AR environments. The ...
Extending X3D Realism with Audio Graphs, Acoustic Properties and 3D Spatial Sound
Web3D '20: Proceedings of the 25th International Conference on 3D Web Technology

A fundamental requirement for both realistic modeling and immersive presence is spatial audio, correctly rendering the presentation of each object with aural characteristics. Auditory attributes involve perceived directions, distances, and the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

AM '15: Proceedings of the Audio Mostly 2015 on Interaction With Sound

October 2015

250 pages

ISBN:9781450338967

DOI:10.1145/2814895

Conference Chair:
George Kalliris
Aristotle University of Thessaloniki
,
Program Chair:
Charalampos Dimoulas
Aristotle University of Thessaloniki

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 October 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

AM '15

AM '15: Audio Mostly 2015

October 7 - 9, 2015

Thessaloniki, Greece

Acceptance Rates

Overall Acceptance Rate 177 of 275 submissions, 64%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
471
Total Downloads

Downloads (Last 12 months)51
Downloads (Last 6 weeks)1

Reflects downloads up to 18 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Anne de Haas ELee L(2022)Deceiving Audio Design in Augmented Environments : A Systematic Review of Audio Effects in Augmented Reality2022 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct57072.2022.00018(36-43)Online publication date: Oct-2022
https://doi.org/10.1109/ISMAR-Adjunct57072.2022.00018
Lakka EPapadaki MBrutzman DPuk RMalamos A(2021)X3D Audio Graph for the consistent declarative representation of the W3C Audio APIProceedings of the 26th International Conference on 3D Web Technology10.1145/3485444.3487645(1-5)Online publication date: 8-Nov-2021
https://dl.acm.org/doi/10.1145/3485444.3487645
Salselas IPenha RBernardes G(2020)Sound design inducing attention in the context of audiovisual immersive environmentsPersonal and Ubiquitous Computing10.1007/s00779-020-01386-3Online publication date: 14-Apr-2020
https://doi.org/10.1007/s00779-020-01386-3
Bala PMasu RNisi VNunes NBrewster SFitzpatrick GCox AKostakos V(2019)"When the Elephant Trumps"Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300925(1-13)Online publication date: 2-May-2019
https://dl.acm.org/doi/10.1145/3290605.3300925
Vryzas NKotsakis RDimoulas CKalliris GFagerlönn J(2016)Investigating Multimodal Audiovisual Event Detection and LocalizationProceedings of the Audio Mostly 201610.1145/2986416.2986426(97-104)Online publication date: 4-Oct-2016
https://dl.acm.org/doi/10.1145/2986416.2986426
Kardaris NRodomagoulakis IPitsikalis VArvanitakis AMaragos PHanjalic ASnoek CWorring MBulterman DHuet BKelliher AKompatsiaris YLi J(2016)A Platform for Building New Human-Computer Interface Systems that Support Online Automatic Recognition of Audio-Gestural CommandsProceedings of the 24th ACM international conference on Multimedia10.1145/2964284.2973794(1169-1173)Online publication date: 1-Oct-2016
https://dl.acm.org/doi/10.1145/2964284.2973794

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents