research-article

Efficient and practical audio-visual rendering for games using crossmodal perception

Authors:

Nicolas Bonneel,

Michael Wimmer,

Manuel Asselot,

George DrettakisAuthors Info & Claims

I3D '09: Proceedings of the 2009 symposium on Interactive 3D graphics and games

Pages 177 - 182

https://doi.org/10.1145/1507149.1507178

Published: 27 February 2009 Publication History

Abstract

Interactive applications such as computer games, are inherently audio visual, requiring high-quality rendering of complex 3D audio soundscapes and graphics environments. A frequent source of audio events is impact sounds, typically generated with physics engines. In this paper, we first present an optimization allowing efficient usage of impact sounds in a unified audio rendering pipeline, also including prerecorded sounds. We also exploit a recent result on audio-visual crossmodal perception to introduce a new level-of-detail selection algorithm, which jointly chooses the quality level of audio and graphics rendering. We have integrated these two techniques as a comprehensive crossmodal audio-visual rendering pipeline in a home-grown game engine, thus demonstrating the potential utility of our approach.

Supplementary Material

JPG File (p177-grelaud.jpg)

Download
5.38 KB

AVI File (p177-grelaud.avi)

Download
77.77 MB

References

[1]

Agarwal, S., Ramamoorthi, R., Belongie, S., and Jensen, H. W. 2003. Structured importance sampling of environment maps. In ACM Trans. on Graphics (Proc. of SIGGRAPH), vol. 22, 605--612.

Digital Library

[2]

Bonneel, N., Drettakis, G., Tsingos, N., Viaud-Delmon, I., and James, D. 2008. Fast modal sounds with scalable frequency-domain synthesis. ACM Trans. on Graphics (Proc. of SIGGRAPH) 27, 3 (August), 1--9.

Digital Library

[3]

Bonneel, N., Suied, C., Viaud-Delmon, I., and Drettakis, G. 2009. Bimodal perception of audio-visual material properties for virtual environments. ACM Transactions on Applied Perception (Accepted with minor revisions).

Digital Library

[4]

Chiu, D.-F. 2008. Penta G - A Game Engine for Real-Time Rendering Research. Master's thesis, Institute of Computer Graphics and Algorithms, Vienna University of Technology.

[5]

Fleming, R. W., Dror, R. O., and Adelson, E. H. 2003. Real-world illumination and the perception of surface reflectance properties. Journal of Vision 3, 5 (July), 347--368.

[6]

Funkhouser, T., Min, P., and Carlbom, I. 1999. Real-time acoustic modeling for distributed virtual environments. In Proc. of ACM SIGGRAPH 99, 365--374.

Digital Library

[7]

Funkhouser, T., Tsingos, N., Carlbom, I., Elko, G., Sondhi, M., West, J., Pingali, G., Min, P., and Ngan, A. 2004. A beam tracing method for interactive architectural acoustics. The Journal of the Acoustical Society of America (JASA), 2003. 115, 2 (February), 739--756.

[8]

Lokki, T., Savioja, L., Väänänen, R., Huopaniemi, J., and Takala, T. 2002. Creating interactive virtual auditory environments. IEEE Comput. Graph. Appl. 22, 4, 49--57.

Digital Library

[9]

Luebke, D., and Hallen, B. 2001. Perceptually driven simplification for interactive rendering. In Proc. of EG Workshop on Rendering 2001, 223--234.

Digital Library

[10]

Luebke, D., Watson, B., Cohen, J. D., Reddy, M., and Varshney, A. 2002. Level of Detail for 3D Graphics. Elsevier Science Inc., New York, NY, USA.

Digital Library

[11]

Moeck, T., Bonneel, N., Tsingos, N., Drettakis, G., Viaud-Delmon, I., and Alloza, D. 2007. Progressive perceptual audio rendering of complex scenes. In ACM SIGGRAPH Symp. on Interactive 3D Graphics and Games (I3D), 189--196.

Digital Library

[12]

Myszkowski, K. 1998. The Visible Differences Predictor: applications to global illumination problems. In Proc. of EG Workshop on Rendering 1998, 223--236.

[13]

Pellacini, F., Ferwerda, J. A., and Greenberg, D. P. 2000. Toward a psychophysically-based light reflection model for image synthesis. In Proc. of ACM SIGGRAPH 00, 55--64.

Digital Library

[14]

Raghuvanshi, N., and Lin, M. C. 2006. Interactive sound synthesis for large scale environments. In ACM SIGGRAPH Symp. on Interactive 3D Graphics and Games (I3D), 101--108.

Digital Library

[15]

Raghuvanshi, N., and Lin, M. C. 2007. Physically based sound synthesis for large-scale virtual environments. IEEE Comput. Graph. Appl. 27, 1, 14--18.

Digital Library

[16]

Ramanarayanan, G., Ferwerda, J., Walter, B., and Bala, K. 2007. Visual equivalence: Towards a new standard for image fidelity. ACM Trans. on Graphics (Proc. of SIGGRAPH) (August), 76.

Digital Library

[17]

Ramasubramanian, M., Pattanaik, S. N., and Greenberg, D. P. 1999. A perceptually based physical error metric for realistic image synthesis. In Proc. of ACM SIGGRAPH 99, 73--82.

Digital Library

[18]

Sloan, P.-P., Luna, B., and Snyder, J. 2005. Local, deformable precomputed radiance transfer. In ACM Trans. on Graphics (Proc. of SIGGRAPH), vol. 24, 1216--1224.

Digital Library

[19]

Spence, C., and Driver, J. 2004. Crossmodal Space and Cross-modal Attention. Oxford University Press, USA, June.

[20]

Tsingos, N., Gallo, E., and Drettakis, G. 2004. Perceptual audio rendering of complex virtual environments. ACM Trans. on Graphics (Proc. of SIGGRAPH) 23, 3 (July), 249--258.

Digital Library

[21]

van den Doel, K., and Pai, D. K. 1998. The sounds of physical shapes. Presence 7, 4, 382--395.

Digital Library

[22]

van den Doel, K., and Pai, D. K. 2003. Modal synthesis for vibrating objects. Audio Anecdotes.

[23]

Vangorp, P., Laurijssen, J., and Dutré, P. 2007. The influence of shape on the perception of material reflectance. ACM Trans. on Graphics (Proc. of SIGGRAPH) 26, 3 (August), 77.

Digital Library

[24]

Williams, N., Luebke, D., Cohen, J. D., Kelley, M., and Schubert, B. 2003. Perceptually guided simplification of lit, textured meshes. In ACM SIGGRAPH Symp. on Interactive 3D Graphics and Games (I3D), 113--121.

Digital Library

Cited By

Zheng JXiao CLi MLi ZQian FLiu WWu XEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)ParliRobo: Participant Lightweight AI Robots for Massively Multiplayer Online Games (MMOGs)Proceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3613764(9093-9102)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3613764
Chebi H(2021)Unified Approach to Augmented RealityImplementing Augmented Reality Into Immersive Virtual Learning Environments10.4018/978-1-7998-4222-4.ch004(74-88)Online publication date: 2021
https://doi.org/10.4018/978-1-7998-4222-4.ch004
Malpica SSerrano AAllue MBedia MMasia B(2019)Crossmodal perception in virtual realityMultimedia Tools and Applications10.1007/s11042-019-7331-z79:5-6(3311-3331)Online publication date: 26-Feb-2019
https://doi.org/10.1007/s11042-019-7331-z
Show More Cited By

Recommendations

Bimodal perception of audio-visual material properties for virtual environments

High-quality rendering of both audio and visual material properties is very important in interactive virtual environments, since convincingly rendered materials increase realism and the sense of immersion. We studied how the level of detail of auditory ...
Lighting System for Visual Perception Enhancement in Volume Rendering

We introduce a lighting system that enhances the visual cues in a rendered image for the perception of 3D volumetric objects. We divide the lighting effects into global and local effects, and deploy three types of directional lights: the key light and ...
Pipeline rendering: interaction and realism through hardware-based multi-pass rendering

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

I3D '09: Proceedings of the 2009 symposium on Interactive 3D graphics and games

February 2009

253 pages

ISBN:9781605584294

DOI:10.1145/1507149

Conference Chairs:
Eric Haines
Autodesk, Inc.
,
Morgan McGuire
Williams College
,
Daniel G. Aliaga
Purdue University
,
Manuel M. Oliveira
Universidade Federal do Rio Grande do Sul - UFRGS

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 February 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Sixth Framework Programme

Conference

I3D '09

Sponsor:

SIGGRAPH

I3D '09: Symposium on Interactive 3D Graphics and Games

February 27 - March 1, 2009

Massachusetts, Boston

Acceptance Rates

Overall Acceptance Rate 148 of 485 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
494
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)0

Reflects downloads up to 23 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zheng JXiao CLi MLi ZQian FLiu WWu XEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)ParliRobo: Participant Lightweight AI Robots for Massively Multiplayer Online Games (MMOGs)Proceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3613764(9093-9102)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3613764
Chebi H(2021)Unified Approach to Augmented RealityImplementing Augmented Reality Into Immersive Virtual Learning Environments10.4018/978-1-7998-4222-4.ch004(74-88)Online publication date: 2021
https://doi.org/10.4018/978-1-7998-4222-4.ch004
Malpica SSerrano AAllue MBedia MMasia B(2019)Crossmodal perception in virtual realityMultimedia Tools and Applications10.1007/s11042-019-7331-z79:5-6(3311-3331)Online publication date: 26-Feb-2019
https://doi.org/10.1007/s11042-019-7331-z
Harvey CBashford‐Rogers TDebattista KDoukakis EChalmers A(2017)Olfaction and Selective RenderingComputer Graphics Forum10.1111/cgf.1329537:1(350-362)Online publication date: 14-Sep-2017
https://doi.org/10.1111/cgf.13295
Doukakis EDebattista KHarvey CBashford‐Rogers TChalmers A(2017)Audiovisual Resource Allocation for Bimodal Virtual EnvironmentsComputer Graphics Forum10.1111/cgf.1325837:1(172-183)Online publication date: 14-Jul-2017
https://doi.org/10.1111/cgf.13258
Weier MStengel MRoth TDidyk PEisemann EEisemann MGrogorick SHinkenjann AKruijff EMagnor MMyszkowski KSlusallek P(2017)Perception-driven Accelerated RenderingComputer Graphics Forum10.1111/cgf.1315036:2(611-643)Online publication date: 1-May-2017
https://dl.acm.org/doi/10.1111/cgf.13150
Harvey CDebattista KBashford-Rogers TChalmers A(2017)Multi-Modal Perception for Selective RenderingComputer Graphics Forum10.1111/cgf.1279336:1(172-183)Online publication date: 1-Jan-2017
https://dl.acm.org/doi/10.1111/cgf.12793
Langlois TAn SJin KJames D(2014)Eigenmode compression for modal sound modelsACM Transactions on Graphics10.1145/2601097.260117733:4(1-9)Online publication date: 27-Jul-2014
https://dl.acm.org/doi/10.1145/2601097.2601177
Hulusic VDebattista KChalmers A(2013)Smoothness perceptionThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-012-0760-629:11(1159-1172)Online publication date: 1-Nov-2013
https://dl.acm.org/doi/10.1007/s00371-012-0760-6
Hulusic VHarvey CDebattista KTsingos NWalker SHoward DChalmers A(2012)Acoustic Rendering and Auditory–Visual Cross-Modal Perception and InteractionComputer Graphics Forum10.1111/j.1467-8659.2011.02086.x31:1(102-131)Online publication date: 1-Feb-2012
https://dl.acm.org/doi/10.1111/j.1467-8659.2011.02086.x
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents