Augmented Immersive Viewing and Listening Experience Based on Arbitrarily Angled Interactive Audiovisual Representation
Pages 79 - 83
Abstract
We propose an arbitrarily angled interactive audiovisual representation technique that combines a unique sound field synthesis with visual representation in order to augment the possibility of interactive immersive viewing experiences on mobile devices. This technique can synthesize two-channel stereo sound with constant stereo width having an arbitrary angle range from minimum 30 to maximum 360 degrees centering on an arbitrary direction from multi-channel surround sound. The visual representation can be chosen either equirectangular projection or stereographic projection. The developed video player app allows users to enjoy arbitrarily angled 360-degree videos by manipulating the touchscreen, and the stereo sound and the visual representation changes in terms of its spatial synchronization depending on the view. The app was released as a demonstration, and its acceptability and worth were investigated through interviews and subjective assessment tests. The app has been well received, and to date, more than 30 pieces of content have been produced in multiple genres, with a total of more than 200,000 views.
References
[1]
Christoph Anthes, Rubén Jesús García-Hernández, Markus Wiedemann, and Dieter Kranzlmüller. 2016. State of the art of virtual reality technology. In 2016 IEEE Aerospace Conference. 1–19. https://doi.org/10.1109/AERO.2016.7500674
[2]
R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier, and B. MacIntyre. 2001. Recent advances in augmented reality. IEEE Computer Graphics and Applications 21, 6 (2001), 34–47. https://doi.org/10.1109/38.963459
[3]
Stéphanie Bertet, Jérôme Daniel, and Sébastien Moreau. 2006. 3D Sound Field Recording with Higher Order Ambisonics - Objective Measurements and Validation of Spherical Microphone. In Audio Engineering Society Convention 120. http://www.aes.org/e-lib/browse.cfm?elib=13661
[4]
Frederick Brooks, Jr. 1999. What’s Real About Virtual Reality?IEEE Computer Graphics and Applications 19 (12 1999), 16–27. https://doi.org/10.1109/38.799723
[5]
Rep. ITU-R BS.2159-7. 2015. Multichannel sound technology in home and broadcasting applications.
[6]
John David N. Dionisio, William G. Burns III, and Richard Gilbert. 2013. 3D Virtual Worlds and the Metaverse: Current Status and Future Possibilities. ACM Comput. Surv. 45, 3, Article 34 (July 2013), 38 pages. https://doi.org/10.1145/2480741.2480751
[7]
Roger K. Furness. 1990. Ambisonics-An Overview. In Audio Engineering Society Conference: 8th International Conference: The Sound of Audio. http://www.aes.org/e-lib/browse.cfm?elib=5417
[8]
Toshiharu Horiuchi, Sumaru Niida, and Yasuhiro Takishima. 2019. OtonoVR: Arbitrarily Angled Audio-Visual VR Experience Using Selective Synthesis Sound Field Technique. In Proceedings of the 27th ACM International Conference on Multimedia (Nice, France) (MM ’19). ACM, New York, NY, USA, 2211–2213. https://doi.org/10.1145/3343031.3350602
[9]
Toshiharu Horiuchi, Hiroshi Sankoh, Tsuneo Kato, and Sei Naito. 2012. Interactive Music Video Application for Smartphones Based on Free-Viewpoint Video and Audio Rendering. In Proceedings of the 20th ACM International Conference on Multimedia (Nara, Japan) (MM ’12). ACM, New York, NY, USA, 1293–1294. https://doi.org/10.1145/2393347.2396449
[10]
J. Blauert. 1999. Spatial hearing – The psychophysics of human sound localization. The MIT Press.
[11]
T. Kanade, P. Rander, and P.J. Narayanan. 1997. Virtualized reality: constructing virtual worlds from real scenes. IEEE MultiMedia 4, 1 (1997), 34–47. https://doi.org/10.1109/93.580394
[12]
Tao Ni, G.S. Schmidt, O.G. Staadt, M.A. Livingston, R. Ball, and R. May. 2006. A Survey of Large High-Resolution Display Technologies, Techniques, and Applications. In IEEE Virtual Reality Conference (VR 2006). 223–236. https://doi.org/10.1109/VR.2006.20
[13]
Markus Noisternig, Alois Sontacchi, Thomas Musil, and Robert Holdrich. 2003. A 3D Ambisonic Based Binaural Sound Reproduction System. In Audio Engineering Society Conference: 24th International Conference: Multichannel Audio, The New Reality. http://www.aes.org/e-lib/browse.cfm?elib=12314
[14]
Mehrdad Panahpour Tehrani, Kenta Niwa, Norishige Fukushima, Yasushi Hirano, Toshiaki Fujii, Masayuki Tanimoto, Kazuya Takeda, Kenji Mase, Akio Ishikawa, Shigeyuki Sakazawa, and Atsushi Koike. 2008. 3DAV integrated system featuring arbitrary listening-point and viewpoint generation. In 2008 IEEE 10th Workshop on Multimedia Signal Processing. 855–860. https://doi.org/10.1109/MMSP.2008.4665193
[15]
Shankar Shivappa, Martin Morrell, Deep Sen, Nils Peters, and S. M. Akramus Salehin. 2016. Efficient, Compelling, and Immersive VR Audio Experience Using Scene Based Audio/Higher Order Ambisonics. In Audio Engineering Society Conference: 2016 AES International Conference on Audio for Virtual and Augmented Reality. http://www.aes.org/e-lib/browse.cfm?elib=18493
[16]
T. Umayahara, H. Hokari, and S. Shimada. 2006. Stereo width control using interpolation and extrapolation of time-frequency representation. IEEE Transactions on Audio, Speech, and Language Processing 14, 4 (2006), 1364–1377. https://doi.org/10.1109/TASL.2006.872612
[17]
O. Yilmaz and S. Rickard. 2004. Blind separation of speech mixtures via time-frequency masking. IEEE Transactions on Signal Processing 52, 7 (2004), 1830–1847. https://doi.org/10.1109/TSP.2004.828896
Index Terms
- Augmented Immersive Viewing and Listening Experience Based on Arbitrarily Angled Interactive Audiovisual Representation
Recommendations
OtonoVR: Arbitrarily Angled Audio-visual VR Experience Using Selective Synthesis Sound Field Technique
MM '19: Proceedings of the 27th ACM International Conference on MultimediaWe present an arbitrarily angled audio-visual VR experience app called OtonoVR for 360-degree panoramic videos using our selective synthesis sound field technique. This technique can synthesize two-channel stereo sound with scaled stereo width having an ...
Experience Design of the Immersive Film: Panoramic View and Interactive Narratives
Design, User Experience, and Usability: Theory and PracticeAbstractWhen the immersive filmwork is experienced by the audiences with head-mounted display,the interactive experience of immersive images lies in that the audiences (users) need to be part of the story in terms of the visual perception experience and ...
Enhancing Audience Engagement Through Immersive 360-Degree Videos: An Experimental Study
Augmented Cognition: Intelligent TechnologiesAbstractThe use of 360-degree videos to engage audiences in diverse contexts is increasing. While 360-degree videos have the potential to create new value in enhancing audiences’ viewing experiences, they often decrease audience engagement by causing ...
Comments
Information & Contributors
Information
Published In
October 2023
858 pages
ISBN:9798400700552
DOI:10.1145/3577190
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 09 October 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
Conference
ICMI '23
Sponsor:
Acceptance Rates
Overall Acceptance Rate 453 of 1,080 submissions, 42%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 88Total Downloads
- Downloads (Last 12 months)88
- Downloads (Last 6 weeks)7
Reflects downloads up to 11 Aug 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format