AI-assisted affective computing and spatial audio for interactive multimodal virtual environments: research proposal
Pages 372 - 377
Abstract
This paper contains the research proposal of Juan Antonio De Rus that was presented at the MMSys 2022 doctoral symposium. The use of virtual reality (VR) is growing every year. With the normalization of remote work it is to expect that the use of immersive virtual environments to support tasks as online meetings, education, etc, will grow even more. VR environments typically include multimodal content formats (synthetic content, video, audio, text) and even multi-sensory stimuli to provide an enriched user experience. In this context, Affective Computing (AC) techniques assisted by Artificial Intelligence (AI) become a powerful means to determine the user's perceived Quality of Experience (QoE). In the field of AC, we investigate a variety of tools to obtain accurate emotional analysis by using AI techniques applied on physiological data. In this doctoral study we have formulated a set of open research questions and objectives on which we plan to generate valuable contributions and knowledge in the field of AC, spatial audio, and multimodal interactive virtual environments, one of which is the creation of tools to automatically evaluate the QoE, even in real-time, which can provide valuable benefits both to service providers and consumers. For data acquisition we use sensors of different quality to study the scalability, reliability and replicability of our solutions, as clinical-grade sensors are not always within the reach of the average user.
References
[1]
Jai Shree Adhyaru and Charlotte Kemp. 2022. Virtual reality as a tool to promote wellbeing in the workplace. DIGITAL HEALTH 8 (2022), 20552076221084473. arXiv:https://doi.org/10.1177/20552076221084473 35284084.
[2]
Fabio Antonacci, Jason Filos, Mark R. P. Thomas, Emanuël A. P. Habets, Augusto Sarti, Patrick A. Naylor, and Stefano Tubaro. 2012. Inference of Room Geometry From Acoustic Impulse Responses. IEEE Transactions on Audio, Speech, and Language Processing 20, 10 (2012), 2683--2695.
[3]
Miguel Arevalillo-Herráez, Maximo Cobos, Sandra Roger, and Miguel Garcia-Pineda. 2019. Combining Inter-Subject Modeling with a Subject-Based Data Transformation to Improve Affect Recognition from EEG Signals. Sensors 19 (07 2019), 2999.
[4]
Resham Arya, Jaiteg Singh, and Ashok Kumar. 2021. A survey of multidisciplinary domains contributing to affective computing. Computer Science Review 40 (2021), 100399.
[5]
Christopher Ball, Kuo-Ting Huang, and Jess Francis. 2021. Virtual reality adoption during the COVID-19 pandemic: A uses and gratifications perspective. Telematics and Informatics 65 (2021), 101728.
[6]
Miguel Barreda-Ángeles and Tilo Hartmann. 2022. Psychological benefits of using social virtual reality platforms during the covid-19 pandemic: The role of social and spatial presence. Computers in Human Behavior 127 (2022), 107047.
[7]
Sandra Barteit, Lucia Lanfermann, Till Bärnighausen, Florian Neuhann, and Claudia Beiersmann. 2021. Augmented, Mixed, and Virtual Reality-Based Head-Mounted Devices for Medical Education: Systematic Review. JMIR Serious Games 9, 3 (8 Jul 2021), e29080.
[8]
Teah-Marie Bynion and Matthew Feldner. 2017. Self-Assessment Manikin. 1--3.
[9]
Manuela Chessa and Fabio Solari. 2021. The sense of being there during online classes: analysis of usability and presence in web-conferencing systems and virtual reality social platforms. Behaviour & Information Technology 40, 12 (2021), 1237--1249. arXiv:https://doi.org/10.1080/0144929X.2021.1957017
[10]
Lucio Ciabattoni, Francesco Ferracuti, Sauro Longhi, Lucia Pepa, Luca Romeo, and Federica Verdini. 2017. Real-time mental stress detection based on smartwatch. 110--111.
[11]
Henrique Debarba, Mario Montagud, Sylvain Chagué, Javier Herrero, Ignacio Lacosta, Sergi Langa, and Caecilia Charbonnier. 2022. Content format and quality of experience in virtual reality. Multimedia Tools and Applications (03 2022).
[12]
Krantiraditya Dhalmahapatra, J. Maiti, and O.B. Krishna. 2021. Assessment of virtual reality based safety training simulator for electric overhead crane operations. Safety Science 139 (2021), 105241.
[13]
Diego Di Carlo, Pinchas Tandeitnik, Cédric Foy, Antoine Deleforge, Nancy Bertin, and Sharon Gannot. 2021. dEchorate: a Calibrated Room Impulse Response Database for Echo-aware Signal Processing.
[14]
Vishnunarayan G Prabhu, Courtney Linder, Laura M Stanley, and Robert Morgan. 2019. An Affective Computing in Virtual Reality Environments for Managing Surgical Pain and Anxiety. In 2019 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR). 235--2351.
[15]
Andrzej Grabowski and Kamil Jach. 2021. The use of virtual reality in the training of professionals: with the example of firefighters. Computer Animation and Virtual Worlds 32, 2 (2021), e1981. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/cav.1981
[16]
Inc Grand View Research. 2022. virtual reality market share and trends report, 2021--2028 2022. https://www.grandviewresearch.com/industry-analysis/virtual-reality-vr-market Online; accessed 01 April 2022.
[17]
David Hamilton, Jim McKechnie, Edward Edgerton, and Claire Wilson. 2021. Immersive virtual reality as a pedagogical tool in education: a systematic literature review of quantitative learning outcomes and experimental design. Journal of Computers in Education 8, 1 (2021), 1--32.
[18]
Jiang Haowen, Sunitha Vimalesvaran, Bhone Myint Kyaw, and Lorainne Tudor Car. 2021. Virtual reality in medical students' education: a scoping review protocol. BMJ Open 11, 5 (2021). arXiv:https://bmjopen.bmj.com/content/11/5/e046986.full.pdf
[19]
International Data Corporation (IDC). 2022. ar/vr headset shipments grew dramatically in 2021. https://www.idc.com/getdoc.jsp?containerId=prUS48969722
[20]
Capgemini Research Institute. 2022. ar vr in operations 2022. https://www.capgemini.com/wp-content/uploads/2018/09/AR-VR-in-Operations1.pdf
[21]
Stamos Katsigiannis and Naeem Ramzan. 2018. DREAMER: A Database for Emotion Recognition Through EEG and ECG Signals From Wireless Low-cost Off-the-Shelf Devices. IEEE Journal of Biomedical and Health Informatics 22, 1 (2018), 98--107.
[22]
Sander Koelstra, Christian Muhl, Mohammad Soleymani, Jong-Seok Lee, Ashkan Yazdani, Touradj Ebrahimi, Thierry Pun, Anton Nijholt, and Ioannis Patras. 2012. DEAP: A Database for Emotion Analysis ;Using Physiological Signals. IEEE Transactions on Affective Computing 3, 1 (2012), 18--31.
[23]
Ayşenur Kılıç, Ashley Brown, Işıl Aras, Rita Hui, Jennifer Hare, Lyndsay D Hughes, and Lance M McCracken. 2021. Using Virtual Technology for Fear of Medical Procedures: A Systematic Review of the Effectiveness of Virtual Reality-Based Interventions. Annals of Behavioral Medicine 55, 11 (04 2021), 1062--1079. arXiv:https://academic.oup.com/abm/article-pdf/55/11/1062/40933759/kaab016.pdf
[24]
Sergi Fernández Langa, Mario Montagud Climent, Gianluca Cernigliaro, and David Rincón Rivera. 2022. Toward Hyper-Realistic and Interactive Social VR Experiences in Live TV Scenarios. IEEE Transactions on Broadcasting 68, 1 (2022), 13--32.
[25]
Ivan Lau, Ashu Gupta, and Zhonghua Sun. 2021. Clinical Value of Virtual Reality versus 3D Printing in Congenital Heart Disease. Biomolecules 11, 6 (2021).
[26]
Xiaoming Li, Haotian Zhou, Shengzun Song, Tian Ran, and Xiaolan Fu. 2005. The Reliability and Validity of the Chinese Version of Abbreviated PAD Emotion Scales. 513--518.
[27]
David B. Lindell, Gordon Wetzstein, and Vladlen Koltun. 2019. Acoustic Non-Line-Of-Sight Imaging. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 6773--6782.
[28]
Irene Martín-Morató, Maximo Cobos, and Francesc J. Ferri. 2020. Adaptive Distance-Based Pooling in Convolutional Neural Networks for Audio Event Classification. IEEE/ACM Transactions on Audio, Speech, and Language Processing 28 (2020), 1925--1935.
[29]
Cameron McCarthy, Nikhilesh Pradhan, Calum Redpath, and Andy Adler. 2016. Validation of the Empatica E4 wristband. In 2016 IEEE EMBS International Student Conference (ISC). 1--4.
[30]
Albert Mehrabian. 1996. Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in Temperament. Current Psychology 14 (1996), 261--292.
[31]
Juan Abdon Miranda-Correa, Mojtaba Khomami Abadi, Nicu Sebe, and Ioannis Patras. 2021. AMIGOS: A Dataset for Affect, Personality and Mood Research on Individuals and Groups. IEEE Transactions on Affective Computing 12, 2 (2021), 479--493.
[32]
Oana Mitruṭ, Gabriela Moise, Livia Petrescu, Alin Moldoveanu, Marius Leordeanu, and Florica Moldoveanu. 2019. Emotion Classification Based on Biophysical Signals and Machine Learning Techniques. Symmetry 12 (12 2019), 21.
[33]
Mario Montagud, Cristian Hurtado, Juan Rus, and Sergi Fernández. 2021. Subtitling 3D VR Content with Limited 6DoF: Presentation Modes and Guiding Methods. Applied Sciences 11 (08 2021), 7472.
[34]
Mario Montagud, Jie Li, Gianluca Cernigliario, Abdallah El Ali, Sergi Fernandez, and Pablo Cesar. 2022. Towards SocialVR: Evaluating a Novel Technology for Watching Videos Together. Accepted for publication in Virtual Reality (Springer).
[35]
Mario Montagud, Jaume Segura Garcia, J. Rus, and Rafael Fayos-Jordan. 2020. Towards an Immersive and Accessible Virtual Reconstruction of Theaters from the Early Modern: Bringing Back Cultural Heritage from the Past. 143--147.
[36]
Sihem Nita, Salim Bitam, Matthieu Heidet, and Abdelhamid Mellouk. 2022. A new data augmentation convolutional neural network for human emotion recognition based on ECG signals. Biomedical Signal Processing and Control 75 (2022), 103580.
[37]
Norma J. Ontiveros-Hernández, Miguel Pérez-Ramírez, and Yasmín Hernández. 2013. Virtual Reality and Affective Computing for Improving Learning. Res. Comput. Sci. 65 (2013), 121--131.
[38]
Robert Plutchik. 2001. The Nature of Emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice. American Scientist 89, 4 (2001), 344--350. http://www.jstor.org/stable/27857503
[39]
Anton Ratnarajah, Shi-Xiong Zhang, Meng Yu, Zhenyu Tang, Dinesh Manocha, and Dong Yu. 2021. FAST-RIR: Fast neural diffuse room impulse response generator.
[40]
David Rubin and Jennifer Talarico. 2009. A Comparison of Dimensional Models of Emotion: Evidence from Emotions, Prototypical Events, Autobiographical Memories, and Words. Memory (Hove, England) 17 (09 2009), 802--8.
[41]
James Russell. 1980. A Circumplex Model of Affect. Journal of Personality and Social Psychology 39 (12 1980), 1161--1178.
[42]
Karan Sharma, Claudio Castellini, Egon L. van den Broek, Alin Albu-Schäeffer, and Friedhelm Schwenker. 2019. A dataset of continuous affect annotations and physiological signals for emotion analysis. Scientific Data 6 (10 2019).
[43]
Mohammad Soleymani, Jeroen Lichtenauer, Thierry Pun, and Maja Pantic. 2012. A Multimodal Database for Affect Recognition and Implicit Tagging. IEEE Transactions on Affective Computing 3, 1 (2012), 42--55.
[44]
Yan Wang, Wei Song, Wei Tao, Antonio Liotta, Dawei Yang, Xinlei Li, Shuyong Gao, Yixuan Sun, Weifeng Ge, Wei Zhang, and Wenqiang Zhang. 2022. A systematic review on affective computing: emotion models, databases, and recent advances. Information Fusion 83--84 (2022), 19--52.
[45]
David Watson and Auke Tellegen. 1985. Toward a consensual structure of mood. Psychological bulletin 98, 2 (1985), 219.
[46]
Shen Zhang, Zhiyong Wu, Helen Meng, and Lianhong Cai. 2010. Facial Expression Synthesis Based on Emotion Dimensions for Affective Talking Avatar. Vol. 2010. 109--132.
[47]
Sicheng Zhao, Shangfei Wang, Mohammad Soleymani, Dhiraj Joshi, and Qiang Ji. 2019. Affective Computing for Large-Scale Heterogeneous Multimedia Data: A Survey. ACM Trans. Multimedia Comput. Commun. Appl. 15, 3s, Article 93 (dec 2019), 32 pages.
[48]
Franz Zotter and Matthias Frank. 2019. Ambisonics: A Practical 3D Audio Theory for Recording, Studio Production, Sound Reinforcement, and Virtual Reality.
Index Terms
- AI-assisted affective computing and spatial audio for interactive multimodal virtual environments: research proposal
Recommendations
Spatial Audio in Virtual Reality: A systematic review
SVR '23: Proceedings of the 25th Symposium on Virtual and Augmented RealitySpatial Audio represents sound across a full sphere around the listener’s position and is being widely implemented in Virtual Reality and games. However there’s not an extensive body of work in the full benefits of immersive audio in virtual reality, as ...
Captivating the Senses: Crafting a Multisensory Virtual Experience for Enhanced Realism
UbiComp/ISWC '23 Adjunct: Adjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable ComputingThis paper explores the challenges and opportunities in achieving a high level of realism and presence in Virtual Reality (VR) experiences. Realism and presence, which refer to the level of authenticity of the virtual environment and the user’s ...
Subjective Evaluation of an Olfaction Enhanced Immersive Virtual Reality Environment
AltMM '17: Proceedings of the 2nd International Workshop on Multimedia Alternate RealitiesRecent research efforts have reported findings on user Quality of Experience (QoE) of immersive virtual reality (VR) experiences. Truly immersive multimedia experiences also include multisensory components such as factional, tactile etc., in addition to ...
Comments
Information & Contributors
Information
Published In
June 2022
432 pages
ISBN:9781450392839
DOI:10.1145/3524273
- General Chairs:
- Niall Murray,
- Gwendal Simon,
- Mylene Farias,
- Program Chairs:
- Irene Viola,
- Mario Montagud
Copyright © 2022 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 05 August 2022
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
MMSys '22
Sponsor:
Acceptance Rates
Overall Acceptance Rate 176 of 530 submissions, 33%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 176Total Downloads
- Downloads (Last 12 months)61
- Downloads (Last 6 weeks)3
Reflects downloads up to 15 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in