Machine learning-based strategies for streaming and experiencing 3DoF virtual reality: research proposal
Pages 398 - 402
Abstract
This paper contains the research proposal of Quentin Guimard that was presented at the MMSys 2022 doctoral symposium.
The development of 360° videos experienced in virtual reality (VR) is hindered by network, cybersickness, and content perception challenges. Many levers have already been proposed to address these challenges, but separately. This PhD thesis intends to jointly address these issues by dynamically controlling levers and making quality decisions, with a view to improving the VR streaming experience.
This paper describes the steps necessary to the building of such approach, by separating work that has already been achieved over the course of this PhD from tasks that are still left to do. First results are also presented.
References
[1]
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. 2018. Stochastic Variational Video Prediction. In Proceedings of the 6th International Conference on Learning Representations (ICLR). ICLR.
[2]
Rosa María Baños, Cristina Botella, Isabel Rubió, Soledad Quero, Azucena García-Palacios, and Mariano Luis Alcañiz Raya. 2008. Presence and Emotions in Virtual Environments: The Influence of Stereoscopy. Cyberpsychology & behavior : the impact of the Internet, multimedia and virtual reality on behavior and society 11 1 (2008), 1--8.
[3]
Wolfram Boucsein. 2012. Electrodermal activity, 2nd ed. Springer Science + Business Media, New York, NY, US. Pages: xviii, 618.
[4]
Margaret M. Bradley and Peter J. Lang. 1994. Measuring emotion: The self-assessment manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry 25, 1 (1994), 49--59.
[5]
Fang-Yi Chao, Cagri Ozcinar, and Aljosa Smolic. 2021. Transformer-based Long-Term Viewport Prediction in 360° Video: Scanpath is All You Need. In IEEE 23nd International Workshop on Multimedia Signal Processing (MMSP). IEEE.
[6]
Jinyu Chen, Xianzhuo Luo, Miao Hu, Di Wu, and Yipeng Zhou. 2021. Sparkle: User-Aware Viewport Prediction in 360-Degree Video Streaming. IEEE Transactions on Multimedia 23 (2021), 3853--3866.
[7]
Lovish Chopra, Sarthak Chakraborty, Abhijit Mondal, and Sandip Chakraborty. 2021. PARIMA: Viewport Adaptive 360-Degree Video Streaming. In Proceedings of the Web Conference 2021. ACM, 2379--2391.
[8]
Savino Dambra, Giuseppe Samela, Lucile Sassatelli, Romaric Pighetti, Ramon Aparicio-Pardo, and Anne-Marie Pinna-Déry. 2018. Film Editing: New Levers to Improve VR Streaming. In Proceedings of the 9th ACM Multimedia Systems Conference (MMSys '18). Association for Computing Machinery, New York, NY, USA, 27--39.
[9]
Erwan J. David, Jesús Gutiérrez, Antoine Coutrot, Matthieu Perreira Da Silva, and Patrick Le Callet. 2018. A dataset of head and eye movements for 360° videos. In Proceedings of the 9th ACM Multimedia Systems Conference (MMSys '18). ACM, New York, NY, USA, 432--437.
[10]
Matteo Diano, Alessia Celeghin, Arianna Bagnis, and Marco Tamietto. 2017. Amygdala Response to Emotional Stimuli without Awareness: Facts and Interpretations. Frontiers in Psychology 7 (2017).
[11]
Mark Draper, Erik Viirre, and Valerie Gawron. 2001. Effects of Image Scale and System Time Delay on Simulator Sickness within Head-Coupled Virtual Environments. Human Factors 43 (03 2001), 129--146.
[12]
Xiaoxiong Fan, Yun Cai, Yufei Yang, Tianxing Xu, Yike Li, Songhai Zhang, and Fanglue Zhang. 2021. Detection of scene-irrelevant head movements via eye-head coordination information. Virtual Reality & Intelligent Hardware 3 (2021), 14.
[13]
Anna Felnhofer, Oswald D. Kothgassner, Mareike Schmidt, Anna-Katharina Heinzle, Leon Beutl, Helmut Hlavacs, and Ilse Kryspin-Exner. 2015. Is Virtual Reality Emotionally Arousing? Investigating Five Emotion Inducing Virtual Park Scenarios. Int. J. Hum.-Comput. Stud. 82, C (oct 2015), 48--56.
[14]
Yu Guan, Chengyuan Zheng, Xinggong Zhang, Zongming Guo, and Junchen Jiang. 2019. Pano: Optimizing 360° Video Streaming with a Better Understanding of Quality Perception. In Proceedings of the ACM Special Interest Group on Data Communication (SIGCOMM '19). Association for Computing Machinery, New York, NY, USA, 394--407.
[15]
Quentin Guimard, Florent Robert, Camille Bauce, Aldric Ducreux, Lucile Sassatelli, Hui-Yin Wu, Marco Winckler, and Auriane Gros. 2022. PEM360: A dataset of 360° videos with continuous Physiological measurements, subjective Emotional ratings and Motion traces. In Proceedings of the 13th ACM Multimedia Systems Conference (MMSys '22). ACM.
[16]
Quentin Guimard and Lucile Sassatelli. 2022. Effects of Emotions on Head Motion Predictability in 360° Videos. In International Workshop on Immersive Mixed and Virtual Environment System (MMVE '22). ACM.
[17]
Quentin Guimard, Lucile Sassatelli, Francesco Marchetti, Federico Becattini, Lorenzo Seidenari, and Alberto Del Bimbo. 2022. Deep Variational Learning for Multiple Trajectory Prediction of 360° Head Movements. In Proceedings of the 13th ACM Multimedia Systems Conference (MMSys '22). ACM.
[18]
Xueshi Hou, Sujit Dey, Jianzhong Zhang, and Madhukar Budagavi. 2021. Predictive Adaptive Streaming to Enable Mobile 360-Degree and VR Experiences. IEEE Transactions on Multimedia 23 (2021), 716--731.
[19]
Han Hu, Zhimin Xu, Xinggong Zhang, and Zongming Guo. 2019. Optimal Viewport-Adaptive 360-Degree Video Streaming Against Random Head Movement. In 2019 IEEE International Conference on Communications (ICC). IEEE, Shanghai, China, 1--6.
[20]
Nuowen Kan, Chenglin Li, Caiyi Yang, Wenrui Dai, Junni Zou, and Hongkai Xiong. 2021. Uncertainty-aware robust adaptive video streaming with bayesian neural network and model predictive control. In Proceedings of the 31st ACM Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV '21). ACM, New York, NY, USA, 17--24.
[21]
Benjamin J. Li, Jeremy N. Bailenson, Adam Pines, Walter J. Greenleaf, and Leanne M. Williams. 2017. A Public Database of Immersive VR Videos with Corresponding Ratings of Arousal, Valence, and Correlations between Head Movements and Self Report Measures. Frontiers in Psychology 8 (Dec. 2017), 2116.
[22]
Yixiang Mao, Liyang Sun, Yong Liu, and Yao Wang. 2020. Low-latency FoV-adaptive Coding and Streaming for Interactive 360° Video Streaming. In Proceedings of the 28th ACM International Conference on Multimedia (MM '20). ACM, New York, NY, USA, 3696--3704.
[23]
Francesco Marchetti, Federico Becattini, Lorenzo Seidenari, and Alberto Del Bimbo. 2020. MANTRA: Memory Augmented Networks for Multiple Trajectory Prediction. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Seattle, WA, USA, 7141--7150.
[24]
Afshin Taghavi Nasrabadi, Aliehsan Samiei, Anahita Mahzari, Ryan P. McMahan, Ravi Prakash, Mylène C. Q. Farias, and Marcelo M. Carvalho. 2019. A Taxonomy and Dataset for 360° Videos. In Proceedings of the 10th ACM Multimedia Systems Conference (MMSys '19). ACM, New York, NY, USA, 273--278.
[25]
Federica Pallavicini, Alessandro Pepe, and Maria Eleonora Minissi. 2019. Gaming in Virtual Reality: What Changes in Terms of Usability, Emotional Response and Sense of Presence Compared to Non-Immersive Video Games? Simulation & Gaming 50, 2 (2019), 136--159.
[26]
Jounsup Park, Philip A. Chou, and Jenq-Neng Hwang. 2019. Rate-utility optimized streaming of volumetric media for augmented reality. IEEE Journal on Emerging and Selected Topics in Circuits and Systems 9, 1 (2019), 149--162.
[27]
Thiago Porcino, Esteban Clua, Cristina Vasconcelos, Daniela Trevisan, and Luis Valente. 2016. Minimizing cyber sickness in head mounted display systems: design guidelines and applications. (11 2016).
[28]
Thiago Porcino, Daniela Trevisan, and Esteban Clua. 2019. DEMO: Using gameplay data to classify cybersickness level in virtual environments. 29--30.
[29]
Miguel Fabián Romero-Rondón, Lucile Sassatelli, Ramón Aparicio-Pardo, and Frédéric Precioso. 2021. TRACK: A New Method from a Re-examination of Deep Architectures for Head Motion Prediction in 360-degree Videos. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).
[30]
James A. Russell. 1980. A circumplex model of affect. Journal of Personality and Social Psychology 39, 6 (12 1980), 1161--1178.
[31]
Lucile Sassatelli, Marco Winckler, Thomas Fisichella, Ramon Aparicio, and AnneMarie Pinna-Déry. 2019. A New Adaptation Lever in 360° Video Streaming. In Proceedings of the 29th ACM Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV '19). ACM, New York, NY, USA, 37--42.
[32]
Lucile Sassatelli, Marco Winckler, Thomas Fisichella, Antoine Dezarnaud, Julien Lemaire, Ramon Aparicio-Pardo, and Daniela Trevisan. 2020. New interactive strategies for virtual reality streaming in degraded context of use. Computers & Graphics 86 (2020), 27--41.
[33]
Shashank Srikanth, Junaid Ahmed Ansari, Sarthak Sharma, et al. 2019. INFER: INtermediate representations for FuturE pRediction. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2019). IEEE.
[34]
Chenglei Wu, Zhi Wang, and Lifeng Sun. 2021. PAAS: a preference-aware deep reinforcement learning approach for 360° video streaming. In Proceedings of the 31st ACM Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV'21). ACM, Istanbul Turkey, 34--41.
[35]
Mai Xu, Yuhang Song, Jianyi Wang, Minglang Qiao, Liangyu Huo, and Zulin Wang. 2019. Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning Approach. IEEE Transactions on Pattern Analysis and Machine Intelligence 41, 11 (2019), 2693--2708.
[36]
Yanyu Xu, Yanbing Dong, Junru Wu, Zhengzhong Sun, Zhiru Shi, Jingyi Yu, and Shenghua Gao. 2018. Gaze Prediction in Dynamic 360° Immersive Videos. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 5333--5342.
[37]
Tong Xue, Abdallah El Ali, Gangyi Ding, and Pablo Cesar. 2021. Investigating the Relationship between Momentary Emotion Self-reports and Head and Eye Movements in HMD-based 360° VR Video Watching. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama Japan, 1--8.
[38]
Tong Xue, Abdallah El Ali, Tianyi Zhang, Gangyi Ding, and Pablo Cesar. 2021. CEAP-360VR: A Continuous Physiological and Behavioral Emotion Annotation Dataset for 360 VR Videos. IEEE Transactions on Multimedia (2021), 1--1.
[39]
Francis Y. Yan, Hudson Ayers, Chenzhi Zhu, Sadjad Fouladi, James Hong, Keyi Zhang, Philip Levis, and Keith Winstein. 2020. Learning in situ: a randomized experiment in video streaming. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20). USENIX Association, Santa Clara, CA, 495--511.
[40]
Li Yang, Mai Xu, Yichen Guo, Xin Deng, Fangyuan Gao, and Zhenyu Guan. 2021. Hierarchical Bayesian LSTM for Head Trajectory Prediction on Omnidirectional Images. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).
[41]
Ran Zhang, Jiang Liu, Fangqi Liu, Tao Huang, Qinqin Tang, Shangguang Wang, and F. Richard Yu. 2021. Buffer-Aware Virtual Reality Video Streaming with Personalized and Private Viewport Prediction. IEEE Journal on Selected Areas in Communications (2021).
[42]
Xue Zhang, Gene Cheung, Yao Zhao, Patrick Le Callet, Chunyu Lin, and Jack Z. G. Tan. 2021. Graph Learning Based Head Movement Prediction for Interactive 360 Video Streaming. IEEE Transactions on Image Processing 30 (2021), 4622--4636.
Index Terms
- Machine learning-based strategies for streaming and experiencing 3DoF virtual reality: research proposal
Recommendations
Interactive Augmented Live Virtual Reality Streaming: A Health Care Application
ICMHI '18: Proceedings of the 2nd International Conference on Medical and Health InformaticsVirtual Reality (VR) technology has been around for decades; however, we have probably only begun to realize the practical applications until recent years. With increase of computing power, decrease of cost and physical dimension, a new class of ...
Experiencing Virtual Reality Together: Social VR Use Case Study
TVX '18: Proceedings of the 2018 ACM International Conference on Interactive Experiences for TV and Online VideoAs Virtual Reality (VR) applications gain more momentum recently, the social and communication aspects of VR experiences become more relevant. In this paper, we present some initial results of understanding the type of applications and factors that ...
Experiencing 3D interactions in virtual reality and augmented reality
EUSAI '04: Proceedings of the 2nd European Union symposium on Ambient intelligenceWe demonstrate basic 2D and 3D interactions in both a Virtual Reality (VR) system, called the Personal Space Station, and an Augmented Reality (AR) system, called the Visual Interaction Platform. Since both platforms use identical (optical) tracking ...
Comments
Information & Contributors
Information
Published In
June 2022
432 pages
ISBN:9781450392839
DOI:10.1145/3524273
- General Chairs:
- Niall Murray,
- Gwendal Simon,
- Mylene Farias,
- Program Chairs:
- Irene Viola,
- Mario Montagud
Copyright © 2022 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 05 August 2022
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
MMSys '22
Sponsor:
Acceptance Rates
Overall Acceptance Rate 176 of 530 submissions, 33%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 141Total Downloads
- Downloads (Last 12 months)14
- Downloads (Last 6 weeks)1
Reflects downloads up to 03 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in