research-article

Millimeter Wave and Free-space-optics for Future Dual-connectivity 6DOF Mobile Multi-user VR Streaming

Authors:

Jacob Chakareski,

Tanguy Ropitault,

Steve BlandinoAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications and Applications, Volume 19, Issue 2

Article No.: 57, Pages 1 - 25

https://doi.org/10.1145/3544494

Published: 06 February 2023 Publication History

Abstract

Dual-connectivity streaming is a key enabler of next-generation six Degrees Of Freedom (6DOF) Virtual Reality (VR) scene immersion. Indeed, using conventional sub-6 GHz WiFi only allows to reliably stream a low-quality baseline representation of the VR content, while emerging high-frequency communication technologies allow to stream in parallel a high-quality user viewport-specific enhancement representation that synergistically integrates with the baseline representation to deliver high-quality VR immersion. We investigate holistically as part of an entire future VR streaming system two such candidate emerging technologies, Free Space Optics (FSO) and millimeter-Wave (mmWave), that benefit from a large available spectrum to deliver unprecedented data rates. We analytically characterize the key components of the envisioned dual-connectivity 6DOF VR streaming system that integrates in addition edge computing and scalable 360° video tiling, and we formulate an optimization problem to maximize the immersion fidelity delivered by the system, given the WiFi and mmWave/FSO link rates, and the computing capabilities of the edge server and the users’ VR headsets. This optimization problem is mixed integer programming of high complexity and we formulate a geometric programming framework to compute the optimal solution at low complexity. We carry out simulation experiments to assess the performance of the proposed system using actual 6DOF navigation traces from multiple mobile VR users that we collected. Our results demonstrate that our system considerably advances the traditional state of the art and enables streaming of 8K-120 frames-per-second (fps) 6DOF content at high fidelity.

References

[1]

Jacob Chakareski. 2017. Drone networks for virtual human teleportation. In Proc. ACM Workshop on Micro Aerial Vehicle Networks, Systems, and Applications. 21–26.

[2]

J. G. Apostolopoulos, P. A. Chou, Bruce Culbertson, T. Kalker, M. D. Trott, and S. Wee. 2012. The road to immersive communication. Proc. IEEE 100, 4 (April 2012), 974–990.

[3]

Jacob Chakareski. 2019. UAV-IoT for next generation virtual reality. IEEE Transactions on Image Processing 28, 12 (2019), 5977–5990.

[4]

T. Stockhammer Champel, T. Fautier, E. Thomas, and R. Koenen. 2016. Quality requirements for VR. In Proc. 116th MPEG Meeting of ISO/IEC JTC1/SC29/WG11 MPEG, vol. 116, p. m39532.

[5]

Svilen Dimitrov and Harald Haas. 2015. Principles of LED Light Communications: Towards Networked Li-Fi. Cambridge University Press.

[6]

E. Calvanese Strinati, S. Barbarossa, J. L. Gonzalez-Jimenez, Dimitri Ktenas, Nicolas Cassiau, Luc Maret, and Cedric Dehos. 2019. 6G: The next frontier: From holographic messaging to artificial intelligence using subterahertz and visible light communication. IEEE Vehicular Technology Magazine 14, 3 (2019), 42–50. DOI:

[7]

Md Shaifur Rahman, Kai Zheng, and Himanshu Gupta. 2018. FSO-VR: Steerable free space optics link for virtual reality headsets. In Proc. ACM Workshop on Wearable Systems and Applications. 11–15.

[8]

J. Beysens, Q. Wang, A. Galisteo, D. Giustiniano, and S. Pollin. 2020. A cell-free networking system with visible light. IEEE/ACM Transactions on Networking 28, 2 (2020), 461–476. DOI:

Digital Library

[9]

S. Blandino, G. Mangraviti, C. Desset, et al. 2019. Multi-user hybrid MIMO at 60 GHz using 16-antenna transmitters. IEEE Transactions on Circuits and Systems I: Regular Papers 66, 2 (2019), 848–858. DOI:

[10]

J. Chakareski and P. Frossard. 2008. Distributed collaboration for enhanced sender-driven video streaming. IEEE Trans. Multimedia 10, 5 (Aug. 2008), 858–870.

Digital Library

[11]

Bo Begole. 2016. Why the Internet pipes will burst when virtual reality takes off. Forbes Magazine.

[12]

J. Chakareski, R. Aksu, X. Corbillon, G. Simon, and V. Swaminathan. 2018. Viewport-driven rate-distortion optimized 360 \(^\circ\) video streaming. In Proc. IEEE Int’l Conf. Communications.

[13]

X. Corbillon, A. Devlic, G. Simon, and J. Chakareski. 2017. Viewport-adaptive navigable 360-degree video delivery. In Proc. Int’l Conf. Communications. IEEE.

[14]

Mohammad Hosseini and Viswanathan Swaminathan. 2016. Adaptive 360 VR video streaming: Divide and conquer! In Proc. IEEE Int’l Symp. Multimedia.

[15]

Eduardo Cuervo, Krishna Chintalapudi, and Manikanta Kotaru. 2018. Creating the perfect illusion: What will it take to create life-like virtual reality headsets? In Proc. HotMobile. ACM, 7–12.

[16]

Ruiguang Zhong, Manni Wang, Zijian Chen, Luyang Liu, Yunxin Liu, Jiansong Zhang, Lintao Zhang, and Thomas Moscibroda. 2017. On building a programmable wireless high-quality virtual reality system using commodity hardware. In Proceedings of the 8th Asia-Pacific Workshop on Systems. ACM, 7.

Digital Library

[17]

Omid Abari, Dinesh Bharadia, Austin Duffield, and Dina Katabi. 2017. Enabling high-quality untethered virtual reality. In Proc. Symposium on Networked Systems Design and Implementation. 531–544.

[18]

Mahmudur Khan and Jacob Chakareski. 2019. Visible light communication for next generation untethered virtual reality systems. In Proc. IEEE International Conference on Communications Workshops. 1–6.

[19]

Alexander Clemm, Hemanth Torres Vega, Maria Kumar Ravuri, Tim Wauters, and Filip De Turck. 2020. Viewport-adaptive scalable multi-user virtual reality mobile-edge streaming. IEEE Communications Magazine 58, 1 (Jan. 2020), 93–99.

Digital Library

[20]

Zhi Liu, Qiyue Li, Xianfu Chen, Celimuge Wu, Susumu Ishihara, Jie Li, and Yusheng Ji. 2021. Point cloud video streaming: Challenges and solutions. IEEE Network Magazine 35, 5 (Sept. 2021), 202–209.

Digital Library

[21]

J. Chakareski and P. Frossard. 2006. Rate-distortion optimized distributed packet scheduling of multiple video streams over shared communication resources. IEEE Trans. Multimedia 8, 2 (April 2006), 207–218. Special issue on Distributed Media Technologies and Applications.

Digital Library

[22]

A. B. Reis, J. Chakareski, A. Kassler, and S. Sargento. 2010. Distortion optimized multi-service scheduling for next generation wireless mesh networks. In Proc. IEEE INFOCOM Int’l Workshop on Carrier-grade Wireless Mesh Networks. USA.

[23]

J. Chakareski, J. Apostolopoulos, W.-T. Tan, S. Wee, and B. Girod. 2004. Distortion chains for predicting the video distortion for general packet loss patterns. In Proc. Int’l Conf. Acoustics, Speech, and Signal Processing, Vol. 5. IEEE, 1001–1004.

[24]

J. Chakareski, J. Apostolopoulos, S. Wee, W.-T. Tan, and B. Girod. 2005. Rate-distortion hint tracks for adaptive video streaming. IEEE Trans. Circuits and Systems for Video Technology 15, 10 (Oct. 2005), 1257–1269. Special issue on Analysis and Understanding for Video Adaptation.

Digital Library

[25]

J. Chakareski and P. A. Chou. 2006. RaDiO edge: Rate-distortion optimized proxy-driven streaming from the network edge. IEEE/ACM Trans. Networking 14, 6 (Dec. 2006), 1302–1312.

Digital Library

[26]

J. Chakareski and P. A. Chou. 2004. Application layer error correction coding for rate-distortion optimized streaming to wireless clients. IEEE Trans. Communications 52, 10 (Oct. 2004), 1675–1687.

[27]

Ahmed Taha Hussein and Jaafar M. H. Elmirghani. 2015. 10 Gbps mobile visible light communication system employing angle diversity, imaging receivers, and relay nodes. OSA Journal of Optical Communications and Networking 7, 8 (2015), 718–735.

[28]

NVIDIA GeForce GTX 980. n.d. https://www.techpowerup.com/gpu-specs/geforce-gtx-980.c2621.

[29]

GPU Systems. n.d. https://www.supermicro.com/en/products/GPU/.

[30]

NVIDIA Video Encode and Decode GPU Support Matrix. n.d. https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new.

[31]

Jill M. Boyce, Yan Ye, Jianle Chen, and Adarsh K. Ramasubramonian. 2015. Overview of SHVC: Scalable extensions of the high efficiency video coding standard. IEEE Transactions on Circuits and Systems for Video Technology 26, 1 (2015), 20–34.

Digital Library

[32]

J. Chakareski. 2020. Viewport-adaptive scalable multi-user virtual reality mobile-edge streaming. IEEE Trans. Image Processing 29 (2020), 6330–6342.

[33]

QualComm Adreno 650 GPU. n.d. https://www.qualcomm.com/snapdragon/processors/comparison.

[34]

Gongxian Xu. 2014. Global optimization of signomial geometric programming problems. European Journal of Operational Research 233, 3 (2014), 500–510.

[35]

Stephen Boyd, Stephen P. Boyd, and Lieven Vandenberghe. 2004. Convex Optimization. Cambridge University Press.

[36]

Heinz Willebrand and Baksheesh S. Ghuman. 2002. Free Space Optics: Enabling Optical Connectivity in Today’s Networks. SAMS Publishing.

[37]

Zhihong Zeng, Mohammad Dehghani Soltani, Majid Safari, and Harald Haas. May 2019. Angle diversity receiver in LiFi cellular networks. In Proc. IEEE International Conference on Communications. 1–6.

[38]

D. G. Brennan. 2003. Linear diversity combining techniques. Proc. IEEE 91, 2 (2003), 331–356.

[39]

Alexander Maltsev, Andrey Pudeyev, Ingolf Karls, Ilya Bolotin, Gregory Morozov, Richard Weiler, Michael Peter, and Wilhelm Keusgen. 2014. Quasi-deterministic approach to mmWave channel modeling in a non-stationary environment. In 2014 IEEE Globecom Workshops (GC Wkshps’14). IEEE, 966–971.

[40]

Abraham P. Punnen and K. P. K. Nair. 1994. Improved complexity bound for the maximum cardinality bottleneck bipartite matching problem. Discrete Applied Mathematics 55, 1 (1994), 91–93.

Digital Library

[41]

Steve Blandino, Jelena Senic, Camillo Gentile, Derek Caudill, Jack Chuang, and Atiqa Kayani. 2022. Markov multi-beamtracking on 60 GHz mobile channel measurements. IEEE Open Journal of Vehicular Technology 3 (2022), 26–39. DOI:

[42]

Virtual Museum. n.d. https://assetstore.unity.com/packages/3d/environments/museum-117927.

[43]

SteamVR SDK. n.d.https://github.com/ValveSoftware/openvr.

[44]

Opentrack tracking. n.d. https://github.com/opentrack/opentrack.

[45]

M. Khan and J. Chakareski. n.d. NJIT 6DOF VR Navigation Dataset. https://www.jakov.org.

[46]

NIST and Università di Padova. n.d. Q-D realization software. https://github.com/wigig-tools/qd-realization.

[47]

Hany Assasa, Joerg Widmer, Tanguy Ropitault, and Nada Golmie. 2019. Enhancing the Ns-3 IEEE 802.11ad model fidelity: Beam codebooks, multi-antenna beamforming training, and quasi-deterministic mmwave channel. In Proceedings of the 2019 Workshop on Ns-3 (WNS3’19). Association for Computing Machinery, New York, NY, 33–40. DOI:

Digital Library

[48]

Hany Assasa, Joerg Widmer, Tanguy Ropitault, Anuraag Bodi, and Nada Golmie. 2019. High fidelity simulation of IEEE 802.11ad in ns-3 using a quasi-deterministic channel model. In Proceedings of the 2019 Workshop on Next-Generation Wireless with ns-3. 22–25.

Digital Library

[49]

Stefano Petrangeli, Viswanathan Swaminathan, Mohammad Hosseini, and Filip De Turck. 2017. Improving virtual reality streaming using HTTP/2. In Proc. Multimedia Systems Conference. ACM, 225–228.

Digital Library

Cited By

Wang LSingh SChakareski JHajiesmaili MSitaraman R(2024)BONES: Near-Optimal Neural-Enhanced Video StreamingProceedings of the ACM on Measurement and Analysis of Computing Systems10.1145/36560148:2(1-28)Online publication date: 29-May-2024
https://dl.acm.org/doi/10.1145/3656014
Chakareski JKhan M(2024)Live 360° Video Streaming to Heterogeneous Clients in 5G NetworksIEEE Transactions on Multimedia10.1109/TMM.2024.338291026(8860-8873)Online publication date: 2024
https://doi.org/10.1109/TMM.2024.3382910
Ravuri HStruye Jvan der Hooft JWauters TDe Turck FFamaey JTorres Vega M(2024)Context-Aware and Reliable Transport Layer Framework for Interactive Immersive Media Delivery Over Millimeter WaveJournal of Network and Systems Management10.1007/s10922-024-09845-532:4Online publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1007/s10922-024-09845-5
Show More Cited By

Index Terms

Millimeter Wave and Free-space-optics for Future Dual-connectivity 6DOF Mobile Multi-user VR Streaming

Recommendations

Wifi-VLC dual connectivity streaming system for 6DOF multi-user virtual reality
NOSSDAV '21: Proceedings of the 31st ACM Workshop on Network and Operating Systems Support for Digital Audio and Video

We investigate a future WiFi-VLC dual connectivity streaming system for 6DOF multi-user virtual reality that enables reliable high-fidelity remote scene immersion. The system integrates an edge server that uses scalable 360° tiling to adaptively split ...
Future Standards for Immersive VR: Report on the IEEE Virtual Reality 2007 Workshop

The Future Standards for Immersive VR Workshop at the IEEE Virtual Reality 2007 Conference aimed to foster an understanding between the VR and Web3D communities. Through multiple tool and application examples, contributors looked at the challenges and ...
Effective stereoscopic rendering for mobile VR
SA '16: SIGGRAPH ASIA 2016 Posters

Lately, the VR market is expanding rapidly with the advent of mobile phone based VR such as Google Cardboard, Samsung Gear VR, LG 360VR starting with VR(Virtual Reality) HMD(Head Mount Device) PC based VR using Stereoscopic rendering such as Oculus Rift,...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 19, Issue 2

March 2023

540 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/3572860

Editor:
Abdulmotaleb El Saddik
Mohamed Bin Zayed University of Artificial Intelligence, UAE and University of Ottawa, Canada

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 February 2023

Online AM: 16 June 2022

Accepted: 31 May 2022

Revised: 19 May 2022

Received: 09 February 2022

Published in TOMM Volume 19, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

National Science Foundation
National Institutes of Health
Panasonic Chair of Sustainability at the New Jersey Institute for Technology

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
585
Total Downloads

Downloads (Last 12 months)193
Downloads (Last 6 weeks)20

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang LSingh SChakareski JHajiesmaili MSitaraman R(2024)BONES: Near-Optimal Neural-Enhanced Video StreamingProceedings of the ACM on Measurement and Analysis of Computing Systems10.1145/36560148:2(1-28)Online publication date: 29-May-2024
https://dl.acm.org/doi/10.1145/3656014
Chakareski JKhan M(2024)Live 360° Video Streaming to Heterogeneous Clients in 5G NetworksIEEE Transactions on Multimedia10.1109/TMM.2024.338291026(8860-8873)Online publication date: 2024
https://doi.org/10.1109/TMM.2024.3382910
Ravuri HStruye Jvan der Hooft JWauters TDe Turck FFamaey JTorres Vega M(2024)Context-Aware and Reliable Transport Layer Framework for Interactive Immersive Media Delivery Over Millimeter WaveJournal of Network and Systems Management10.1007/s10922-024-09845-532:4Online publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1007/s10922-024-09845-5
Li YDou CWu YJia WLu R(2023)NOMA Assisted Two-Tier VR Content Transmission: A Tile-Based Approach for QoE OptimizationIEEE Transactions on Mobile Computing10.1109/TMC.2023.328073923:5(3769-3784)Online publication date: 29-May-2023
https://dl.acm.org/doi/10.1109/TMC.2023.3280739
Gartrell RBlack RBush KKhan MMoscola JGilbert J(2023)Reliable Communication in a Multi-Transceiver Mobile Optical Wireless NetworkMILCOM 2023 - 2023 IEEE Military Communications Conference (MILCOM)10.1109/MILCOM58377.2023.10356290(444-449)Online publication date: 30-Oct-2023
https://doi.org/10.1109/MILCOM58377.2023.10356290
Feng QYang PHuang ZChen JZhang N(2023)Deadline Aware Two-Timescale Resource Allocation for VR Video StreamingGLOBECOM 2023 - 2023 IEEE Global Communications Conference10.1109/GLOBECOM54140.2023.10437141(982-987)Online publication date: 4-Dec-2023
https://doi.org/10.1109/GLOBECOM54140.2023.10437141
Mahmoud MRizou SPanayides AKantartzis NKaragiannidis GLazaridis PZaharis Z(2023)A Survey on Optimizing Mobile Delivery of 360° Videos: Edge Caching and MulticastingIEEE Access10.1109/ACCESS.2023.329233511(68925-68942)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3292335

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents