research-article

DeepVista: 16K Panoramic Cinema on Your Mobile Device

Authors:

Pan HuiAuthors Info & Claims

WWW '21: Proceedings of the Web Conference 2021

Pages 2232 - 2244

https://doi.org/10.1145/3442381.3449829

Published: 03 June 2021 Publication History

Abstract

In this paper, we design, implement, and evaluate, which is to our knowledge the first consumer-class system that streams panoramic videos far beyond the ultra high-definition resolution (up to 16K) to mobile devices, offering truly immersive experiences. Such an immense resolution makes streaming video-on-demand (VoD) content extremely resource-demanding. To tackle this challenge, introduces a novel framework that leverages an edge server to perform efficient, intelligent, and quality-guaranteed content transcoding, by extracting from panoramic frames the viewport stream that will be delivered to the client. To support real-time transcoding of 16K content, employs several key mechanisms such as dual-GPU acceleration, lossless viewport extraction, deep viewport prediction, and a two-layer streaming design. Our extensive evaluations using real users’ viewport movement data indicate that outperforms existing solutions, and can smoothly stream 16K panoramic videos to mobile devices over diverse wireless networks including WiFi, LTE, and mmWave 5G.

References

[1]

[n.d.]. 2019 New Year Illumination in Moscow, Russia (360 Video). https://www.youtube.com/watch?v=LdpzR_pZ9-w.

[2]

[n.d.]. Android MediaCodec API. https://developer.android.com/reference/android/media/MediaCodec.

[3]

[n.d.]. Angel Falls (360 Video). https://www.youtube.com/watch?v=L_tqK4eqelA.

[4]

[n.d.]. Big Screens, Big Data: The Future For Smartphones. https://www.npd.com/wps/portal/npd/us/blog/2018/big-screens-big-data-the-future-for-smartphones/.

[5]

[n.d.]. CRF Guide (Constant Rate Factor in x264, x265 and libvpx). https://slhck.info/video/2017/02/24/crf-guide.html.

[6]

[n.d.]. Explaining 360 video resolution: how to measure it, quality comparisons, and other metrics to look at. https://www.immersiveshooter.com/2017/08/31/explaining-360-video-resolution-how-measure-quality-comparisons/.

[7]

[n.d.]. Google AR and VR: Bringing pixels front and center in VR video. https://blog.google/products/google-ar-vr/bringing-pixels-front-and-center-vr-video/.

[8]

[n.d.]. Google Cardboard. https://vr.google.com/cardboard/index.html.

[9]

[n.d.]. Google Compute Engine Pricing. https://cloud.google.com/compute/pricing.

[10]

[n.d.]. Google Stadia. https://stadia.com/.

[11]

[n.d.]. Keras: The Python Deep Learning library. https://keras.io/.

[12]

[n.d.]. Making 12K 360° VR Streaming a Reality. https://medium.com/visbit/making-12k-360%C2%BA-vr-streaming-a-reality-why-and-how-we-did-it-ce65e9aa0bc3.

[13]

[n.d.]. New 8K OLED Displays for Tablets and Laptops: 8.3 and 13.3 Inches. https://www.anandtech.com/show/13742/new-8k-oled-displays.

[14]

[n.d.]. New York City VR 360 Drive (360 Video). https://www.youtube.com/watch?v=2Lq86MKesG4.

[15]

[n.d.]. Next-generation video encoding techniques for 360 video and VR. https://code.fb.com/virtual-reality/next-generation-video-encoding-techniques-for-360-video-and-vr/.

[16]

[n.d.]. Nvidia CUDA Runtime API. https://docs.nvidia.com/cuda/cuda-runtime-api/.

[17]

[n.d.]. NVIDIA Launches Edge Computing Platform to Bring Real-Time AI to Global Industries. https://nvidianews.nvidia.com/news/nvidia-launches-edge-computing-platform-to-bring-real-time-ai-to-global-industries.

[18]

[n.d.]. Nvidia Video Codec SDK. https://developer.nvidia.com/nvidia-video-codec-sdk.

[19]

[n.d.]. NVIDIA Video Encode and Decode GPU Support Matrix. https://developer.nvidia.com/video-encode-decode-gpu-support-matrix.

[20]

[n.d.]. Per-Title Encode Optimization. https://medium.com/netflix-techblog/per-title-encode-optimization-7e99442b62a2.

[21]

[n.d.]. Programming with pcap. https://www.tcpdump.org/pcap.html.

[22]

[n.d.]. Roller Coaster (360 Video). https://www.youtube.com/watch?v=s14wKgPPQ-c.

[23]

Omid Abari, Dinesh Bharadia, Austin Duffield, and Dina Katabi. 2017. Enabling High-Quality Untethered Virtual Reality. In Proceedings of NSDI.

Digital Library

[24]

Shahryar Afzal, Jiasi Chen, and KK Ramakrishnan. 2017. Characterization of 360-Degree Videos. In Proceedings of the Workshop on VR/AR Network. ACM, 1–6.

Digital Library

[25]

Shivang Aggarwal, Sibendu Paul, Pranab Dash, Nuka Saranya Illa, Y Charlie Hu, Dimitrios Koutsonikolas, and Zhisheng Yan. 2020. How to Evaluate Mobile 360° Video Streaming Systems?. In ACM HotMobile.

[26]

Patrice Rondao Alface, Maarten Aerts, Donny Tytgat, Sammy Lievens, Christoph Stevens, Nico Verzijp, and Jean-Francois Macq. 2017. 16K Cinematic VR Streaming. In ACM Multimedia.

[27]

Bo Chen, Zhisheng Yan, Haiming Jin, and Klara Nahrstedt. 2019. Event-driven stitching for tile-based live 360 video streaming. In ACM MMSys.

[28]

Eduardo Cuervo, Alec Wolman, Landon P. Cox, Kiron Lebeck, Ali Razeen, Stefan Saroiu, and Madanlal Musuvathi. 2015. Kahawai: High-Quality Mobile Gaming Using GPU Offload. In Proceedings of ACM MobiSys.

Digital Library

[29]

Mallesham Dasari, Arani Bhattacharya, Santiago Vargas, Pranjal Sahu, Aruna Balasubramanian, and Samir R Das. 2020. Streaming 360 Videos Using Super-Resolution. In IEEE INFOCOM.

[30]

Ching-Ling Fan, Jean Lee, Wen-Chih Lo, Chun-Ying Huang, Kuan-Ta Chen, and Cheng-Hsin Hsu. 2017. Fixation Prediction for 360 Video Streaming in Head-Mounted Virtual Reality. In ACM NOSSDAV.

[31]

Google. [n.d.]. Android Profiler. https://developer.android.com/studio/profile/android-profiler.

[32]

Mario Graf, Christian Timmerer, and Christopher Mueller. 2017. Towards bandwidth efficient adaptive streaming of omnidirectional video over HTTP: Design, implementation, and evaluation. In Proceedings of MMSys 2017. ACM, 261–271.

Digital Library

[33]

Yu Guan, Chengyuan Zheng, Xinggong Zhang, Zongming Guo, and Junchen Jiang. 2019. Pano: Optimizing 360 video streaming with a better understanding of quality perception. In SIGCOMM.

Digital Library

[34]

Bo Han, Yu Liu, and Feng Qian. 2020. ViVo: visibility-aware mobile volumetric video streaming. In ACM MobiCom.

[35]

Jian He, Mubashir Adnan Qureshi, Lili Qiu, Jin Li, Feng Li, and Lei Han. 2018. Rubiks: Practical 360-Degree Streaming for Smartphones. In Proceedings of MobiSys 2018. ACM.

Digital Library

[36]

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation 9, 8 (1997), 1735–1780.

Digital Library

[37]

Xueshi Hou, Sujit Dey, Jianzhong Zhang, and Madhukar Budagavi. 2018. Predictive View Generation to Enable Mobile 360-degree and VR Experiences. In Proceedings of the Workshop on Virtual Reality and Augmented Reality Network. ACM, 20–26.

Digital Library

[38]

Junchen Jiang, Vyas Sekar, and Hui Zhang. 2012. Improving Fairness, Efficiency, and Stability in HTTP-Based Adaptive Video Streaming With Festive. In ACM CoNEXT.

[39]

Teemu Kämäräinen, Matti Siekkinen, Jukka Eerikäinen, and Antti Ylä-Jääski. 2018. CloudVR: Cloud accelerated interactive mobile virtual reality. In ACM Multimedia.

[40]

Zeqi Lai, Y Charlie Hu, Yong Cui, Linhui Sun, and Ningwei Dai. [n.d.]. Furion: Engineering high-quality immersive virtual reality on today’s mobile devices. In ACM MobiCom 2017.

Digital Library

[41]

Yong Li and Wei Gao. 2018. MUVR: Supporting multi-user mobile virtual reality with resource constrained edge cloud. In IEEE/ACM Symp. on Edge Computing.

[42]

Luyang Liu, Ruiguang Zhong, Wuyang Zhang, Yunxin Liu, Jiansong Zhang, Lintao Zhang, and Marco Gruteser. 2018. Cutting the Cord: Designing a High-quality Untethered VR System with Low Latency Remote Rendering. In ACM MobiSys.

[43]

Hongzi Mao, Ravi Netravali, and Mohammad Alizadeh. 2017. Neural Adaptive Video Streaming with Pensieve. In Proceedings of SIGCOMM 2017. ACM, 197–210.

Digital Library

[44]

Jiayi Meng, Sibendu Paul, and Y Charlie Hu. 2020. Coterie: Exploiting Frame Similarity to Enable High-Quality Multiplayer VR on Commodity Mobile Devices. In ASPLOS.

Digital Library

[45]

NVIDIA. [n.d.]. NVIDIA System Management Interface. https://developer.nvidia.com/nvidia-system-management-interface.

[46]

Stefano Petrangeli, Viswanathan Swaminathan, Mohammad Hosseini, and Filip De Turck. [n.d.]. An HTTP/2-based adaptive streaming framework for 360° virtual reality videos. In ACM MM 2017.

[47]

Feng Qian, Bo Han, Qingyang Xiao, and Vijay Gopalakrishnan. 2018. Flare: Practical Viewport-Adaptive 360-Degree Video Streaming for Mobile Devices. In ACM MobiCom.

[48]

Xukan Ran, Carter Slocum, Maria Gorlatova, and Jiasi Chen. 2019. ShareAR: Communication-efficient multi-user mobile augmented reality. In ACM HotNets.

[49]

Shu Shi, Varun Gupta, Michael Hwang, and Rittwik Jana. 2019. Mobile VR on edge cloud: a latency-driven design. In ACM MMSys.

[50]

Shu Shi, Varun Gupta, and Rittwik Jana. 2019. Freedom: Fast Recovery Enhanced VR Delivery Over Mobile Networks. In ACM MobiSys.

[51]

Liyang Sun, Fanyi Duanmu, Yong Liu, Yao Wang, Yinghua Ye, Hang Shi, and David Dai. 2018. Multi-path multi-tier 360-degree video streaming in 5G networks. In ACM MMSys.

[52]

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600–612.

Digital Library

[53]

Lan Xie, Xinggong Zhang, and Zongming Guo. 2018. CLS: A Cross-user Learning based System for Improving QoE in 360 Video Adaptive Streaming. In ACM Multimedia.

Digital Library

[54]

Xiufeng Xie and Xinyu Zhang. 2017. POI360: Panoramic Mobile Video Telephony over LTE Cellular Networks. In Proceedings of CoNEXT 2017. ACM, 336–349.

Digital Library

[55]

Tan Xu, Bo Han, and Feng Qian. 2019. Analyzing viewport prediction under different VR interactions. In ACM CoNEXT.

[56]

Zhisheng Yan and Chang Wen Chen. 2016. RnB: Rate and Brightness Adaptation for Rate-distortion-energy Tradeoff in HTTP Adaptive Streaming over Mobile Devices. In ACM MobiCom.

[57]

Xiaoqi Yin, Abhishek Jindal, Vyas Sekar, and Bruno Sinopoli. 2015. A Control-Theoretic Approach for Dynamic Adaptive Video Streaming over HTTP. In ACM SIGCOMM.

[58]

Alireza Zare, Alireza Aminlou, Miska M Hannuksela, and Moncef Gabbouj. 2016. HEVC-compliant tile-based streaming of panoramic video for virtual reality applications. In Proceedings of MM 2016. ACM, 601–605.

Digital Library

[59]

Yuanxing Zhang, Yushuo Guan, Kaigui Bian, Yunxin Liu, Hu Tuo, Lingyang Song, and Xiaoming Li. 2020. EPASS360: QoE-aware 360-degree Video Streaming over Mobile Devices. IEEE Transactions on Mobile Computing(2020).

[60]

Yuanxing Zhang, Pengyu Zhao, Kaigui Bian, Yunxin Liu, Lingyang Song, and Xiaoming Li. 2019. DRL360: 360-degree Video Streaming with Deep Reinforcement Learning. In IEEE INFOCOM.

Cited By

Zhang YKutscher DCui Y(2024)Networked Metaverse Systems: Foundations, Gaps, Research DirectionsIEEE Open Journal of the Communications Society10.1109/OJCOMS.2024.34260985(5488-5539)Online publication date: 2024
https://doi.org/10.1109/OJCOMS.2024.3426098
Huang XWu WHu SLi MZhou CShen X(2024)Digital Twin Based User-Centric Resource Management for Multicast Short Video StreamingIEEE Journal of Selected Topics in Signal Processing10.1109/JSTSP.2023.334362618:1(50-65)Online publication date: Jan-2024
https://doi.org/10.1109/JSTSP.2023.3343626
Zhang YPu LLin TYan J(2023)QoE-Oriented Mobile Virtual Reality Game in Distributed Edge NetworksIEEE Transactions on Multimedia10.1109/TMM.2023.324718225(9132-9146)Online publication date: 13-Mar-2023
https://dl.acm.org/doi/10.1109/TMM.2023.3247182
Show More Cited By

Recommendations

Active feedback for enhancing the construction of panoramic live mobile video streams
ICME '11: Proceedings of the 2011 IEEE International Conference on Multimedia and Expo

Constructing a panoramic video out of multiple incoming live mobile video streams is a challenging problem. This problem involves multiple users live streaming the same scene from different angles, using their mobile phones, with the objective of ...
16K Cinematic VR Streaming
MM '17: Proceedings of the 25th ACM international conference on Multimedia

We present an end-to-end system for streaming Cinematic Virtual Reality (VR) content (also called 360 or omnidirectional content). Content is captured and ingested at a resolution of 16K at 25Hz and streamed towards untethered mobile VR devices. Besides ...
Mobile device security
InfoSecCD '04: Proceedings of the 1st annual conference on Information security curriculum development

Because of their small size, memory capability, and the case with which information can be downloaded and removed from a facility, mobile devices pose a risk to organizations when used and transported outside physical boundaries. Mobile devices, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '21: Proceedings of the Web Conference 2021

April 2021

4054 pages

ISBN:9781450383127

DOI:10.1145/3442381

Editors:
Jure Leskovec
Stanford
,
Marko Grobelnik
Jožef Stefan Institute
,
Marc Najork
Google
,
Jie Tang
Tsinghua University
,
Leila Zia
Wikimedia Foundation

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 June 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article
Research
Refereed limited

Conference

WWW '21

Sponsor:

SIGWEB

WWW '21: The Web Conference 2021

April 19 - 23, 2021

Ljubljana, Slovenia

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
361
Total Downloads

Downloads (Last 12 months)97
Downloads (Last 6 weeks)12

Reflects downloads up to 22 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang YKutscher DCui Y(2024)Networked Metaverse Systems: Foundations, Gaps, Research DirectionsIEEE Open Journal of the Communications Society10.1109/OJCOMS.2024.34260985(5488-5539)Online publication date: 2024
https://doi.org/10.1109/OJCOMS.2024.3426098
Huang XWu WHu SLi MZhou CShen X(2024)Digital Twin Based User-Centric Resource Management for Multicast Short Video StreamingIEEE Journal of Selected Topics in Signal Processing10.1109/JSTSP.2023.334362618:1(50-65)Online publication date: Jan-2024
https://doi.org/10.1109/JSTSP.2023.3343626
Zhang YPu LLin TYan J(2023)QoE-Oriented Mobile Virtual Reality Game in Distributed Edge NetworksIEEE Transactions on Multimedia10.1109/TMM.2023.324718225(9132-9146)Online publication date: 13-Mar-2023
https://dl.acm.org/doi/10.1109/TMM.2023.3247182
Zhang QWei JWang SMa SGao W(2023)RealVR: Efficient, Economical, and Quality-of- Experience-Driven VR Video System Based on MPEG OMAFIEEE Transactions on Multimedia10.1109/TMM.2022.319069725(5386-5399)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TMM.2022.3190697
Han BChen SMartin JPathak PVarshney AXue HYu LZhang JZhao X(2023)Immersive Computing: Vision, Infrastructure, and Use Cases2023 IEEE 9th International Conference on Collaboration and Internet Computing (CIC)10.1109/CIC58953.2023.00022(1-10)Online publication date: 1-Nov-2023
https://doi.org/10.1109/CIC58953.2023.00022
Cheng RWu NVarvello MChen SHan BBarakat CPelsser CBenson TChoffnes D(2022)Are we ready for metaverse?Proceedings of the 22nd ACM Internet Measurement Conference10.1145/3517745.3561417(504-518)Online publication date: 25-Oct-2022
https://dl.acm.org/doi/10.1145/3517745.3561417
Cheng RWu NChen SHan B(2022)Reality Check of Metaverse: A First Look at Commercial Social Virtual Reality Platforms2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW55335.2022.00040(141-148)Online publication date: Mar-2022
https://doi.org/10.1109/VRW55335.2022.00040
Chen SDuinkharjav BSun XWei LPetrangeli SEchevarria JSilva CSun Q(2022)Instant Reality: Gaze-Contingent Perceptual Optimization for 3D Virtual Reality StreamingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.315052228:5(2157-2167)Online publication date: May-2022
https://doi.org/10.1109/TVCG.2022.3150522
Sethuraman MSarma ABauskar ADhekne ARamachandran U(2022)ClairvoyantEdge: Prescient Prefetching of On-demand Video at the Edge of the Network2022 IEEE/ACM 7th Symposium on Edge Computing (SEC)10.1109/SEC54971.2022.00010(26-39)Online publication date: Dec-2022
https://doi.org/10.1109/SEC54971.2022.00010
Cheng RWu NChen SHan B(2022)Will Metaverse Be NextG Internet? Vision, Hype, and RealityIEEE Network10.1109/MNET.117.220005536:5(197-204)Online publication date: Sep-2022
https://doi.org/10.1109/MNET.117.2200055

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents