research-article

Towards retina-quality VR video streaming: 15ms could save you 80% of your bandwidth

Authors:

Brooke Krajancich,

Gordon Wetzstein,

Keith WinsteinAuthors Info & Claims

ACM SIGCOMM Computer Communication Review, Volume 52, Issue 1

Pages 10 - 19

https://doi.org/10.1145/3523230.3523233

Published: 01 March 2022 Publication History

Abstract

Virtual reality systems today cannot yet stream immersive, retina-quality virtual reality video over a network. One of the greatest challenges to this goal is the sheer data rates required to transmit retina-quality video frames at high resolutions and frame rates. Recent work has leveraged the decay of visual acuity in human perception in novel gaze-contingent video compression techniques. In this paper, we show that reducing the motion-to-photon latency of a system itself is a key method for improving the compression ratio of gaze-contingent compression. Our key finding is that a client and streaming server system with sub-15ms latency can achieve 5x better compression than traditional techniques while also using simpler software algorithms than previous work.

References

[1]

Rachel Albert, Anjul Patney, David Luebke, and Joohwan Kim. 2017. Latency Requirements for Foveated Rendering in Virtual Reality. ACM Transactions on Applied Perception 14, 4 (Sept. 2017), 25:1–25:13. 1544-3558

Digital Library

[2]

Stephen J. Anderson, Kathy T. Mullen, and Robert F. Hess. 1991. Human Peripheral Spatial Resolution for Achromatic and Chromatic Stimuli: Limits Imposed By Optical and Retinal Factors. The Journal of Physiology 442, 1 (1991), 47–64.

[3]

Anastasios N. Angelopoulos, Julien N.P. Martel, Amit P. Kohli, Jörg Conradt, and Gordon Wetzstein. 2021. Event-Based Near-Eye Gaze Tracking Beyond 10,000 Hz. IEEE Transactions on Visualization and Computer Graphics 27, 5 (2021), 2577–2586.

[4]

Jean-Baptiste Bernard, Scherlen Anne-Catherine, and Castet Eric. 2007. Page Mode Reading With Simulated Scotomas: A Modest Effect of Interline Spacing on Reading Speed. Vision research 47, 28 (2007), 3447–3459.

[5]

Christopher J. Bockisch and Joel M. Miller. 1999. Different Motor Systems Use Similar Damped Extraretinal Eye Position Information. Vision research 39, 5 (1999), 1025–1038.

[6]

Roger H.S. Carpenter. 1988. Movements of the Eyes, 2nd Rev. Pion Limited.

[7]

Jiawen Chen, Miao Hu, Zhenxiao Luo, Zelong Wang, and Di Wu. 2020. SR360: Boosting 360-Degree Video Streaming with Super-Resolution. In Proceedings of the 30th ACM Workshop on Network and Operating Systems Support for Digital Audio and Video (Istanbul, Turkey) (NOSSDAV '20). Association for Computing Machinery, New York, NY, USA, 1–6.

Digital Library

[8]

Eduardo Cuervo, Krishna Chintalapudi, and Manikanta Kotaru. 2018. Creating the Perfect Illusion: What Will It Take to Create Life-Like Virtual Reality Headsets?. In Proceedings of the 19th International Workshop on Mobile Computing Systems & Applications (Tempe, Arizona, USA) (HotMobile '18). Association for Computing Machinery, New York, NY, USA, 7–12.

Digital Library

[9]

Michael F. Deering. 1998. The Limits of Human Vision. In 2nd International Immersive Projection Technology Workshop, Vol. 2. 1.

[10]

Sebastian Friston, Tobias Ritschel, and Anthony Steed. 2019. Perceptual Rasterization for Head-Mounted Display Image Synthesis. ACM Transactions on Graphics 38, 4, Article 97 (July 2019), 14 pages. 0730-0301

Digital Library

[11]

Wilson S. Geisler and Jeffrey S. Perry. 1998. Real-Time Foveated Multiresolution System for Low-Bandwidth Video Communication. In Human Vision and Electronic Imaging III, Bernice E. Rogowitz and Thrasyvoulos N. Pappas (Eds.), Vol. 3299. International Society for Optics and Photonics, SPIE, 294–305.

[12]

Yu Guan, Chengyuan Zheng, Xinggong Zhang, Zongming Guo, and Junchen Jiang. 2019. Pano: Optimizing 360 Video Streaming with a Better Understanding of Quality Perception. In Proceedings of the ACM Special Interest Group on Data Communication (Beijing, China) (SIGCOMM '19). Association for Computing Machinery, New York, NY, USA, 394–407.

Digital Library

[13]

Brian Guenter, Mark Finch, Steven Drucker, Desney Tan, and John Snyder. 2012. Foveated 3D graphics. ACM Transactions on Graphics 31, 6 (Nov. 2012), 164:1–164:10. 0730-0301

Digital Library

[14]

horsten Hansen, Lars Pracejus, and Karl R. Gegenfurtner. 2009. Color Perception in the Intermediate Periphery of the Visual Field. Journal of Vision 9, 4 (04 2009), 26–26. 1534-7362

[15]

. Hartmann, B. Lachenmayr, and H. Brettel. 1979. The Peripheral Critical Flicker Frequency. Vision Research 19, 9 (1979), 1019–1023. 0042-6989

[16]

azi Karam Illahi, Thomas Van Gemert, Matti Siekkinen, Enrico Masala, Antti Oulasvirta, and Antti Ylä-Jääski. 2020. Cloud Gaming with Foveated Video Encoding. ACM Transactions on Multimedia Compututing, Communications, and Applications 16, 1, Article 7 (Feb. 2020), 24 pages. 1551-6857

Digital Library

[17]

attis Jeppsson, Håvard Espeland, Tomas Kupka, Ragnar Langseth, Andreas Petlund, Peng Qiaoqiao, Chuansong Xue, Konstantin Pogorelov, Micheal Riegler, Dag Johansen, Carsten Griwodz, and Pål Halvorsen. 2018. Efficient Live and On-Demand Tiled HEVC 360 VR Video Streaming. In 2018 IEEE International Symposium on Multimedia (ISM). 81–88.

[18]

nton S. Kaplanyan, Anton Sochenov, Thomas Leimkühler, Mikhail Okunev, Todd Goodall, and Gizem Rufo. 2019. DeepFovea: Neural Reconstruction for Foveated Rendering and Video Compression Using Learned Statistics of Natural Videos. ACM Transactions on Graphics 38, 6, Article 212 (Nov. 2019), 13 pages. 0730-0301

Digital Library

[19]

Jonghyun Kim, Youngmo Jeong, Michael Stengel, Kaan Akşit, Rachel Albert, Ben Boudaoud, Trey Greer, Joohwan Kim, Ward Lopes, Zander Majercik, Peter Shirley, Josef Spjut, Morgan McGuire, and David Luebke. 2019. Foveated AR: Dynamically-Foveated Augmented Reality Display. ACM Transactions on Graphics 38, 4, Article 99 (July 2019), 15 pages. 0730-0301

Digital Library

[20]

Arnold Knapp. 1938. An Introduction to Clinical Perimetry. Archives of Ophthalmology 20, 6 (1938), 1116–1117.

[21]

G. A. Koulieris, K. Akşit, M. Stengel, R. K. Mantiuk, K. Mania, and C. Richardt. 2019. Near-Eye Display and Tracking Technologies for Virtual and Augmented Reality. Computer Graphics Forum 38, 2 (2019), 493–519.

[22]

Eileen Kowler. 2011. Eye Movements: The Past 25 Years. Vision Research 51, 13 (2011), 1457–1483. 0042-6989 Vision Research 50th Anniversary Issue: Part 2.

[23]

Brooke Krajancich, Petr Kellnhofer, and Gordon Wetzstein. 2021. A Perceptual Model for Eccentricity-dependent Spatio-temporal Flicker Fusion and its Applications to Foveated Graphics. arXiv preprint arXiv:2104.13514 (2021).

Digital Library

[24]

Sanghoon Lee, M.S. Pattichis, and A.C. Bovik. 2001. Foveated Video Compression With Optimal Rate Control. IEEE Transactions on Image Processing 10, 7 (July 2001), 977–992. 1941-0042

Digital Library

[25]

Lester C. Loschky and Gary S. Wolverton. 2007. How Late Can You Update Gaze-Contingent Multiresolutional Displays without Detection? ACM Transactions on Multimedia Computing, Communications, and Applications 3, 4, Article 7 (Dec. 2007), 10 pages. 1551-6857

Digital Library

[26]

David Luebke and Benjamin Hallen. 2001. Perceptually Driven Simplification for Interactive Rendering. In Eurographics Workshop on Rendering Techniques. Springer, 223–234.

Digital Library

[27]

John D. McCarthy, M. Angela Sasse, and Dimitrios Miras. 2004. Sharp or Smooth? Comparing the Effects of Quantization vs. Frame Rate for Streamed Video. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vienna, Austria) (CHI '04). Association for Computing Machinery, New York, NY, USA, 535–542.

Digital Library

[28]

Hunter Murphy and Andrew T Duchowski. 2001. Gaze-Contingent Level of Detail Rendering. EuroGraphics (2001).

[29]

T. Ohshima, H. Yamamoto, and H. Tamura. 1996. Gaze-Directed Adaptive Rendering for Interacting With Virtual Space. In Proceedings of the IEEE 1996 Virtual Reality Annual International Symposium. 103–110.

Digital Library

[30]

Anjul Patney, Marco Salvi, Joohwan Kim, Anton Kaplanyan, Chris Wyman, Nir Benty, David Luebke, and Aaron Lefohn. 2016. Towards Foveated Rendering for Gaze-Tracked Virtual Reality. ACM Transactions on Graphics 35, 6, Article 179 (Nov. 2016), 12 pages. 0730-0301

Digital Library

[31]

Eyal M. Reingold. 2014. Eye Tracking Research and Technology: Towards Objective Measurement of Data Quality. Visual Cognition 22, 3–4 (2014), 635–652.

[32]

J.G. Robson and Norma Graham. 1981. Probability Summation and Regional Variation in Contrast Sensitivity Across the Visual Field. Vision Research 21, 3 (1981), 409–418. 0042-6989

[33]

Miguel Fabian Romero-Rondón, Lucile Sassatelli, Frédéric Precioso, and Ramon Aparicio-Pardo. 2018. Foveated Streaming of Virtual Reality Videos. In Proceedings of the 9th ACM Multimedia Systems Conference (Amsterdam, Netherlands) (MMSys '18). Association for Computing Machinery, New York, NY, USA, 494–497.

Digital Library

[34]

Ruth Rosenholtz. 2016. Capabilities and Limitations of Peripheral Vision. Annual Review of Vision Science 2, 1 (2016), 437–457.

[35]

John Ross, M. Concetta Morrone, Michael E. Goldberg, and David C. Burr. 2001. Changes in Visual Perception at the Time of Saccades. Trends in Neurosciences 24, 2 (2001), 113–121. 0166-2236

[36]

Michele Rucci and Martina Poletti. 2015. Control and Functions of Fixational Eye Movements. Annual Review of Vision Science 1, 1 (2015), 499–518.

[37]

John Siderov and Ronald S. Harwerth. 1995. Stereopsis, Spatial Frequency and Retinal Eccentricity. Vision Research 35, 16 (1995), 2329–2337. 0042-6989

[38]

Niklas Stein, Diederick C Niehorster, Tamara Watson, Frank Steinicke, Katharina Rifai, Siegfried Wahl, and Markus Lappe. 2021. A Comparison of Eye Tracking Latencies Among Several Commercial Head-Mounted Displays. i-Perception 12, 1 (2021), 1–16.

[39]

Michael Stengel, Steve Grogorick, Martin Eisemann, and Marcus Magnor. 2016. Adaptive Image-Space Sampling for Gaze-Contingent Real-time Rendering. Computer Graphics Forum 35, 4 (July 2016), 129–139. 0167-7055

Digital Library

[40]

Hans Strasburger, Ingo Rentschler, and Martin Jüttner. 2011. Peripheral Vision and Pattern Recognition: A Review. Journal of Vision 11, 5 (12 2011), 1–82. 1534-7362

[41]

Liyang Sun, Yixiang Mao, Tongyu Zong, Yong Liu, and Yao Wang. 2020. Flocking-Based Live Streaming of 360-Degree Video. In Proceedings of the 11th ACM Multimedia Systems Conference (Istanbul, Turkey) (MMSys '20). Association for Computing Machinery, New York, NY, USA, 26–37.

Digital Library

[42]

Guanjun Tan, Yun-Han Lee, Tao Zhan, Jilin Yang, Sheng Liu, Dongfeng Zhao, and Shin-Tson Wu. 2018. Foveated Imaging for Near-Eye Displays. Optics Express 26, 19 (Sept. 2018), 25076–25085.

[43]

Timothy Terriberry. [n. d.]. Derf's Test Media Collection. https://media.xiph.org/video/derf/ Retrieved March, 2021 from

[44]

L. N. Thibos, F. E. Cheney, and D. J. Walsh. 1987. Retinal Limits to the Detection and Resolution of Gratings. Journal of the Optical Society of America A 4, 8 (Aug. 1987), 1524–1529.

[45]

Robin Thunström. 2014. Passive Gaze-Contingent Techniques Relation to System Latency. Master'sthesis. Blekinge Institute of Technology.

[46]

Oliver Wiedemann, Vlad Hosu, Hanhe Lin, and Dietmar Saupe. 2020. Foveated Video Coding for Real-Time Streaming Applications. In 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX). 1–6.

Cited By

Wu NLiu KCheng RHan BZhou POkoshi TKo JLiKamWa R(2024)Theia: Gaze-driven and Perception-aware Volumetric Content Delivery for Mixed Reality HeadsetsProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661858(70-84)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3643832.3661858
Mikawa YFukiage T(2024)Low-Latency Ocular Parallax Rendering and Investigation of Its Effect on Depth Perception in Virtual RealityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337207830:5(2228-2238)Online publication date: 5-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372078
Li NChang MRaychowdhury A(2024)E-Gaze: Gaze Estimation With Event CameraIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.335960646:7(4796-4811)Online publication date: Jul-2024
https://doi.org/10.1109/TPAMI.2024.3359606
Show More Cited By

Index Terms

Towards retina-quality VR video streaming: 15ms could save you 80% of your bandwidth
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Virtual reality
    2. Image compression
2. Hardware
  1. Communication hardware, interfaces and storage
    1. Displays and imagers

Recommendations

Creating the Perfect Illusion: What will it take to Create Life-Like Virtual Reality Headsets?
HotMobile '18: Proceedings of the 19th International Workshop on Mobile Computing Systems & Applications

As Virtual Reality (VR) Head Mounted Displays (HMD) push the boundaries of technology, in this paper, we try and answer the question, "What would it take to make the visual experience of a VR-HMD Life-Like, i.e., indistinguishable from physical reality?"...
VR Grabbers: Ungrounded Haptic Retargeting for Precision Grabbing Tools
UIST '18: Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology

Haptic feedback in VR is important for realistic simulation in virtual reality. However, recreating the haptic experience for hand tools in VR traditionally requires hardware with precise actuators, adding complexity to the system. We propose Ungrounded ...
Power Evaluation of 360 VR Video Streaming on Head Mounted Display Devices
NOSSDAV'17: Proceedings of the 27th Workshop on Network and Operating Systems Support for Digital Audio and Video

Virtual reality (VR) video streaming with 360-degree views has become a trending video application recently. While providing the users with immersive video viewing experiences, the 360 video streaming introduces significantly higher overhead than the ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGCOMM Computer Communication Review

ACM SIGCOMM Computer Communication Review Volume 52, Issue 1

January 2022

44 pages

ISSN:0146-4833

DOI:10.1145/3523230

Editor:
Steve Uhlig

Issue’s Table of Contents

Copyright © 2022 Copyright is held by the owner/author(s).

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 March 2022

Published in SIGCOMM-CCR Volume 52, Issue 1

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
482
Total Downloads

Downloads (Last 12 months)88
Downloads (Last 6 weeks)15

Reflects downloads up to 03 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wu NLiu KCheng RHan BZhou POkoshi TKo JLiKamWa R(2024)Theia: Gaze-driven and Perception-aware Volumetric Content Delivery for Mixed Reality HeadsetsProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661858(70-84)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3643832.3661858
Mikawa YFukiage T(2024)Low-Latency Ocular Parallax Rendering and Investigation of Its Effect on Depth Perception in Virtual RealityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337207830:5(2228-2238)Online publication date: 5-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372078
Li NChang MRaychowdhury A(2024)E-Gaze: Gaze Estimation With Event CameraIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.335960646:7(4796-4811)Online publication date: Jul-2024
https://doi.org/10.1109/TPAMI.2024.3359606
Wang YLi JLi ZShang SLiu Y(2024)Synergistic Temporal-Spatial User-Aware Viewport Prediction for Optimal Adaptive 360-Degree Video StreamingIEEE Transactions on Broadcasting10.1109/TBC.2024.337411970:2(453-467)Online publication date: Jun-2024
https://doi.org/10.1109/TBC.2024.3374119
Kozuma MKomura YMiyata SOkamoto YTamatsukuri YInoue HSaito TKobayashi HOnuki TYanagisawa YTakeuchi TOkazaki YKunitake HNakamura DNagata TYamane YIkeda MYamazaki S(2024)OLED Microdisplay With Monolithically Integrated CAAC-OS FET and Si CMOS Achieved by Two-Dimensionally Arranged Silicon Display DriversIEEE Journal of the Electron Devices Society10.1109/JEDS.2024.336693812(187-194)Online publication date: 2024
https://doi.org/10.1109/JEDS.2024.3366938
Guo PSu WZhang XChen HMa Z(2024)Modeling the non-uniform retinal perception for viewport-dependent streaming of immersive videoMultimedia Systems10.1007/s00530-024-01434-530:4Online publication date: 5-Aug-2024
https://dl.acm.org/doi/10.1007/s00530-024-01434-5
Singh RHuzaifa MLiu JPatney ASharif HZhao YAdve S(2023)Power, Performance, and Image Quality Tradeoffs in Foveated Rendering2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR55154.2023.00036(205-214)Online publication date: Mar-2023
https://doi.org/10.1109/VR55154.2023.00036
Li JXu YCao YZhu JWang D(2023)Utility-Driven Joint Caching and Bitrate Allocation for Real-Time Immersive VideosIEEE Journal of Selected Topics in Signal Processing10.1109/JSTSP.2023.329559717:5(1106-1118)Online publication date: Sep-2023
https://doi.org/10.1109/JSTSP.2023.3295597
Yang YZhao GShen Y(2023)High Frequency Event-based Eye Tracking Towards Mental Health Diagnosis2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct60411.2023.00072(335-339)Online publication date: 16-Oct-2023
https://doi.org/10.1109/ISMAR-Adjunct60411.2023.00072
Dong ZLing YXu CLi YSu Y(2023)Gaze-contingent efficient hologram compression for foveated near-eye holographic displaysDisplays10.1016/j.displa.2023.10246479(102464)Online publication date: Sep-2023
https://doi.org/10.1016/j.displa.2023.102464

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents