research-article

GPU-accelerated depth codec for real-time, high-quality light field reconstruction

Authors:

Babis Koniaris,

David Sinclair,

Kenny MitchellAuthors Info & Claims

Proceedings of the ACM on Computer Graphics and Interactive Techniques, Volume 1, Issue 1

Article No.: 3, Pages 1 - 15

https://doi.org/10.1145/3203193

Published: 25 July 2018 Publication History

Abstract

Pre-calculated depth information is essential for efficient light field video rendering, due to the prohibitive cost of depth estimation from color when real-time performance is desired. Standard state-of-the-art video codecs fail to satisfy such performance requirements when the amount of data to be decoded becomes too large. In this paper, we propose a depth image and video codec based on block compression, that exploits typical characteristics of depth streams, drawing inspiration from S3TC texture compression and geometric wavelets. Our codec offers very fast hardware-accelerated decoding that also allows partial extraction for view-dependent decoding. We demonstrate the effectiveness of our codec in a number of multi-view 360 degree video datasets, with quantitative analysis of storage cost, reconstruction quality and decoding performance.

References

[1]

Edward H Adelson, James R Bergen, et al. 1991. The plenoptic function and the elements of early vision. (1991).

[2]

Filippo Bannò, Paolo Simone Gasparello, Franco Tecchia, and Massimo Bergamasco. 2012. Real-time Compression of Depth Streams Through Meshification and Valence-based Encoding. In Proceedings of the 11th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry (VRCAI '12). ACM, New York, NY, USA, 263--270.

Digital Library

[3]

Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese, Hugues Hoppe, Adam Kirk, and Steve Sullivan. 2015. High-quality Streamable Free-viewpoint Video. ACM Trans. Graph. 34, 4, Article 69 (July 2015), 13 pages.

Digital Library

[4]

Piotr Didyk Tobias Ritschel, Elmar Eisemann, Karol Myszkowski, and Hans-Peter Seidel. 2010. Adaptive Image-space Stereo View Synthesis. In VMV. 299--306.

[5]

C Goktug Gurler, Anil Aksay, Gozde Bozdagi Akar, and A Murat Tekalp. 2009. Multi-threaded architectures and benchmark tests for real-time multi-view video decoding. In Multimedia and Expo, 2009. ICME 2009. IEEE International Conference on. IEEE, 237--240.

Digital Library

[6]

Fabian Jäger. 2011. Contour-based segmentation and coding for depth map compression. In Visual Communications and Image Processing (VCIP), 2011 IEEE. IEEE, 1--4.

[7]

Xiaoran Jiang, Mikaël Le Pendu, and Christine Guillemot. 2017. Light field compression using depth image based view synthesis. In Multimedia 8 Expo Workshops (ICMEW), 2017IEEE International Conference on. IEEE, 19--24.

[8]

Vahid Kiani, Ahad Harati, and Abedin Vahedian. 2017. Planelets - A Piecewise Linear Fractional Model for Preserving Scene Geometry in Intra-Coding of Indoor Depth Images. IEEE Transactions on Image Processing 26 (2017), 590--602.

Digital Library

[9]

Babis Koniaris, Maggie Kosek, David Sinclair, and Kenny Mitchell. 2017. Real-time Rendering with Compressed Animated Light Fields. In Proceedings of the 43rd Graphics Interface Conference (GI '17). Canadian Human-Computer Communications Society, School of Computer Science, University of Waterloo, Waterloo, Ontario, Canada, 33--40.

Digital Library

[10]

Jianjun Lei, Shuai Li, Ce Zhu, Ming-Ting Sun, and Chunping Hou. 2015. Depth coding based on depth-texture motion and structure similarities. IEEE Transactions on Circuits and Systems for Video Technology 25, 2 (2015), 275--286.

[11]

Marc Levoy and Pat Hanrahan. 1996. Light field rendering. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. ACM, 31--42.

Digital Library

[12]

Shujie Liu, PoLin Lai, Dong Tian, and Chang Wen Chen. 2011. New depth coding techniques with utilization of corresponding video. IEEE Transactions on broadcasting 57, 2 (2011), 551--561.

[13]

Yunpeng Liu, Stephan Beck, Renfang Wang, Jin Li, Huixia Xu, Shijie Yao, Xiaopeng Tong, and Bernd Froehlich. 2015. Hybrid Lossless-Lossy Compression for Real-Time Depth-Sensor Streams in 3D Telepresence Applications. In PCM (1) (Lecture Notes in Computer Science), Yo-Sung Ho, Jitao Sang, Yong Man Ro, Junmo Kim, and Fei Wu (Eds.), Vol. 9314. Springer, 442--452.

[14]

Morgan McGuire, Mike Mara, Derek Nowrouzezahrai, and David Luebke. 2017. Real-time global illumination using precomputed light field probes. In Proceedings of the 21st ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games. ACM, 2.

Digital Library

[15]

Philipp Merkle, Aljoscha Smolic, Karsten Muller, and Thomas Wiegand. 2007. Multi-view video plus depth representation and coding. In Image Processing, 2007. ICIP 2007. IEEE International Conference on, Vol. 1. IEEE, I-201.

[16]

Yannick Morvan, Dirk Farin, et al. 2006. Platelet-based coding of depth maps for the transmission of multiview images. In Proceedings of SPIE, Stereoscopic Displays and Applications, Vol. 6055. 93--100.

[17]

Karsten Muller, Heiko Schwarz, Detlev Marpe, Christian Bartnik, Sebastian Bosse, Heribert Brust, Tobias Hinz, Haricharan Lakshman, Philipp Merkle, Franz Hunn Rhee, Gerhard Tech, Martin Winken, and Thomas Wiegand. 2013. 3D High-Efficiency Video Coding for Multi-View Video and Depth Data. Trans. Img. Proc. 22, 9 (Sept. 2013), 3366--3378.

Digital Library

[18]

Dawid Pająk, Robert Herzog, Radosław Mantiuk, Piotr Didyk, Elmar Eisemann, Karol Myszkowski, and Kari Pulli. 2014. Perceptual depth compression for stereo applications. In Computer Graphics Forum, Vol. 33. Wiley Online Library, 195--204.

Digital Library

[19]

Fabrizio Pece, Jan Kautz, and Tim Weyrich. 2011. Adapting Standard Video Codecs for Depth Streaming. In Joint Virtual Reality Conference of EGVE - EuroVR, Sabine Coquillart, Anthony Steed, and Greg Welch (Eds.). The Eurographics Association.

Digital Library

[20]

Jeff Pool, Anselmo Lastra, and Montek Singh. 2012. Lossless compression of variable-precision floating-point buffers on GPUs. In Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games. ACM, 47--54.

Digital Library

[21]

Jacob Ström, Per Wennersten, Jim Rasmusson, Jon Hasselgren, Jacob Munkberg, Petrik Clarberg, and Tomas Akenine-Möller. 2008. Floating-point buffer compression in a unified codec architecture. In Proceedings of the 23rd ACM SIGGRAPH/EUROGRAPHICS symposium on Graphics hardware. Eurographics Association, 75--84.

Digital Library

[22]

Krzysztof Wegner, Olgierd Stankiewicz, and Marek Domanski. 2014. Fast View Synthesis using platelet-based depth representation. In Systems, Signals and Image Processing (IWSSIP), 2014 International Conference on. IEEE, 55--58.

[23]

M. O. Wildeboer, T. Yendo, M. P. Tehrani, T. Fujii, and M. Tanimoto. 2010. Color based depth up-sampling for depth compression. In 28th Picture Coding Symposium. 170--173.

[24]

Andrew D. Wilson. 2017. Fast Lossless Depth Image Compression. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces (ISS '17). ACM, New York, NY, USA, 100--105.

Digital Library

Cited By

Siekkinen MKämäräinen T(2023)Neural Network Assisted Depth Map Packing for Compression Using Standard Hardware Video CodecsACM Transactions on Multimedia Computing, Communications, and Applications10.1145/358844019:5s(1-20)Online publication date: 7-Jun-2023
https://dl.acm.org/doi/10.1145/3588440
Hladky JStengel MVining NKerbl BSeidel HSteinberger M(2022)QuadStreamACM Transactions on Graphics10.1145/3550454.355552441:6(1-13)Online publication date: 30-Nov-2022
https://dl.acm.org/doi/10.1145/3550454.3555524

Index Terms

GPU-accelerated depth codec for real-time, high-quality light field reconstruction
1. Computing methodologies
  1. Computer graphics

Recommendations

Multiple light field rendering
GRAPHITE '03: Proceedings of the 1st international conference on Computer graphics and interactive techniques in Australasia and South East Asia

A light field is a 4D function describing the radiance across a boundary between the volume containing a scene, and the disjoint volume in which the eyepoint may be placed. Light field rendering is the process of rendering novel views of a scene ...
Real-time Rendering with Compressed Animated Light Fields
GI '17: Proceedings of the 43rd Graphics Interface Conference

We propose an end-to-end solution for presenting movie quality animated graphics to the user while still allowing the sense of presence afforded by free viewpoint head motion. By transforming offline rendered movie content into a novel immersive ...
Light Field Display Point Rendering

Rendering for light field displays (LFDs) requires rendering of dozens or hundreds of views, which must then be combined into a single image on the display, making real-time LFD rendering extremely difficult. We introduce light field display point ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Computer Graphics and Interactive Techniques

Proceedings of the ACM on Computer Graphics and Interactive Techniques Volume 1, Issue 1

July 2018

378 pages

EISSN:2577-6193

DOI:10.1145/3242771

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2018

Published in PACMCGIT Volume 1, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Innovate UK

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
216
Total Downloads

Downloads (Last 12 months)32
Downloads (Last 6 weeks)11

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Siekkinen MKämäräinen T(2023)Neural Network Assisted Depth Map Packing for Compression Using Standard Hardware Video CodecsACM Transactions on Multimedia Computing, Communications, and Applications10.1145/358844019:5s(1-20)Online publication date: 7-Jun-2023
https://dl.acm.org/doi/10.1145/3588440
Hladky JStengel MVining NKerbl BSeidel HSteinberger M(2022)QuadStreamACM Transactions on Graphics10.1145/3550454.355552441:6(1-13)Online publication date: 30-Nov-2022
https://dl.acm.org/doi/10.1145/3550454.3555524

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents