research-article

Real-time Rendering with Compressed Animated Light Fields

Authors:

Babis Koniaris,

David Sinclair,

Kenny MitchellAuthors Info & Claims

GI '17: Proceedings of the 43rd Graphics Interface Conference

Pages 33 - 40

Published: 06 January 2017 Publication History

Abstract

We propose an end-to-end solution for presenting movie quality animated graphics to the user while still allowing the sense of presence afforded by free viewpoint head motion. By transforming offline rendered movie content into a novel immersive representation, we display the content in real-time according to the tracked head pose. For each frame, we generate a set of cubemap images per frame (colors and depths) using a sparse set of of cameras placed in the vicinity of the potential viewer locations. The cameras are placed with an optimization process so that the rendered data maximise coverage with minimum redundancy, depending on the lighting environment complexity. We compress the colors and depths separately, introducing an integrated spatial and temporal scheme tailored to high performance on GPUs for Virtual Reality applications. We detail a real-time rendering algorithm using multi-view ray casting and view dependent decompression. Compression rates of 150:1 and greater are demonstrated with quantitative analysis of image reconstruction quality and performance.

References

[1]

NVIDIA VIDEO CODEC SDK. https://developer.nvidia. com/nvidia-video-codec-sdk.

[2]

RGTC Texture Compression. https://www.khronos. org/registry/OpenGL/extensions/ARB/ARB_texture_ compression_bptc.txt.

[3]

RGTC Texture Compression. https://www.khronos. org/registry/OpenGL/extensions/ARB/ARB_texture_ compression_rgtc.txt.

[4]

E. H. Adelson and J. R. Bergen. The plenoptic function and the elements of early vision. Vision and Modeling Group, Media Laboratory, Massachusetts Institute of Technology, 1991.

[5]

G. v. d. Bergen. Efficient collision detection of complex deformable models using aabb trees. Journal of Graphics Tools, 2(4):1–13, 1997.

Digital Library

[6]

H. Bowles, K. Mitchell, R. W. Sumner, J. Moore, and M. Gross. Iterative image warping. In Computer graphics forum, volume 31, pages 237–246. Wiley Online Library, 2012.

[7]

A. Collet, M. Chuang, P. Sweeney, D. Gillett, D. Evseev, D. Calabrese, H. Hoppe, A. Kirk, and S. Sullivan. High-quality streamable freeviewpoint video. ACM Transactions on Graphics (TOG), 34(4):69, 2015.

Digital Library

[8]

L. Dabala, M. Ziegler, P. Didyk, F. Zilly, J. Keinert, K. Myszkowski, H.-P. Seidel, P. Rokita, and T. Ritschel. Efficient Multi-image Correspondences for On-line Light Field Video Processing. Computer Graphics Forum, 2016.

[9]

S. Donow. Light probe selection algorithms for real-time rendering of light fields, 2016.

[10]

S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen. The lumigraph. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 43–54. ACM, 1996.

Digital Library

[11]

J. Hooker. Volumetric Global Illumination at Treyarch. In Advances in Real-Time Rendering Part I Course, SIGGRAPH ’16, New York, NY, USA, 2016. ACM.

[12]

K. I. Iourcha, K. S. Nayak, and Z. Hong. System and method for fixed-rate block-based image compression with inferred pixel values, Sept. 21 1999. US Patent 5,956,431.

[13]

Y. J. Jeong, H. S. Chang, Y. H. Cho, D. Nam, and C.-C. J. Kuo. Efficient direct light field rendering for autostereoscopic 3d displays. SID, 2015.

[14]

J. T. Kajiya. The rendering equation. In ACM Siggraph Computer Graphics, volume 20, pages 143–150. ACM, 1986.

Digital Library

[15]

C. Kim, K. Subr, K. Mitchell, A. Sorkine-Hornung, and M. Gross. Online view sampling for estimating depth from light fields. In Image Processing (ICIP), 2015 IEEE International Conference on, pages 1155–1159. IEEE, 2015.

[16]

C. Kim, H. Zimmer, Y. Pritch, A. Sorkine-Hornung, and M. H. Gross. Scene reconstruction from high spatio-angular resolution light fields. ACM Trans. Graph., 32(4):73–1, 2013.

Digital Library

[17]

B. Koniaris, I. Huerta, M. Kosek, K. Darragh, C. Malleson, J. Jamrozy, N. Swafford, J. Guitian, B. Moon, A. Israr, S. Andrews, and K. Mitchell. Iridium: immersive rendered interactive deep media. In ACM SIGGRAPH 2016 VR Village, page 11. ACM, 2016.

Digital Library

[18]

K. Lee, D. Chu, E. Cuervo, J. Kopf, Y. Degtyarev, S. Grizan, A. Wolman, and J. Flinn. Outatime: Using speculation to enable low-latency continuous interaction for mobile cloud gaming. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services, pages 151–165. ACM, 2015.

Digital Library

[19]

M. Levoy and P. Hanrahan. Light field rendering. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 31–42. ACM, 1996.

Digital Library

[20]

M. McGuire, M. Mara, D. Nowrouzezahrai, and D. Luebke. Realtime global illumination using precomputed light field probes. In Proceedings of the 21st ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, page 2. ACM, 2017.

Digital Library

[21]

P. Merkle, Y. Morvan, A. Smolic, D. Farin, K. Mueller, P. de With, and T. Wiegand. The effects of multiview depth video compression on multiview rendering. Signal Processing: Image Communication, 24(1):73–88, 2009.

Digital Library

[22]

P. Merkle, K. Muller, A. Smolic, and T. Wiegand. Efficient compression of multi-view video exploiting inter-view dependencies based on h. 264/mpeg4-avc. In 2006 IEEE International Conference on Multimedia and Expo, pages 1717–1720. IEEE, 2006.

[23]

P. Merkle, A. Smolic, K. Muller, and T. Wiegand. Multi-view video plus depth representation and coding. In 2007 IEEE International Conference on Image Processing, volume 1, pages I–201. IEEE, 2007.

[24]

J. Nystad, A. Lassen, A. Pomianowski, S. Ellis, and T. Olson. Adaptive scalable texture compression. In Proceedings of the Fourth ACM SIGGRAPH/Eurographics conference on High-Performance Graphics, pages 105–114. Eurographics Association, 2012.

Digital Library

[25]

B. Reinert, J. Kopf, T. Ritschel, C. Eduardo, D. Chu, and H. Seidel. Proxy-guided image-based rendering for mobile devices. In Computer Graphics Forum: the international journal of the Eurographics Association, volume 35, 2016.

[26]

S. M. Seitz, B. Curless, J. Diebel, D. Scharstein, and R. Szeliski. A comparison and evaluation of multi-view stereo reconstruction algorithms. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 1, pages 519–528. IEEE, 2006.

Digital Library

[27]

L. Szirmay-Kalos, B. Aszodi, I. Lazanyi, and M. Premecz. Approximate Ray-Tracing on the GPU with Distance Impostors. Computer Graphics Forum, 2005.

[28]

G. Valenzise, F. De Simone, P. Lauga, and F. Dufaux. Performance evaluation of objective quality metrics for hdr image compression. In SPIE Optical Engineering+ Applications, pages 92170C–92170C. International Society for Optics and Photonics, 2014.

[29]

A. Vlachos. Advanced vr rendering performance. Game Developers Conference 2016, 2016.

Cited By

Kim JKnowles PSpjut JBoudaoud BMcguire M(2020)Post-Render Warp with Late Input Sampling Improves Aiming Under High Latency ConditionsProceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/34061873:2(1-18)Online publication date: 26-Aug-2020
https://dl.acm.org/doi/10.1145/3406187
Pratapa SManocha DAndrews STatarchuk N(2019)RLFCProceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games10.1145/3306131.3317018(1-10)Online publication date: 21-May-2019
https://dl.acm.org/doi/10.1145/3306131.3317018
Wijnants MLievens HMichiels NPut JQuax PLamotte WMorishima SItoh YShiratori TYue YLindeman R(2018)Standards-compliant HTTP adaptive streaming of static light fieldsProceedings of the 24th ACM Symposium on Virtual Reality Software and Technology10.1145/3281505.3281539(1-12)Online publication date: 28-Nov-2018
https://dl.acm.org/doi/10.1145/3281505.3281539
Show More Cited By

Recommendations

Multiple light field rendering
GRAPHITE '03: Proceedings of the 1st international conference on Computer graphics and interactive techniques in Australasia and South East Asia

A light field is a 4D function describing the radiance across a boundary between the volume containing a scene, and the disjoint volume in which the eyepoint may be placed. Light field rendering is the process of rendering novel views of a scene ...
Opacity light fields: interactive rendering of surface light fields with view-dependent opacity
I3D '03: Proceedings of the 2003 symposium on Interactive 3D graphics

We present new hardware-accelerated techniques for rendering surface light fields with opacity hulls that allow for interactive visualization of objects that have complex reflectance properties and elaborate geometrical details. The opacity hull is a ...
Progressive Coding of Surface Light Fields for Efficient Image Based Rendering

Surface light field is a light field parameterized to the surface of a geometric model. Although this multiview object representation enables rendering of objects from arbitrary views, the data size of a surface light field is huge. In this paper, a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

GI '17: Proceedings of the 43rd Graphics Interface Conference

June 2017

239 pages

ISBN:9780994786821

Conference Chairs:
Elmar Eisemann
Delft University of Technology
,
Scott Bateman
University of New Brunswick

Sponsors

The Canadian Human-Computer Communications Society / Société Canadienne du Dialogue Humaine Machine (CHCCS/SCDHM)
Microsoft: Microsoft
University of Alberta: University of Alberta

In-Cooperation

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques
SIGCHI: ACM Special Interest Group on Computer-Human Interaction
EUROGRAPHICS: The European Association for Computer Graphics

Publisher

Canadian Human-Computer Communications Society

Waterloo, Canada

Publication History

Published: 06 January 2017

Check for updates

Author Tags

Qualifiers

Research-article

Acceptance Rates

GI '17 Paper Acceptance Rate 26 of 52 submissions, 50%;

Overall Acceptance Rate 206 of 508 submissions, 41%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kim JKnowles PSpjut JBoudaoud BMcguire M(2020)Post-Render Warp with Late Input Sampling Improves Aiming Under High Latency ConditionsProceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/34061873:2(1-18)Online publication date: 26-Aug-2020
https://dl.acm.org/doi/10.1145/3406187
Pratapa SManocha DAndrews STatarchuk N(2019)RLFCProceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games10.1145/3306131.3317018(1-10)Online publication date: 21-May-2019
https://dl.acm.org/doi/10.1145/3306131.3317018
Wijnants MLievens HMichiels NPut JQuax PLamotte WMorishima SItoh YShiratori TYue YLindeman R(2018)Standards-compliant HTTP adaptive streaming of static light fieldsProceedings of the 24th ACM Symposium on Virtual Reality Software and Technology10.1145/3281505.3281539(1-12)Online publication date: 28-Nov-2018
https://dl.acm.org/doi/10.1145/3281505.3281539
Lall PBorac SRichardson DPharr MErnst M(2018)View-Region Optimized Image-Based Scene SimplificationProceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/32333111:2(1-22)Online publication date: 24-Aug-2018
https://dl.acm.org/doi/10.1145/3233311
Koniaris BKosek MSinclair DMitchell K(2018)GPU-accelerated depth codec for real-time, high-quality light field reconstructionProceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/32031931:1(1-15)Online publication date: 25-Jul-2018
https://dl.acm.org/doi/10.1145/3203193

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

Media

Figures

Other

Tables

View Table of Contents