research-article

View-Region Optimized Image-Based Scene Simplification

Authors:

Dave Richardson,

Manfred ErnstAuthors Info & Claims

Proceedings of the ACM on Computer Graphics and Interactive Techniques, Volume 1, Issue 2

Article No.: 26, Pages 1 - 22

https://doi.org/10.1145/3233311

Published: 24 August 2018 Publication History

Abstract

We present a new algorithm for image-based simplification of complex scenes for virtual reality (VR). The algorithm transforms geometrically-detailed environments into a layered quad tile representation that is optimized for a specified viewing region and is renderable on low-power mobile-class VR devices. A novel constrained optimization formulation ensures that the scene can be rendered within a predetermined compute budget, with limits on both primitive count and fill rate. Furthermore, we introduce a new method for texturing from point samples of the original scene geometry that generates high-quality silhouettes without the drawbacks of traditional point splatting.

The resulting representation achieves a visual fidelity that was previously impossible on mobile graphics hardware; our algorithm can typically generate a high-quality representation of visually-rich scenes with billions of triangles using just 72k triangles and a single high-resolution texture map (with generally only about 50% more texels than a stereo panorama). The effectiveness of the approach is demonstrated with a set of challenging test cases.

Supplementary Material

lall (lall.zip)

Supplemental movie, appendix, image and software files for, View-Region Optimized Image-Based Scene Simplification

Download
50.67 MB

References

[1]

Sameer Agarwal, Keir Mierle, and Others. 2018. Ceres Solver. (2018). http://ceres-solver.org

[2]

Alcon Interactive. 2018. Blade Runner: Revelations. (2018). https://play.google.com/store/apps/details?id=com.alconinteractive.brvr

[3]

Robert Anderson, David Gallup, Jonathan T. Barron, Janne Kontkanen, Noah Snavely, Carlos Hernández, Sameer Agarwal, and Steven M. Seitz. 2016. Jump: Virtual Reality Video. ACM Trans. Graph. 35, 6, Article 198, 13 pages.

Digital Library

[4]

C. Andujar, P. Brunet, A. Chica, I. Navazo, J. Rossignac, and A. Vinacua. 2004. Computing Maximal Tiles and Application to Impostor-Based Simplification. Computer Graphics Forum 23, 3 (2004), 401--410.

[5]

S. Behrendt, C. Colditz, O. Franzke, J. Kopf, and O. Deussen. 2005. Realistic real-time rendering of landscapes using billboard clouds. Computer Graphics Forum (2005).

[6]

Marius Bjørge, Sam Martin, Sandeep Kakarlapudi, and Jan-Harald Fredriksen. 2014. Efficient Rendering with Tile Local Storage. In ACM SIGGRAPH 2014 Talks (SIGGRAPH '14). Article 51, 1 pages.

Digital Library

[7]

Kevin Boos, David Chu, and Eduardo Cuervo. 2016. FlashBack: Immersive Virtual Reality on Mobile Devices via Rendering Memoization. In Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '16). 291--304.

Digital Library

[8]

David Cohen-Steiner, Pierre Alliez, and Mathieu Desbrun. 2004. Variational Shape Approximation. ACM Trans. Graph. 23, 3 (Aug. 2004), 905--914.

Digital Library

[9]

Cyril Crassin, Fabrice Neyret, Sylvain Lefebvre, and Elmar Eisemann. 2009. GigaVoxels: Ray-guided Streaming for Efficient and Detailed Voxel Rendering. In Proceedings of the 2009 Symposium on Interactive 3D Graphics and Games (I3D '09). 15--22.

Digital Library

[10]

Philippe Decaudin and Fabrice Neyret. 2009. Volumetric Billboards. Computer Graphics Forum 28, 8 (2009), 2079--2089.

[11]

Xavier Décoret, Frédo Durand, François X. Sillion, and Julie Dorsey. 2003. Billboard Clouds for Extreme Model Simplification. ACM Trans. Graph. 22, 3 (July 2003), 689--696.

Digital Library

[12]

Hugh Everett. 1963. Generalized Lagrange Multiplier Method for Solving Problems of Optimum Allocation of Resources. Operations Research 11, 3 (1963), 399--417.

Digital Library

[13]

Marshall L. Fisher. 2004. The Lagrangian Relaxation Method for Solving Integer Programming Problems. Management Science 50, 12 (2004), 1861--1871.

Digital Library

[14]

Anton L. Fuhrmann, Eike Umlauf, and Stephan Mantler. 2005. Extreme Model Simplification for Forest Rendering. In Proceedings of the First Eurographics Conference on Natural Phenomena (NPH'05). Eurographics Association, 57--67.

Digital Library

[15]

Michael Garland and Paul S. Heckbert. 1997. Surface Simplification Using Quadric Error Metrics. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '97). 209--216.

Digital Library

[16]

Enrico Gobbetti and Fabio Marton. 2005. Far Voxels: A Multiresolution Framework for Interactive Rendering of Huge Complex 3D Models on Commodity Graphics Platforms. In ACM SIGGRAPH 2005 Papers (SIGGRAPH '05). 878--885.

Digital Library

[17]

Google Inc. 2018. Expeditions. (2018). https://play.google.com/store/apps/details?id=com.google.vr.expeditions

[18]

Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen. 1996. The Lumigraph. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '96). 43--54.

Digital Library

[19]

J. Huang, Z. Chen, D. Ceylan, and H. Jin. 2017. 6-DOF VR videos with a single 360-camera. In 2017 IEEE Virtual Reality (VR). 37--44.

[20]

H. Ishiguro, M. Yamamoto, and S. Tsuji. 1990. Omni-directional stereo for making global map. In Proceedings Third International Conference on Computer Vision. 540--547.

[21]

Stefan Jeschke and Michael Wimmer. 2002. Textured Depth Meshes for Real-time Rendering of Arbitrary Scenes. In Eurographics Workshop on Rendering, P. Debevec and S. Gibson (Eds.). The Eurographics Association.

Digital Library

[22]

Alexander Kharlamov, Iain Cantlay, and Yury Stepanenko. 2007. Next-Generation SpeedTree Rendering. In GPU Gems 3 (first ed.), Hubert Nguyen (Ed.). Addison-Wesley Professional.

[23]

Babis Koniaris, Ivan Huerta, Maggie Kosek Karen Darragh, Charles Malleson, Joanna Jamrozy, Nick Swafford, Jose Guitian, Bochang Moon, Ali Israr, and Kenny Mitchell. 2016. IRIDiuM: Immersive Rendered Interactive Deep Media. In ACM SIGGRAPH 2016 VR Village (SIGGRAPH '16). Article 11, 2 pages.

Digital Library

[24]

Babis Koniaris, Maggie Kosek, David Sinclair, and Kenny Mitchell. 2017. Real-time Rendering with Compressed Animated Light Fields. In Proceedings of the 43rd Graphics Interface Conference (GI '17). 33--40.

Digital Library

[25]

Martin Kraus and Thomas Ertl. 2002. Adaptive Texture Maps. In Proceedings of the ACM SIGGRAPH/EUROGRAPHICS Conference on Graphics Hardware (HWWS '02). Eurographics Association, 7--15.

Digital Library

[26]

K.N. Kutulakos and S.M. Seitz. 1999. A Theory of Shape by Space Carving. In The Proceedings of the Seventh IEEE International Conference on Computer Vision. 307--314.

[27]

Dylan Lacewell, David Edwards, Peter Shirley, and William B. Thompson. 2006. Stochastic Billboard Clouds for Interactive Foliage Rendering. Journal of Graphics Tools 11 (1 2006), 1--12.

[28]

Marc Levoy and Pat Hanrahan. 1996. Light Field Rendering. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '96). 31--42.

Digital Library

[29]

Bruno Lévy, Sylvain Petitjean, Nicolas Ray, and Jérome Maillot. 2002. Least Squares Conformal Maps for Automatic Texture Atlas Generation. ACM Trans. Graph. 21, 3 (July 2002), 362--371.

Digital Library

[30]

A. Lodi, S. Martello, and D. Vigo. 2004. Models and Bounds for Two-Dimensional Level Packing Problems. Journal of Combinatorial Optimization 8, 3 (9 2004), 363--379.

[31]

B. Luo, F. Xu, C. Richardt, and J. H. Yong. 2018. Parallax360: Stereoscopic 360° Scene Representation for Head-Motion Parallax. IEEE Transactions on Visualization and Computer Graphics (2018).

Digital Library

[32]

Manufactura K4. 2015. Asia -- Far East Environment. https://assetstore.unity.com/packages/3d/environments/asia-far-east-environment-21298. (2015). Accessed on 2018-01-22.

[33]

Harald Niederreiter. 1992. Random Number Generation and Quasi-Monte Carlo Methods. Society for Industrial and Applied Mathematics.

Digital Library

[34]

J. Nystad, A. Lassen, A. Pomianowski, S. Ellis, and T. Olson. 2012. Adaptive Scalable Texture Compression. In Proceedings of the Fourth ACM SIGGRAPH/Eurographics Conference on High-Performance Graphics (EGGH-HPG'12). Eurographics Association, 105--114.

Digital Library

[35]

S. Peleg, M. Ben-Ezra, and Y. Pritch. 2001. Omnistereo: panoramic stereo imaging. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 3 (Mar 2001), 279--290.

Digital Library

[36]

Eric Penner and Li Zhang. 2017. Soft 3D Reconstruction for View Synthesis. ACM Trans. Graph. 36, 6, Article 235 (Nov. 2017), 11 pages.

Digital Library

[37]

Hanspeter Pfister, Matthias Zwicker, Jeroen van Baar, and Markus Gross. 2000. Surfels: Surface Elements As Rendering Primitives. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '00). 335--342.

Digital Library

[38]

Matt Pharr. 2017. Mobile VR: Challenges and Opportunities. (2017). http://pharr.org/matt/mobilevr.pdf ACM I3D Keynote.

[39]

Thomas Porter and Tom Duff. 1984. Compositing Digital Images. SIGGRAPH Comput. Graph. 18, 3 (Jan. 1984), 253--259.

Digital Library

[40]

Liu Ren, Hanspeter Pfister, and Matthias Zwicker. 2002. Object Space EWA Surface Splatting: A Hardware Accelerated Approach to High Quality Point Rendering. Computer Graphics Forum 21, 3 (2002), 461--470.

[41]

C. Richardt, Y. Pritch, H. Zimmer, and A. Sorkine-Hornung. 2013. Megastereo: Constructing High-Resolution Stereo Panoramas. In 2013 IEEE Conference on Computer Vision and Pattern Recognition. 1256--1263.

Digital Library

[42]

Takafumi Saito and Tokiichiro Takahashi. 1991. NC Machining with G-buffer Method. In Proceedings of the 18th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '91). 207--216.

Digital Library

[43]

Gernot Schaufler. 1998. Image-based Object Representation by Layered Impostors. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology (VRST '98). 99--104.

Digital Library

[44]

Mark Segal, Carl Korobkin, Rolf van Widenfelt, Jim Foran, and Paul Haeberli. 1992. Fast Shadows and Lighting Effects Using Texture Mapping. In Proceedings of the 19th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '92). 249--252.

Digital Library

[45]

Jonathan Shade, Steven Gortler, Li-Wei He, and Richard Szeliski. 1998. Layered Depth Images. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '98). 231--242.

Digital Library

[46]

Unity Technologies. 2015. The Blacksmith Environments. https://assetstore.unity.com/packages/3d/environments/asia-far-east-environment-21298. (2015). Accessed on 2018-01-22.

[47]

Alex Vlachos. 2016. Advanced VR Rendering Performance. (2016). https://alex.vlachos.com/graphics/Alex_Vlachos_Advanced_VR_Rendering_Performance_GDC2016.pdf GDC Talk.

[48]

Zhou Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. 2004. Image Quality Assessment: From Error Visibility to Structural Similarity. Trans. Img. Proc. 13, 4 (April 2004), 600--612.

Digital Library

[49]

Andrew Wilson and Dinesh Manocha. 2003. Simplifying Complex Environments Using Incremental Textured Depth Meshes. In ACM SIGGRAPH 2003 Papers (SIGGRAPH '03). 678--688.

Digital Library

[50]

Daniel N. Wood, Daniel I. Azuma, Ken Aldinger, Brian Curless, Tom Duchamp, David H. Salesin, and Werner Stuetzle. 2000. Surface Light Fields for 3D Photography. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '00). 287--296.

Digital Library

[51]

Matthias Zwicker, Hanspeter Pfister, Jeroen van Baar, and Markus Gross. 2001. Surface Splatting. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '01). 371--378.

Digital Library

Cited By

Hirzle TMüller FDraxler FSchmitz MKnierim PHornbæk K(2023)When XR and AI Meet - A Scoping Review on Extended Reality and Artificial IntelligenceProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581072(1-45)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581072
Hladky JStengel MVining NKerbl BSeidel HSteinberger M(2022)QuadStreamACM Transactions on Graphics10.1145/3550454.355552441:6(1-13)Online publication date: 30-Nov-2022
https://dl.acm.org/doi/10.1145/3550454.3555524
Misiak MFuhrmann ALatoschik M(2021)Impostor-based Rendering Acceleration for Virtual, Augmented, and Mixed RealityProceedings of the 27th ACM Symposium on Virtual Reality Software and Technology10.1145/3489849.3489865(1-10)Online publication date: 8-Dec-2021
https://dl.acm.org/doi/10.1145/3489849.3489865

Index Terms

View-Region Optimized Image-Based Scene Simplification
1. Computing methodologies
  1. Computer graphics

Recommendations

Simplification of Moving 3D Scene Data on GPU
ICETE 2016: Proceedings of the 13th International Joint Conference on e-Business and Telecommunications

Real-time large scale continuous image and geometry based data visualization, with an uninterrupted content

delivery, quality and rendering, on home and mobile devices is difficult or even mostly impossible because of

the low processing capabilities of ...
Image-based bidirectional scene reprojection

We introduce a method for increasing the framerate of real-time rendering applications. Whereas many existing temporal upsampling strategies only reuse information from previous frames, our bidirectional technique reconstructs intermediate frames from a ...
Physically-Based Editing of Indoor Scene Lighting from a Single Image
Computer Vision – ECCV 2022
Abstract
We present a method to edit complex indoor lighting from a single image with its predicted depth and light source segmentation masks. This is an extremely challenging problem that requires modeling complex light transport, and disentangling HDR ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Computer Graphics and Interactive Techniques

Proceedings of the ACM on Computer Graphics and Interactive Techniques Volume 1, Issue 2

August 2018

223 pages

EISSN:2577-6193

DOI:10.1145/3273023

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 August 2018

Published in PACMCGIT Volume 1, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
288
Total Downloads

Downloads (Last 12 months)20
Downloads (Last 6 weeks)2

Reflects downloads up to 27 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hirzle TMüller FDraxler FSchmitz MKnierim PHornbæk K(2023)When XR and AI Meet - A Scoping Review on Extended Reality and Artificial IntelligenceProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581072(1-45)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581072
Hladky JStengel MVining NKerbl BSeidel HSteinberger M(2022)QuadStreamACM Transactions on Graphics10.1145/3550454.355552441:6(1-13)Online publication date: 30-Nov-2022
https://dl.acm.org/doi/10.1145/3550454.3555524
Misiak MFuhrmann ALatoschik M(2021)Impostor-based Rendering Acceleration for Virtual, Augmented, and Mixed RealityProceedings of the 27th ACM Symposium on Virtual Reality Software and Technology10.1145/3489849.3489865(1-10)Online publication date: 8-Dec-2021
https://dl.acm.org/doi/10.1145/3489849.3489865

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents