research-article

Open access

Impostor-based Rendering Acceleration for Virtual, Augmented, and Mixed Reality

Authors:

Arnulph Fuhrmann,

Marc Erich LatoschikAuthors Info & Claims

VRST '21: Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology

Article No.: 3, Pages 1 - 10

https://doi.org/10.1145/3489849.3489865

Published: 08 December 2021 Publication History

All formats PDF

Abstract

This paper presents an image-based rendering approach to accelerate rendering time of virtual scenes containing a large number of complex high poly count objects. Our approach replaces complex objects by impostors, light-weight image-based representations leveraging geometry and shading related processing costs. In contrast to their classical implementation, our impostors are specifically designed to work in Virtual-, Augmented- and Mixed Reality scenarios (XR for short), as they support stereoscopic rendering to provide correct depth perception. Motion parallax of typical head movements is compensated by using a ray marched parallax correction step. Our approach provides a dynamic run-time recreation of impostors as necessary for larger changes in view position. The dynamic run-time recreation is decoupled from the actual rendering process. Hence, its associated processing cost is therefore distributed over multiple frames. This avoids any unwanted frame drops or latency spikes even for impostors of objects with complex geometry and many polygons. In addition to the significant performance benefit, our impostors compare favorably against the original mesh representation, as geometric and textural temporal aliasing artifacts are heavily suppressed.

Supplementary Material

Appendix (Impostor_based_Rendering_System_for_XR_Appendix.pdf)

Download
8.32 MB

MP4 File (Impostor_based_Rendering_System_for_XR_VRST2021.mp4)

Supplemental video

Download
137.56 MB

References

[1]

2018. Unity - Octahedral Impostors. https://github.com/xraxra/IMP. Accessed: 2021-07-05.

[2]

2020. Godot - Octahedral Impostors. https://github.com/wojtekpil/Godot-Octahedral-Impostors. Accessed: 2021-07-05.

[3]

Carlos Andújar, Javier Boo, Pere Brunet, Marta Fairén, Isabel Navazo, Pere Vazquez, and A Vinacua. 2007. Omni-directional Relief Impostors. Computer Graphics Forum 26, 3, 553–560.

[4]

Lionel Baboud and Xavier Décoret. 2006. Rendering geometry with relief textures. In Proceedings of Graphics Interface 2006. Canadian Information Processing Society, 195–201.

Digital Library

[5]

Kevin Boos, David Chu, and Eduardo Cuervo. 2016. Flashback: Immersive virtual reality on mobile devices via rendering memoization. In Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services. 291–304.

Digital Library

[6]

Ryan Brucks. 2018. Octahedral Impostors. https://www.shaderbits.com/blog/octahedral-impostors. Accessed: 2021-07-05.

[7]

Jonathan Cohen, Marc Olano, and Dinesh Manocha. 1998. Appearance-preserving simplification. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques. 115–122.

Digital Library

[8]

Xavier Décoret, Frédo Durand, François X Sillion, and Julie Dorsey. 2003. Billboard clouds for extreme model simplification. In ACM SIGGRAPH 2003 Papers. 689–696.

Digital Library

[9]

Jon Hasselgren, Jacob Munkberg, Marco Salvi, Anjul Patney, and Aaron Lefohn. 2020. Neural temporal adaptive sampling and denoising. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 147–155.

[10]

Jozef Hladky, Hans-Peter Seidel, and Markus Steinberger. 2019. Tessellated Shading Streaming. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 171–182.

[11]

Lloyd A Jeffress. 1937. Anomalous Parallax in Anaglyphs and Stereograms. The Journal of General Psychology 17, 2 (1937), 389–394.

[12]

Zeqi Lai, Y Charlie Hu, Yong Cui, Linhui Sun, Ningwei Dai, and Hung-Sheng Lee. 2019. Furion: Engineering high-quality immersive virtual reality on today’s mobile devices. IEEE Transactions on Mobile Computing(2019).

[13]

Puneet Lall, Silviu Borac, Dave Richardson, Matt Pharr, and Manfred Ernst. 2018. View-Region Optimized Image-Based Scene Simplification. Proceedings of the ACM on Computer Graphics and Interactive Techniques 1, 2(2018), 1–22.

Digital Library

[14]

Kyungmin Lee, David Chu, Eduardo Cuervo, Johannes Kopf, Yury Degtyarev, Sergey Grizan, Alec Wolman, and Jason Flinn. 2015. Outatime: Using speculation to enable low-latency continuous interaction for mobile cloud gaming. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services. 151–165.

Digital Library

[15]

Yong Li and Wei Gao. 2019. DeltaVR: Achieving high-performance mobile VR dynamics through pixel reuse. In Proceedings of the 18th International Conference on Information Processing in Sensor Networks. 13–24.

Digital Library

[16]

Paulo WC Maciel and Peter Shirley. 1995. Visual navigation of large environments using textured clusters. In Proceedings of the 1995 symposium on Interactive 3D graphics. ACM, 95–ff.

Digital Library

[17]

Alexander Majercik, Cyril Crassin, Peter Shirley, and Morgan McGuire. 2018. A ray-box intersection algorithm and efficient dynamic voxel rendering. Journal of Computer Graphics Techniques Vol 7, 3 (2018).

[18]

Stephan Mantler, Stefan Jeschke, and Michael Wimmer. 2007. Displacement mapped billboard clouds. In Proceedings of symposium on interactive 3D graphics and games. Citeseer.

[19]

William R Mark, Leonard McMillan, and Gary Bishop. 1997. Post-rendering 3D warping. In Proceedings of the 1997 symposium on Interactive 3D graphics. 7–ff.

Digital Library

[20]

Joerg H Mueller, Philip Voglreiter, Mark Dokter, Thomas Neff, Mina Makar, Markus Steinberger, and Dieter Schmalstieg. 2018. Shading atlas streaming. ACM Transactions on Graphics (TOG) 37, 6 (2018), 1–16.

Digital Library

[21]

Manuel M Oliveira, Gary Bishop, and David McAllister. 2000. Relief texture mapping. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 359–368.

Digital Library

[22]

Edward M Peek, Burkhard C Wünsche, and Christof Lutteroth. 2014. Image warping for enhancing consumer applications of head-mounted displays. In Proceedings of the Fifteenth Australasian User Interface Conference-Volume 150. 47–55.

Digital Library

[23]

Fabio Policarpo and Manuel M Oliveira. 2006. Relief mapping of non-height-field surface details. In Proceedings of the 2006 symposium on Interactive 3D graphics and games. ACM, 55–62.

Digital Library

[24]

Fábio Policarpo, Manuel M Oliveira, and João LD Comba. 2005. Real-time relief mapping on arbitrary polygonal surfaces. In Proceedings of the 2005 symposium on Interactive 3D graphics and games. ACM, 155–162.

Digital Library

[25]

Voicu Popescu, Kyle Hayward, Paul Rosen, and Chris Wyman. 2009. Non-Pinhole Imposters. Technical Report 09-006. Purdue University.

[26]

Bernhard Reinert, Johannes Kopf, Tobias Ritschel, Eduardo Cuervo, David Chu, and Hans-Peter Seidel. 2016. Proxy-guided image-based rendering for mobile devices. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 353–362.

[27]

Eric Risser, Musawir Shah, and Sumanta Pattanaik. 2007. Faster relief mapping using the secant method. Journal of graphics tools 12, 3 (2007), 17–24.

[28]

Gernot Schaufler. 1996. Exploiting frame-to-frame coherence in a virtual reality system. In Proceedings of the IEEE 1996 Virtual Reality Annual International Symposium. IEEE, 95–102.

[29]

Gernot Schaufler. 1998. Per-object image warping with layered impostors. In Eurographics Workshop on Rendering Techniques. Springer, 145–156.

[30]

Gernot Schaufler and Wolfgang Stürzlinger. 1996. A three dimensional image cache for virtual reality. In Computer Graphics Forum, Vol. 15. Wiley Online Library, 227–235.

[31]

Jonathan Shade, Steven Gortler, Li-wei He, and Richard Szeliski. 1998. Layered depth images. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques. 231–242.

Digital Library

[32]

Jonathan Shade, Dani Lischinski, David H Salesin, Tony DeRose, and John Snyder. 1996. Hierarchical image caching for accelerated walkthroughs of complex environments. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. ACM, 75–82.

Digital Library

[33]

Ferdi Smit, Robert van Liere, Stephan Beck, and Bernd Fröhlich. 2009. An image-warping architecture for vr: Low latency versus image quality. In 2009 IEEE Virtual Reality Conference. IEEE, 27–34.

[34]

Jan-Philipp Stauffert, Kristof Korwisi, Florian Niebling, and Marc Erich Latoschik. 2021. Ka-Boom!!! Visually Exploring Latency Measurements for XR. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems(CHI EA ’21). Association for Computing Machinery, New York, NY, USA.

Digital Library

[35]

Jan-Philipp Stauffert, Florian Niebling, and Marc Erich Latoschik. 2018. Effects of Latency Jitter on Simulator Sickness in a Search Task. In 25th IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR). 121–127.

[36]

Jan-Philipp Stauffert, Florian Niebling, and Marc Erich Latoschik. 2020. Latency and Cybersickness: Impact, Causes, and Measures. A Review. Frontiers in Virtual Reality 1 (2020), 31.

[37]

László Szirmay-Kalos and Tamás Umenhoffer. 2008. Displacement Mapping on the GPU—State of the Art. In Computer graphics forum, Vol. 27. Wiley Online Library, 1567–1592.

[38]

Natalya Tatarchuk. 2006. Dynamic parallax occlusion mapping with approximate soft shadows. In Proceedings of the 2006 symposium on Interactive 3D graphics and games. 63–69.

Digital Library

[39]

Franco Tecchia and Yiorgos Chrysanthou. 2000. Real-time rendering of densely populated urban environments. In Rendering Techniques 2000. Springer, 83–88.

[40]

Severin Todt, Christof Rezk-Salama, Andreas Kolb, and K-D Kuhnert. 2008. GPU-Based Spherical Light Field Rendering with Per-Fragment Depth Correction. In Computer Graphics Forum, Vol. 27. Wiley Online Library, 2081–2095.

[41]

JMP Van Waveren. 2016. The asynchronous time warp for virtual reality on consumer hardware. In Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology. 37–46.

Digital Library

[42]

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600–612.

Digital Library

[43]

Sven Widmer, D Pająk, André Schulz, Kari Pulli, Jan Kautz, Michael Goesele, and David Luebke. 2015. An adaptive acceleration structure for screen-space ray tracing. In Proceedings of the 7th Conference on High-Performance Graphics. 67–76.

Digital Library

[44]

Graham Wihlidal. 2016. Optimizing the graphics pipeline with compute. In Game Developers Conference. 54.

[45]

Niko Wißmann, Martin Mišiak, Arnulph Fuhrmann, and Marc Erich Latoschik. 2020. Accelerated Stereo Rendering with Hybrid Reprojection-Based Rasterization and Adaptive Ray-Tracing. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 828–835.

[46]

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition. 586–595.

Cited By

Waldow KDecker LMišiak MFuhrmann ARoth DLatoschik M(2024)Investigating Incoherent Depth Perception Features in Virtual Reality Using Stereoscopic Impostor-Based Rendering2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00257(921-922)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VRW62533.2024.00257
Geringer SGeiselhart FBäuerle ADec DOdenthal OReina GRopinski TWeiskopf D(2024)mint: Integrating scientific visualizations into virtual realityJournal of Visualization10.1007/s12650-024-01011-yOnline publication date: 5-Aug-2024
https://doi.org/10.1007/s12650-024-01011-y
Mišiak MFuhrmann ALatoschik M(2023)The Impact of Reflection Approximations on Visual Quality in Virtual RealityACM Symposium on Applied Perception 202310.1145/3605495.3605794(1-11)Online publication date: 5-Aug-2023
https://dl.acm.org/doi/10.1145/3605495.3605794
Show More Cited By

Recommendations

Image-based rendering of diffuse, specular and glossy surfaces from a single image
SIGGRAPH '01: Proceedings of the 28th annual conference on Computer graphics and interactive techniques

In this paper, we present a new method to recover an approximation of the bidirectional reflectance distribution function (BRDF) of the surfaces present in a real scene. This is done from a single photograph and a 3D geometric model of the scene. The ...
Sample-Based Cameras for Feed Forward Reflection Rendering

This paper presents sample-based cameras for rendering high quality reflections on convex reflectors at interactive rates. The method supports change of view, moving objects and reflectors, higher order reflections, view-dependent lighting of reflected ...
Relief mapping of non-height-field surface details
I3D '06: Proceedings of the 2006 symposium on Interactive 3D graphics and games

The ability to represent non-height-field mesostructure details is of great importance for rendering complex surface patterns, such as weave and multilayer structures. Currently, such representations are based on the use of 3D textures or large datasets ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

VRST '21: Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology

December 2021

563 pages

ISBN:9781450390927

DOI:10.1145/3489849

Editors:
Yuichi Itoh
Aoyama Gakuin University, Japan
,
Kazuki Takashima
Tohoku University, Japan
,
Parinya Punpongsanon
Osaka University, Japan
,
Misha Sra
University of California, Santa Barbara, USA
,
Kazuyuki Fujita
Tohoku University, Japan
,
Shigeo Yoshida
The University of Tokyo, Japan
,
Tham Piumsomboon
University of Canterbury, New Zealand

Copyright © 2021 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 December 2021

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Ministry of Culture and Science of the State of North Rhine-Westphalia
German Federal Ministry of Education and Research (BMBF)

Conference

VRST '21

Sponsor:

VRST '21: 27th ACM Symposium on Virtual Reality Software and Technology

December 8 - 10, 2021

Osaka, Japan

Acceptance Rates

Overall Acceptance Rate 66 of 254 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
2,004
Total Downloads

Downloads (Last 12 months)679
Downloads (Last 6 weeks)94

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Waldow KDecker LMišiak MFuhrmann ARoth DLatoschik M(2024)Investigating Incoherent Depth Perception Features in Virtual Reality Using Stereoscopic Impostor-Based Rendering2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00257(921-922)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VRW62533.2024.00257
Geringer SGeiselhart FBäuerle ADec DOdenthal OReina GRopinski TWeiskopf D(2024)mint: Integrating scientific visualizations into virtual realityJournal of Visualization10.1007/s12650-024-01011-yOnline publication date: 5-Aug-2024
https://doi.org/10.1007/s12650-024-01011-y
Mišiak MFuhrmann ALatoschik M(2023)The Impact of Reflection Approximations on Visual Quality in Virtual RealityACM Symposium on Applied Perception 202310.1145/3605495.3605794(1-11)Online publication date: 5-Aug-2023
https://dl.acm.org/doi/10.1145/3605495.3605794
Ishihara AAga HIshihara YIchikawa HKaji HKawasaki KKobayashi DKobayashi TNishida KHamasaki TMori HMorikubo Y(2023)Integrating Both Parallax and Latency Compensation into Video See-through Head-mounted DisplayIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324746029:5(2826-2836)Online publication date: 27-Feb-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3247460
Liu SXie JWang X(2023)QoE enhancement of the industrial metaverse based on Mixed Reality application optimizationDisplays10.1016/j.displa.2023.10246379(102463)Online publication date: Sep-2023
https://doi.org/10.1016/j.displa.2023.102463

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents