abstract

Reprojection-Free VR Passthrough

Authors:

Nathan MatsudaAuthors Info & Claims

SIGGRAPH '23: ACM SIGGRAPH 2023 Emerging Technologies

Article No.: 14, Pages 1 - 2

https://doi.org/10.1145/3588037.3595391

Published: 26 July 2023 Publication History

Get Access

Abstract

Virtual reality (VR) passthrough uses external cameras on the front of a headset to allow the user to see their environment. However, passthrough cameras cannot physically be co-located with the user’s eyes, so the passthrough images have a different perspective than what the user would see without the headset. Although the images can be computationally reprojected into the desired view, errors in depth estimation and missing information at occlusion boundaries can lead to undesirable artifacts.

We propose a novel computational camera that directly samples the rays that would have gone into the user’s eye, several centimeters behind the sensor. Our design contains an array of lenses with an aperture behind each lens, and the apertures are strategically placed to allow through only the desired rays. The resulting thin, flat architecture has suitable form factor for VR, and the image reconstruction is computationally lightweight, enabling low-latency passthrough. We demonstrate our approach experimentally in a fully functional binocular passthrough prototype with practical calibration and real-time image reconstruction.

References

[1]

Gaurav Chaurasia, Arthur Nieuwoudt, Alexandru-Eugen Ichim, Richard Szeliski, and Alexander Sorkine-Hornung. 2020. Passthrough+ real-time stereoscopic view synthesis for mobile mixed reality. Proceedings of the ACM on Computer Graphics and Interactive Techniques 3, 1 (2020), 1–17.

Digital Library

Google Scholar

[2]

Sei-Young Kim, Joong Ho Lee, and Ji Hyung Park. 2014. The effects of visual displacement on simulator sickness in video see-through head-mounted displays. In Proceedings of the ACM International Symposium on Wearable Computers. 79–82.

Digital Library

Google Scholar

[3]

Anat Levin, Assaf Zomet, Shmuel Peleg, and Yair Weiss. 2004. Seamless image stitching in the gradient domain. In European Conference on Computer Vision. Springer, 377–389.

Crossref

Google Scholar

[4]

Marc Levoy and Pat Hanrahan. 1996. Light field rendering. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques. 31–42.

Digital Library

Google Scholar

[5]

Jannick P Rolland, Frank A Biocca, Todd Barlow, and Anantha Kancherla. 1995. Quantification of adaptation to virtual-eye location in see-thru head-mounted displays. In Proceedings Virtual Reality Annual International Symposium’95. IEEE, 56–66.

Crossref

Google Scholar

[6]

Akinari Takagi, Shoichi Yamazaki, Yoshihiro Saito, and Naosato Taniguchi. 2000. Development of a stereo video see-through HMD for AR systems. In Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000). IEEE, 68–77.

Crossref

Google Scholar

[7]

Lei Xiao, Salah Nouri, Joel Hegland, Alberto Garcia Garcia, and Douglas Lanman. 2022. NeuralPassthrough: Learned Real-Time View Synthesis for VR. In ACM SIGGRAPH 2022 Conference Proceedings. 1–9.

Google Scholar

Cited By

View all

Guo ZWang HDeng HXu WBaghaei NLo CLiang H(2024)Breaking the Isolation: Exploring the Impact of Passthrough in Shared Spaces on Player Performance and Experience in VR ExergamesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337211430:5(2580-2590)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372114

Recommendations

Perspective-Correct VR Passthrough Without Reprojection
SIGGRAPH '23: ACM SIGGRAPH 2023 Conference Proceedings

Virtual reality (VR) passthrough uses external cameras on the front of a headset to allow the user to see their environment. However, passthrough cameras cannot physically be co-located with the user’s eyes, so the passthrough images have a different ...
Passthrough+: Real-time Stereoscopic View Synthesis for Mobile Mixed Reality

We present an end-to-end system for real-time environment capture, 3D reconstruction, and stereoscopic view synthesis on a mobile VR headset. Our solution allows the user to use the cameras on their VR headset as their eyes to see and interact with the ...
GPU Passthrough Performance: A Comparison of KVM, Xen, VMWare ESXi, and LXC for CUDA and OpenCL Applications
CLOUD '14: Proceedings of the 2014 IEEE International Conference on Cloud Computing

As more scientific workloads are moved into the cloud, the need for high performance accelerators increases. Accelerators such as GPUs offer improvements in both performance and power efficiency over traditional multi-core processors, however, their use ...

Comments

Information & Contributors

Information

Published In

SIGGRAPH '23: ACM SIGGRAPH 2023 Emerging Technologies

July 2023

44 pages

ISBN:9798400701542

DOI:10.1145/3588037

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 July 2023

Check for updates

Qualifiers

Abstract
Research
Refereed limited

Conference

SIGGRAPH '23

Sponsor:

SIGGRAPH

SIGGRAPH '23: Special Interest Group on Computer Graphics and Interactive Techniques Conference

August 6 - 10, 2023

CA, Los Angeles, USA

Acceptance Rates

Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
328
Total Downloads

Downloads (Last 12 months)94
Downloads (Last 6 weeks)6

Reflects downloads up to 17 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Guo ZWang HDeng HXu WBaghaei NLo CLiang H(2024)Breaking the Isolation: Exploring the Impact of Passthrough in Shared Spaces on Player Performance and Experience in VR ExergamesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337211430:5(2580-2590)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372114

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Abstract

References

Cited By

Recommendations

Perspective-Correct VR Passthrough Without Reprojection

Passthrough+: Real-time Stereoscopic View Synthesis for Mobile Mixed Reality

GPU Passthrough Performance: A Comparison of KVM, Xen, VMWare ESXi, and LXC for CUDA and OpenCL Applications

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

HTML Format

Share

Share this Publication link

Share on social media

Affiliations