research-article

Open access

Holographic Parallax Improves 3D Perceptual Realism

Authors:

Gordon Wetzstein,

Yoonchan JeongAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 43, Issue 4

Article No.: 68, Pages 1 - 13

https://doi.org/10.1145/3658168

Published: 19 July 2024 Publication History

Abstract

Holographic near-eye displays are a promising technology to solve long-standing challenges in virtual and augmented reality display systems. Over the last few years, many different computer-generated holography (CGH) algorithms have been proposed that are supervised by different types of target content, such as 2.5D RGB-depth maps, 3D focal stacks, and 4D light fields. It is unclear, however, what the perceptual implications are of the choice of algorithm and target content type. In this work, we build a perceptual testbed of a full-color, high-quality holographic near-eye display. Under natural viewing conditions, we examine the effects of various CGH supervision formats and conduct user studies to assess their perceptual impacts on 3D realism. Our results indicate that CGH algorithms designed for specific viewpoints exhibit noticeable deficiencies in achieving 3D realism. In contrast, holograms incorporating parallax cues consistently outperform other formats across different viewing conditions, including the center of the eyebox. This finding is particularly interesting and suggests that the inclusion of parallax cues in CGH rendering plays a crucial role in enhancing the overall quality of the holographic experience. This work represents an initial stride towards delivering a perceptually realistic 3D experience with holographic near-eye displays.

Supplementary Material

ZIP File (papers_410.zip)

supplemental

Download
450.62 MB

References

[1]

A. T. Bahill, M. R. Clark, and L. Stark. The main sequence, a tool for studying human eye movements. Mathematical biosciences, 24(3--4):191--204, 1975.

[2]

R. Bogacz, E. Brown, J. Moehlis, P. Holmes, and J. D. Cohen. The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. Psychological Review, 113(4):700, 2006.

[3]

O. Bryngdahl and A. Lohmann. Single-sideband holography. JOSA, 58(5):620--624, 1968.

[4]

P. Chakravarthula, E. Tseng, T. Srivastava, H. Fuchs, and F. Heide. Learned hardware-in-the-loop phase retrieval for holographic near-eye displays. ACM Transactions on Graphics (TOG), 39(6):1--18, 2020.

[5]

P. Chakravarthula, S.-H. Baek, F. Schiffers, E. Tseng, G. Kuo, A. Maimone, N. Matsuda, O. Cossairt, D. Lanman, and F. Heide. Pupil-aware holography. ACM Transactions on Graphics (TOG), 41(6):1--15, 2022.

[6]

J.-H. R. Chang, A. Levin, B. V. Kumar, and A. C. Sankaranarayanan. Towards occlusion-aware multifocal displays. ACM Transactions on Graphics (TOG), 39(4):68--1, 2020.

[7]

C. Chen, B. Lee, N.-N. Li, M. Chae, D. Wang, Q.-H. Wang, and B. Lee. Multi-depth hologram generation using stochastic gradient descent algorithm with complex loss function. Opt. Express, 29(10):15089--15103, 2021.

[8]

S. Choi, M. Gopakumar, Y. Peng, J. Kim, and G. Wetzstein. Neural 3d holography: Learning accurate wave propagation models for 3d holographic virtual and augmented reality displays. ACM Trans. Graph. (SIGGRAPH Asia), 2021.

Digital Library

[9]

S. Choi, M. Gopakumar, Y. Peng, J. Kim, M. O'Toole, and G. Wetzstein. Time-multiplexed neural holography: a flexible framework for holographic near-eye displays with fast heavily-quantized spatial light modulators. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1--9, 2022.

Digital Library

[10]

V. R. Curtis, N. W. Caira, J. Xu, A. G. Sata, and N. C. Pégard. Dcgh: dynamic computer generated holography for speckle-free, high fidelity 3d displays. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR), pages 1--9. IEEE, 2021.

[11]

J. E. Cutting and P. M. Vishton. Perceiving layout and knowing distances: The integration, relative potency, and contextual use of different information about depth. In Perception of space and motion, pages 69--117. Elsevier, 1995.

[12]

S. De Groot and J. Gebhard. Pupil size as determined by adapting luminance. JOSA, 42 (7):492--495, 1952.

[13]

A. Duane. Normal values of the accommodation at all ages. Journal of the American Medical Association, 59(12):1010--1013, 1912.

[14]

J. W. Goodman. Introduction to Fourier optics. Roberts and Company Publishers, 2005.

[15]

P. Guan, O. Mercier, M. Shvartsman, and D. Lanman. Perceptual requirements for eye-tracked distortion correction in vr. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1--8, 2022.

Digital Library

[16]

B. Guenter, M. Finch, S. Drucker, D. Tan, and J. Snyder. Foveated 3d graphics. ACM transactions on Graphics (tOG), 31(6):1--10, 2012.

[17]

D. M. Hoffman, A. R. Girshick, K. Akeley, and M. S. Banks. Vergence-accommodation conflicts hinder visual performance and cause visual fatigue. Journal of vision, 8(3): 33--33, 2008.

[18]

C. Jang, K. Bang, G. Li, and B. Lee. Holographic near-eye display with expanded eye-box. ACM Trans. Graph., 37(6), dec 2018.

[19]

C. Jang, K. Bang, M. Chae, B. Lee, and D. Lanman. Waveguide holography for 3d augmented reality glasses. Nature Communications, 15(1):66, 2024.

[20]

E. Jang, S. Gu, and B. Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016.

[21]

H. Kang, E. Stoykova, and H. Yoshikawa. Fast phase-added stereogram algorithm for generation of photorealistic 3d content. Applied optics, 55(3):A135--A143, 2016.

[22]

K. Kavaklı, Y. Itoh, H. Urey, and K. Akşit. Realistic defocus blur for multiplane computergenerated holography. In 2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR), pages 418--426. IEEE, 2023.

[23]

D. Kim, S.-W. Nam, K. Bang, B. Lee, S. Lee, Y. Jeong, J.-M. Seo, and B. Lee. Vision-correcting holographic display: evaluation of aberration correcting hologram. Biomedical Optics Express, 12(8):5179--5195, 2021.

[24]

D. Kim, S.-W. Nam, B. Lee, J.-M. Seo, and B. Lee. Accommodative holography: improving accommodation response for perceptually realistic holographic displays. ACM Transactions on Graphics (TOG), 41(4):1--15, 2022.

[25]

V. Kiran Adhikarla, M. Vinkler, D. Sumin, R. K. Mantiuk, K. Myszkowski, H.-P. Seidel, and P. Didyk. Towards a quality metric for dense light fields. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 58--67, 2017.

[26]

R. Konrad, A. Angelopoulos, and G. Wetzstein. Gaze-contingent ocular parallax rendering for virtual reality. ACM Transactions on Graphics (TOG), 39(2):1--12, 2020.

[27]

G. Kuo, L. Waller, R. Ng, and A. Maimone. High resolution étendue expansion for holographic displays. ACM Transactions on Graphics (TOG), 39(4):66--1, 2020.

[28]

B. Lee, D. Yoo, J. Jeong, S. Lee, D. Lee, and B. Lee. Wide-angle speckleless dmd holographic display using structured illumination with temporal multiplexing. Optics Letters, 45(8):2148--2151, 2020.

[29]

B. Lee, D. Kim, S. Lee, C. Chen, and B. Lee. High-contrast, speckle-free, true 3d holography via binary cgh optimization. Scientific reports, 12(1):2811, 2022.

[30]

A. W. Lohmann and D. Paris. Binary fraunhofer holograms, generated by computer. Applied optics, 6(10):1739--1748, 1967.

[31]

A. Maimone, A. Georgiou, and J. S. Kollin. Holographic near-eye displays for virtual and augmented reality. ACM Trans. Graph. (SIGGRAPH), 36(4):85, 2017.

[32]

R. K. Mantiuk, G. Denes, A. Chapiro, A. Kaplanyan, G. Rufo, R. Bachy, T. Lian, and A. Patney. Fovvideovdp: A visible difference predictor for wide field-of-view video. ACM Transactions on Graphics (TOG), 40(4):1--19, 2021.

[33]

J. March, A. Krishnan, S. Watt, M. Wernikowski, H. Gao, A. Ö. Yöntem, and R. Mantiuk. Impact of correct and simulated focus cues on perceived realism. In SIGGRAPH Asia 2022 Conference Papers, pages 1--9, 2022.

Digital Library

[34]

K. Matsushima and S. Nakahara. Extremely high-definition full-parallax computergenerated hologram created by the polygon-based method. Applied optics, 48(34): H54--H63, 2009.

[35]

A. Mehrfard, J. Fotouhi, G. Taylor, T. Forster, N. Navab, and B. Fuerst. A comparative analysis of virtual reality head-mounted display systems. arXiv preprint arXiv:1912.02913, 2019.

[36]

O. Mercier, Y. Sulai, K. Mackenzie, M. Zannoli, J. Hillis, D. Nowrouzezahrai, and D. Lanman. Fast gaze-contingent optimal decompositions for multifocal displays. ACM Transactions on Graphics (TOG), 36(6):1--15, 2017.

[37]

S. Monin, A. C. Sankaranarayanan, and A. Levin. Analyzing phase masks for wide étendue holographic displays. In 2022 IEEE International Conference on Computational Photography (ICCP), pages 1--12. IEEE, 2022a.

[38]

S. Monin, A. C. Sankaranarayanan, and A. Levin. Exponentially-wide etendue displays using a tilting cascade. In 2022 IEEE International Conference on Computational Photography (ICCP), pages 1--12. IEEE, 2022b.

[39]

J. J. Naji and T. C. Freeman. Perceiving depth order during pursuit eye movement. Vision research, 44(26):3025--3034, 2004.

[40]

S.-W. Nam, Y. Kim, D. Kim, and Y. Jeong. Depolarized holography with polarization-multiplexing metasurface. ACM Transactions on Graphics (TOG), 42(6):1--16, 2023.

[41]

P. Napieralski and F. Rynkiewicz. Modeling human pupil dilation to decouple the pupillary light reflex. Open Physics, 17(1):458--467, 2019.

[42]

M. Nawrot. Eye movements provide the extra-retinal signal required for the perception of depth from motion parallax. Vision research, 43(14):1553--1562, 2003.

[43]

R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan. Light field photography with a hand-held plenoptic camera. PhD thesis, Stanford University, 2005.

[44]

N. Padmanaban, Y. Peng, and G. Wetzstein. Holographic near-eye displays based on overlap-add stereograms. ACM Trans. Graph., 38(6), 2019a.

Digital Library

[45]

N. Padmanaban, Y. Peng, and G. Wetzstein. Holographic near-eye displays based on overlap-add stereograms. ACM Transactions on Graphics (TOG), 38(6):1--13, 2019b.

[46]

J. Park, K. Lee, and Y. Park. Ultrathin wide-angle large-area digital 3d holographic display using a non-periodic photon sieve. Nature communications, 10(1):1304, 2019.

[47]

J.-H. Park. Recent progress in computer-generated holography for three-dimensional scenes. Journal of Information Display, 18(1):1--12, 2017.

[48]

J.-H. Park. Efficient calculation scheme for high pixel resolution non-hogel-based computer generated hologram from light field. Optics Express, 28(5):6663--6683, 2020.

[49]

J.-H. Park and M. Askari. Non-hogel-based computer generated hologram from light field using complex field recovery technique from wigner distribution function. Optics express, 27(3):2562--2574, 2019.

[50]

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.

[51]

Y. Peng, S. Choi, N. Padmanaban, and G. Wetzstein. Neural holography with camera-in-the-loop training. ACM Trans. Graph., 39(6):1--14, 2020.

Digital Library

[52]

M. Perez-Ortiz and R. K. Mantiuk. A practical guide and software for analysing pairwise comparison experiments. arXiv preprint arXiv:1712.03686, 2017.

[53]

K. Ratnam, R. Konrad, D. Lanman, and M. Zannoli. Retinal image quality in near-eye pupil-steered systems. Optics Express, 27(26):38289--38311, 2019.

[54]

L. Shi, F.-C. Huang, W. Lopes, W. Matusik, and D. Luebke. Near-eye light field holographic rendering with spherical waves for wide field of view interactive 3d computer graphics. ACM Trans. Graph., 36(6), 2017.

Digital Library

[55]

L. Shi, B. Li, C. Kim, P. Kellnhofer, and W. Matusik. Towards real-time photorealistic 3d holography with deep neural networks. Nature, 591(7849):234--239, 2021.

[56]

L. Shi, B. Li, and W. Matusik. End-to-end learning of 3d phase-only holograms for holographic display. Light: Science & Applications, 11(1):247, 2022.

[57]

A. Symeonidou, D. Blinder, A. Munteanu, and P. Schelkens. Computer-generated holograms by multiple wavefront recording plane method with occlusion culling. Optics express, 23(17):22149--22161, 2015.

[58]

F. Wang, T. Ito, and T. Shimobaba. High-speed rendering pipeline for polygon-based holograms. Photonics Research, 11(2):313--328, 2023.

[59]

A. B. Watson and D. G. Pelli. Quest: A bayesian adaptive psychometric method. Perception & psychophysics, 33(2):113--120, 1983.

[60]

G. Westheimer. Directional sensitivity of the retina: 75 years of stiles-crawford effect. Proceedings of the Royal Society B: Biological Sciences, 275(1653):2777--2786, 2008.

[61]

G. Wetzstein and D. Lanman. Factored displays: improving resolution, dynamic range, color reproduction, and light field characteristics with advanced signal processing. IEEE Signal Processing Magazine, 33(5):119--129, 2016.

[62]

D. Yang, W. Seo, H. Yu, S. I. Kim, B. Shin, C.-K. Lee, S. Moon, J. An, J.-Y. Hong, G. Sung, et al. Diffraction-engineered holography: Beyond the depth representation limit of holographic displays. Nature Communications, 13(1):6012, 2022.

[63]

D. Yoo, Y. Jo, S.-W. Nam, C. Chen, and B. Lee. Optimization of computer-generated holograms featuring phase randomness control. Optics Letters, 46(19):4769--4772, 2021.

[64]

H. Zhang, Y. Zhao, L. Cao, and G. Jin. Fully computed holographic stereogram based algorithm for computer-generated holograms with accurate depth cues. Optics express, 23(4):3901--3913, 2015.

[65]

H. Zhang, L. Cao, and G. Jin. Computer-generated hologram with occlusion effect using layer-based processing. Applied optics, 56(13), 2017.

[66]

Z. Zhang and M. Levoy. Wigner distributions and how they relate to the light field. In 2009 IEEE International Conference on Computational Photography (ICCP), pages 1--10. IEEE, 2009.

[67]

F. Zhong, A. Jindal, Ö. Yöntem, P. Hanji, S. Watt, and R. Mantiuk. Reproducing reality with a high-dynamic-range multi-focal stereo display. ACM Transactions on Graphics, 40(6):241, 2021.

Digital Library

Cited By

Leria EMäkitalo MJääskeläinen PSjöström MZhang T(2024)Interactive Multi-GPU Light Field Path Tracing Using Multi-Source Spatial ReprojectionProceedings of the 30th ACM Symposium on Virtual Reality Software and Technology10.1145/3641825.3687710(1-11)Online publication date: 9-Oct-2024
https://dl.acm.org/doi/10.1145/3641825.3687710
Nam SKim DChoi SLee JLee SGopakumar MChao BWetzstein GJeong Y(2024)Holographic ParallaxACM SIGGRAPH 2024 Emerging Technologies10.1145/3641517.3664386(1-2)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641517.3664386

Index Terms

Holographic Parallax Improves 3D Perceptual Realism
1. Hardware
  1. Emerging technologies

Recommendations

Neural 3D holography: learning accurate wave propagation models for 3D holographic virtual and augmented reality displays

Holographic near-eye displays promise unprecedented capabilities for virtual and augmented reality (VR/AR) systems. The image quality achieved by current holographic displays, however, is limited by the wave propagation models used to simulate the ...
Neural Holographic Near-eye Displays for Virtual Reality
SIGGRAPH '23: ACM SIGGRAPH 2023 Emerging Technologies

By manipulating light as a wavefront, holographic displays have the potential to revolutionize virtual reality (VR) and augmented reality (AR) systems. These displays support 3D focus cues for visual comfort, vision correcting capabilities, and high ...
Holographic near-eye displays for virtual and augmented reality

We present novel designs for virtual and augmented reality near-eye displays based on phase-only holographic projection. Our approach is built on the principles of Fresnel holography and double phase amplitude encoding with additional hardware, phase ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 43, Issue 4

July 2024

1774 pages

EISSN:1557-7368

DOI:10.1145/3675116

Issue’s Table of Contents

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 July 2024

Published in TOG Volume 43, Issue 4

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Institute of Information & communications Technology Planning & Evaluation (IITP) / Korea government

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
506
Total Downloads

Downloads (Last 12 months)506
Downloads (Last 6 weeks)152

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Leria EMäkitalo MJääskeläinen PSjöström MZhang T(2024)Interactive Multi-GPU Light Field Path Tracing Using Multi-Source Spatial ReprojectionProceedings of the 30th ACM Symposium on Virtual Reality Software and Technology10.1145/3641825.3687710(1-11)Online publication date: 9-Oct-2024
https://dl.acm.org/doi/10.1145/3641825.3687710
Nam SKim DChoi SLee JLee SGopakumar MChao BWetzstein GJeong Y(2024)Holographic ParallaxACM SIGGRAPH 2024 Emerging Technologies10.1145/3641517.3664386(1-2)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641517.3664386

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents