Large Étendue 3D Holographic Display with Content-adaptive Dynamic Fourier Modulation
Authors: 
Brian Chao, Manu Gopakumar, Suyeon Choi, Jonghyun Kim, Liang Shi, Gordon WetzsteinAuthors Info & Claims
Brian Chao, Manu Gopakumar, Suyeon Choi, Jonghyun Kim, Liang Shi, Gordon WetzsteinAuthors Info & ClaimsArticle No.: 26, Pages 1 - 12
Abstract
Emerging holographic display technology offers unique capabilities for next-generation virtual reality systems. Current holographic near-eye displays, however, only support a small étendue, which results in a direct tradeoff between achievable field of view and eyebox size. Étendue expansion has recently been explored, but existing approaches are either fundamentally limited in the image quality that can be achieved or they require extremely high-speed spatial light modulators. We describe a new étendue expansion approach that combines multiple coherent sources with content-adaptive amplitude modulation of the hologram spectrum in the Fourier plane. To generate time-multiplexed phase and amplitude patterns for our spatial light modulators, we devise a pupil-aware gradient-descent-based computer-generated holography algorithm that is supervised by a large-baseline target light field. Compared with relevant baseline approaches, ours demonstrates significant improvements in image quality and étendue in simulation and with an experimental holographic display prototype.
Supplemental Material
PDF File
Supplemental document and video for paper "Large Étendue 3D Holographic Display with Content-adaptive Dynamic Fourier Modulation".
- Download
- 171.53 MB
MP4 File
Supplemental document and video for paper "Large Étendue 3D Holographic Display with Content-adaptive Dynamic Fourier Modulation".
- Download
- 482.00 MB
References
[1]
Jungkwuen An, Kanghee Won, Young Kim, Jong-Young Hong, Hojung Kim, Yongkyu Kim, Hoon Song, Chilsung Choi, Yunhee Kim, Juwon Seo, Alexander Morozov, Hyunsik Park, Sunghoon Hong, Sungwoo Hwang, Kichul Kim, and Hong-Seok Lee. 2020. Slim-panel holographic video display. Nature Communications 11, 1 (10 Nov 2020), 5568.
[2]
Stephen A Benton and V Michael Bove Jr. 2008. Holographic imaging. John Wiley & Sons.
[3]
Colton M. Bigler, Pierre-Alexandre Blanche, and Kalluri Sarma. 2018. Holographic waveguide heads-up display for longitudinal image magnification and pupil expansion. Appl. Opt. 57, 9 (Mar 2018), 2007–2013.
[4]
Colton M. Bigler, Micah S. Mann, and Pierre-Alexandre Blanche. 2019. Holographic waveguide HUD with in-line pupil expansion and 2D FOV expansion. Appl. Opt. 58, 34 (Dec 2019), G326–G331.
[5]
Edward Buckley, Adrian Cable, Nic Lawrence, and Tim Wilkinson. 2006. Viewing angle enhancement for two- and three-dimensional holographic displays with random superresolution phase masks. Appl. Opt. 45, 28 (Oct 2006), 7334–7341.
[6]
Minseok Chae, Kiseung Bang, Dongheon Yoo, and Yoonchan Jeong. 2023. Étendue Expansion in Holographic Near Eye Displays through Sparse Eye-box Generation Using Lens Array Eyepiece. ACM Trans. Graph. 42, 4 (July 2023), 1–13.
[7]
Praneeth Chakravarthula, Seung-Hwan Baek, Florian Schiffers, Ethan Tseng, Grace Kuo, Andrew Maimone, Nathan Matsuda, Oliver Cossairt, Douglas Lanman, and Felix Heide. 2022. Pupil-Aware Holography. ACM Trans. Graph. 41, 6, Article 212 (nov 2022), 15 pages.
[8]
Praneeth Chakravarthula, Ethan Tseng, Tarun Srivastava, Henry Fuchs, and Felix Heide. 2020. Learned hardware-in-the-loop phase retrieval for holographic near-eye displays. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–18.
[9]
C Chang, K Bang, G Wetzstein, B Lee, and L Gao. 2020. Toward the next-generation VR/AR optics: a review of holographic near-eye displays from a human-centric perspective. Optica (2020).
[10]
Chenliang Chang, Wei Cui, Jongchan Park, and Liang Gao. 2019. Computational holographic Maxwellian near-eye display with an expanded eyebox. Scientific Reports 9, 1 (10 Dec 2019), 18749.
[11]
Brian Chao, Manu Gopakumar, Suyeon Choi, and Gordon Wetzstein. 2023. High-brightness holographic projection. Opt. Lett. 48, 15 (Aug. 2023), 4041–4044.
[12]
Suyeon Choi, Manu Gopakumar, Yifan Peng, Jonghyun Kim, Matthew O’Toole, and Gordon Wetzstein. 2022. Time-Multiplexed Neural Holography: A Flexible Framework for Holographic Near-Eye Displays with Fast Heavily-Quantized Spatial Light Modulators. In ACM SIGGRAPH 2022 Conference Proceedings (Vancouver, BC, Canada) (SIGGRAPH ’22). Association for Computing Machinery, New York, NY, USA, Article 32, 9 pages.
[13]
Craig T. Draper, Colton M. Bigler, Micah S. Mann, Kalluri Sarma, and Pierre-Alexandre Blanche. 2019. Holographic waveguide head-up display with 2-D pupil expansion and longitudinal image magnification. Appl. Opt. 58, 5 (Feb 2019), A251–A257.
[14]
Craig T. Draper and Pierre-Alexandre Blanche. 2021. Examining aberrations due to depth of field in holographic pupil replication waveguide systems. Appl. Opt. 60, 6 (Feb 2021), 1653–1659.
[15]
Craig T. Draper and Pierre-Alexandre Blanche. 2022. Holographic curved waveguide combiner for HUD/AR with 1-D pupil expansion. Opt. Express 30, 2 (Jan 2022), 2503–2516.
[16]
Joseph W Goodman. 2005. Introduction to Fourier optics. Introduction to Fourier optics, 3rd ed., by JW Goodman. Englewood, CO: Roberts & Co. Publishers, 2005 1 (2005).
[17]
Manu Gopakumar, Jonghyun Kim, Suyeon Choi, Yifan Peng, and Gordon Wetzstein. 2021. Unfiltered holography: optimizing high diffraction orders without optical filtering for compact holographic displays. Opt. Lett. 46, 23 (Dec 2021), 5822–5825.
[18]
Manu Gopakumar, Gun-Yeal Lee, Suyeon Choi, Brian Chao, Yifan Peng, Jonghyun Kim, and Gordon Wetzstein. 2024. Full-colour 3D holographic augmented-reality displays with metasurface waveguides. Nature (08 May 2024).
[19]
Changwon Jang, Kiseung Bang, Minseok Chae, Byoungho Lee, and Douglas Lanman. 2024. Waveguide holography for 3D augmented reality glasses. Nature Communications 15, 1 (02 Jan 2024), 66.
[20]
Changwon Jang, Kiseung Bang, Gang Li, and Byoungho Lee. 2018. Holographic Near-Eye Display with Expanded Eye-Box. ACM Trans. Graph. 37, 6, Article 195 (dec 2018), 14 pages.
[21]
Bahram Javidi, Artur Carnicer, Arun Anand, George Barbastathis, Wen Chen, Pietro Ferraro, J W Goodman, Ryoichi Horisaki, Kedar Khare, Malgorzata Kujawinska, Rainer A Leitgeb, Pierre Marquet, Takanori Nomura, Aydogan Ozcan, Yongkeun Park, Giancarlo Pedrini, Pascal Picart, Joseph Rosen, Genaro Saavedra, Natan T Shaked, Adrian Stern, Enrique Tajahuerce, Lei Tian, Gordon Wetzstein, and Masahiro Yamaguchi. 2021. Roadmap on digital holography [Invited]. Opt. Express, OE 29, 22 (Oct. 2021), 35078–35118.
[22]
Youngjin Jo, Dongheon Yoo, Dukho Lee, Minkwan Kim, and Byoungho Lee. 2022. Multi-illumination 3D holographic display using a binary mask. Opt. Lett. 47, 10 (May 2022), 2482–2485.
[23]
Koray Kavaklı, Liang Shi, Hakan Urey, Wojciech Matusik, and Kaan Akşit. 2023. Multi-color Holograms Improve Brightness in Holographic Displays. In SIGGRAPH Asia 2023 Conference Papers. Association for Computing Machinery, 1–11.
[24]
Jonghyun Kim, Manu Gopakumar, Suyeon Choi, Yifan Peng, Ward Lopes, and Gordon Wetzstein. 2022. Holographic Glasses for Virtual Reality. In ACM SIGGRAPH 2022 Conference Proceedings (Vancouver, BC, Canada) (SIGGRAPH ’22, Article 33). Association for Computing Machinery, New York, NY, USA, 1–9.
[25]
Bernard C Kress and Ishan Chatterjee. 2020. Waveguide combiners for mixed reality headsets: a nanophotonics design perspective. Nanophotonics 10, 1 (2020), 41–74.
[26]
Grace Kuo, Florian Schiffers, Douglas Lanman, Oliver Cossairt, and Nathan Matsuda. 2023. Multisource Holography. ACM Trans. Graph. 42, 6, Article 203 (dec 2023), 14 pages.
[27]
Grace Kuo, Laura Waller, Ren Ng, and Andrew Maimone. 2020. High Resolution Étendue Expansion for Holographic Displays. ACM Trans. Graph. 39, 4, Article 66 (aug 2020), 14 pages.
[28]
Grigory Lazarev, Stefanie Bonifer, Philip Engel, Daniel Höhne, and Gunther Notni. 2017. High-resolution LCOS microdisplay with sub-kHz frame rate for high performance, high precision 3D sensor. In Digital Optical Technologies 2017, Bernard C. Kress and Peter Schelkens (Eds.), Vol. 10335. International Society for Optics and Photonics, SPIE, 103351B.
[29]
Byounghyo Lee, Dongheon Yoo, Jinsoo Jeong, Seungjae Lee, Dukho Lee, and Byoungho Lee. 2020. Wide-angle speckleless DMD holographic display using structured illumination with temporal multiplexing. Opt. Lett. 45, 8 (Apr 2020), 2148–2151.
[30]
Dukho Lee, Kiseung Bang, Seung-Woo Nam, Byounghyo Lee, Dongyeon Kim, and Byoungho Lee. 2022. Expanding energy envelope in holographic display via mutually coherent multi-directional illumination. Scientific Reports 12, 1 (22 Apr 2022), 6649.
[31]
Wen-Kai Lin, Osamu Matoba, Bor-Shyh Lin, and Wei-Chia Su. 2018. Astigmatism and deformation correction for a holographic head-mounted display with a wedge-shaped holographic waveguide. Appl. Opt. 57, 25 (Sep 2018), 7094–7101.
[32]
Wen-Kai Lin, Osamu Matoba, Bor-Shyh Lin, and Wei-Chia Su. 2020. Astigmatism correction and quality optimization of computer-generated holograms for holographic waveguide displays. Opt. Express 28, 4 (Feb 2020), 5519–5527.
[33]
Andrew Maimone, Andreas Georgiou, and Joel S. Kollin. 2017. Holographic Near-Eye Displays for Virtual and Augmented Reality. ACM Trans. Graph. 36, 4, Article 85 (jul 2017), 16 pages.
[34]
Eric Markley, Nathan Matsuda, Florian Schiffers, Oliver Cossairt, and Grace Kuo. 2023. Simultaneous Color Computer Generated Holography. In SIGGRAPH Asia 2023 Conference Papers (<conf-loc>, <city>Sydney</city>, <state>NSW</state>, <country>Australia</country>, </conf-loc>) (SA ’23, Article 22). Association for Computing Machinery, New York, NY, USA, 1–11.
[35]
Sagi Monin, Aswin C. Sankaranarayanan, and Anat Levin. 2022a. Analyzing phase masks for wide étendue holographic displays. In 2022 IEEE International Conference on Computational Photography (ICCP). 1–12.
[36]
Sagi Monin, Aswin C. Sankaranarayanan, and Anat Levin. 2022b. Exponentially-wide etendue displays using a tilting cascade. In 2022 IEEE International Conference on Computational Photography (ICCP). 1–12.
[37]
N. Padmanaban, Y. Peng, and G. Wetzstein. 2019. Holographic Near-Eye Displays Based on Overlap-Add Stereograms. ACM Trans. Graph. (SIGGRAPH Asia)6 (2019). Issue 38.
[38]
Jongchan Park, KyeoReh Lee, and YongKeun Park. 2019. Ultrathin wide-angle large-area digital 3D holographic display using a non-periodic photon sieve. Nature Communications 10, 1 (21 Mar 2019), 1304.
[39]
Jae-Hyeung Park and Seong-Bok Kim. 2018. Optical see-through holographic near-eye-display with eyebox steering and depth of field control. Opt. Express 26, 21 (Oct. 2018), 27076–27088.
[40]
Y. Peng, S. Choi, N. Padmanaban, and G. Wetzstein. 2020. Neural Holography with Camera-in-the-loop Training. ACM Trans. Graph. (SIGGRAPH Asia) (2020).
[41]
Dapu Pi, Juan Liu, and Yongtian Wang. 2022. Review of computer-generated hologram algorithms for color dynamic holographic three-dimensional display. Light Sci Appl 11, 1 (July 2022), 231.
[42]
Stephan Reichelt, Ralf Haussler, Norbert Leister, Gerald Futterer, Hagen Stolle, and Armin Schwerdtner. 2010. Holographic 3-D Displays - Electro-holography within the Grasp of Commercialization. In Advances in Lasers and Electro Optics, Nelson Costa and Adolfo Cartaxo (Eds.). IntechOpen, Rijeka, Chapter 29.
[43]
Florian Schiffers, Praneeth Chakravarthula, Nathan Matsuda, Grace Kuo, Ethan Tseng, Douglas Lanman, Felix Heide, and Oliver Cossairt. 2023. Stochastic Light Field Holography. IEEE International Conference on Computational Photography (ICCP) (2023).
[44]
Liang Shi, Beichen Li, Changil Kim, Petr Kellnhofer, and Wojciech Matusik. 2021. Towards real-time photorealistic 3D holography with deep neural networks. Nature 591, 7849 (01 Mar 2021), 234–239.
[45]
Liang Shi, Beichen Li, and Wojciech Matusik. 2022. End-to-end learning of 3D phase-only holograms for holographic display. Light Sci Appl 11, 1 (Aug. 2022), 247.
[46]
Liang Shi, DongHun Ryu, and Wojciech Matusik. 2024. Ergonomic-Centric Holography: Optimizing Realism, Immersion, and Comfort for Holographic Display. Laser & Photonics Reviews 18, 4 (2024), 2300651.
[47]
A Starikov. 1982. Effective number of degrees of freedom of partially coherent sources. JOSA 72, 11 (1982), 1538–1544.
[48]
Jie Sun, Erman Timurdogan, Ami Yaacobi, Ehsan Shah Hosseini, and Michael R Watts. 2013. Large-scale nanophotonic phased array. Nature 493, 7431 (2013), 195–199.
[49]
Ethan Tseng, Grace Kuo, Seung-Hwan Baek, Nathan Matsuda, Andrew Maimone, Florian Schiffers, Praneeth Chakravarthula, Qiang Fu, Wolfgang Heidrich, Douglas Lanman, and Felix Heide. 2024. Neural étendue expander for ultra-wide-angle high-fidelity holographic display. Nature Communications 15, 1 (22 Apr 2024), 2907.
[50]
Zi Wang, Guoqiang Lv, Yujian Pang, Qibin Feng, Anting Wang, and Hai Ming. 2023. Lens array-based holographic 3D display with an expanded field of view and eyebox. Opt. Lett. 48, 21 (Nov. 2023), 5559–5562.
[51]
Xinxing Xia, Yunqing Guan, Andrei State, Praneeth Chakravarthula, Tat-Jen Cham, and Henry Fuchs. 2020. Towards Eyeglass-style Holographic Near-eye Displays with Statically. In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 312–319.
[52]
Xinxing Xia, Weisen Wang, Frank Guan, Furong Yang, Xinghua Shui, Huadong Zheng, Yingjie Yu, and Yifan Peng. 2023. Exploring angular-steering illumination-based eyebox expansion for holographic displays. Opt. Express 31, 19 (Sep 2023), 31563–31573. https://opg.optica.org/oe/abstract.cfm?URI=oe-31-19-31563
[53]
Daeho Yang, Wontaek Seo, Hyeonseung Yu, Sun Il Kim, Bongsu Shin, Chang-Kun Lee, Seokil Moon, Jungkwuen An, Jong-Young Hong, Geeyoung Sung, and Hong-Seok Lee. 2022. Diffraction-engineered holography: Beyond the depth representation limit of holographic displays. Nature Communications 13, 1 (12 Oct 2022), 6012.
[54]
Jiwoon Yeom, Yeseul Son, and Kwangsoon Choi. 2021. Crosstalk Reduction in Voxels for a See-Through Holographic Waveguide by Using Integral Imaging with Compensated Elemental Images. Photonics 8, 6 (2021).
[55]
Hyeonseung Yu, KyeoReh Lee, Jongchan Park, and YongKeun Park. 2017. Ultrahigh-definition dynamic 3D holographic display by active control of volume speckle fields. Nature Photonics 11, 3 (01 Mar 2017), 186–192.
[56]
Zhengyun Zhang and Marc Levoy. 2009. Wigner distributions and how they relate to the light field. In 2009 IEEE International Conference on Computational Photography (ICCP). IEEE, 1–10.
Index Terms
- Large Étendue 3D Holographic Display with Content-adaptive Dynamic Fourier Modulation
Recommendations
Holographic Parallax Improves 3D Perceptual Realism
Holographic near-eye displays are a promising technology to solve long-standing challenges in virtual and augmented reality display systems. Over the last few years, many different computer-generated holography (CGH) algorithms have been proposed that ...
Neural 3D holography: learning accurate wave propagation models for 3D holographic virtual and augmented reality displays
Holographic near-eye displays promise unprecedented capabilities for virtual and augmented reality (VR/AR) systems. The image quality achieved by current holographic displays, however, is limited by the wave propagation models used to simulate the ...
Time-multiplexed Neural Holography: A Flexible Framework for Holographic Near-eye Displays with Fast Heavily-quantized Spatial Light Modulators
SIGGRAPH '22: ACM SIGGRAPH 2022 Conference ProceedingsHolographic near-eye displays offer unprecedented capabilities for virtual and augmented reality systems, including perceptually important focus cues. Although artificial intelligence–driven algorithms for computer-generated holography (CGH) have ...
Comments
Information & Contributors
Information
Published In
December 2024
1620 pages
Copyright © 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 03 December 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
SA '24
Sponsor:
Acceptance Rates
Overall Acceptance Rate 178 of 869 submissions, 20%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 359Total Downloads
- Downloads (Last 12 months)359
- Downloads (Last 6 weeks)105
Reflects downloads up to 31 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format