research-article

Open access

Towards occlusion-aware multifocal displays

Authors:

Jen-Hao Rick Chang,

B. V. K. Vijaya Kumar,

Aswin C. SankaranarayananAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 39, Issue 4

Article No.: 68, Pages 68:1 - 68:15

https://doi.org/10.1145/3386569.3392424

Published: 12 August 2020 Publication History

Abstract

The human visual system uses numerous cues for depth perception, including disparity, accommodation, motion parallax and occlusion. It is incumbent upon virtual-reality displays to satisfy these cues to provide an immersive user experience. Multifocal displays, one of the classic approaches to satisfy the accommodation cue, place virtual content at multiple focal planes, each at a different depth. However, the content on focal planes close to the eye do not occlude those farther away; this deteriorates the occlusion cue as well as reduces contrast at depth discontinuities due to leakage of the defocus blur. This paper enables occlusion-aware multifocal displays using a novel ConeTilt operator that provides an additional degree of freedom --- tilting the light cone emitted at each pixel of the display panel. We show that, for scenes with relatively simple occlusion configurations, tilting the light cones provides the same effect as physical occlusion. We demonstrate that ConeTilt can be easily implemented by a phase-only spatial light modulator. Using a lab prototype, we show results that demonstrate the presence of occlusion cues and the increased contrast of the display at depth edges.

Supplemental Material

MP4 File

Presentation video

Transcript for: Presentation video

MP4 File

Download
402.03 MB

ZIP File

Supplemental files.

Download
2.83 MB

References

[1]

Kurt Akeley. 2004. Achieving Near-correct Focus Cues Using Multiple Image Planes. Ph.D. Dissertation. Stanford University.

[2]

Kurt Akeley, Simon J. Watt, Ahna Reza Girshick, and Martin S. Banks. 2004. A Stereo Display Prototype with Multiple Focal Distances. ACM Transactions on Graphics (TOG) 23, 3 (2004), 804--813.

Digital Library

[3]

Stefan Bernet and Monika Ritsch-Marte. 2008. Adjustable Refractive Power From Diffractive Moiré Elements. Applied optics 47, 21 (2008), 3722--3730.

[4]

Jen-Hao Rick Chang, B. V. K. Vijaya Kumar, and Aswin C. Sankaranarayanan. 2018. Towards Multifocal Displays with Dense Focal Stacks. ACM Transactions on Graphics (TOG) 37, 6 (2018), 198:1--198:13.

[5]

Suyeon Choi, Seungjae Lee, Youngjin Jo, Dongheon Yoo, Dongyeon Kim, and Byoungho Lee. 2019. Optimal Binary Representation via Non-convex Optimization on Tomographic Displays. Optics Express 27, 17 (2019), 24362--24381.

[6]

Oliver S. Cossairt, Joshua Napoli, Samuel L. Hill, Rick K. Dorval, and Gregg E. Favalora. 2007. Occlusion-capable Multiview Volumetric Three-dimensional Display. Applied Optics 46, 8 (2007), 1244--1250.

[7]

James E Cutting and Peter M Vishton. 1995. Perceiving Layout and Knowing Distances: The Integration, Relative Potency, and Contextual Use of Different Information About Depth. In Perception of space and motion. 69--117.

[8]

Gerwin Damberg, James Gregson, and Wolfgang Heidrich. 2016. High Brightness HDR Projection Using Dynamic Freeform Lensing. ACM Transactions on Graphics (TOG) 35, 3 (2016), 24:1--24:11.

Digital Library

[9]

Jason Geng. 2013. Three-dimensional Display Technologies. Advances in Optics and Photonics 5, 4 (2013), 456--535.

[10]

Fu-Chung Huang, Kevin Chen, and Gordon Wetzstein. 2015. The Light Field Stereoscope: Immersive Computer Graphics via Factored Near-eye Light Field Displays with Focus Cues. ACM Transactions on Graphics (TOG) 34, 4 (2015), 60:1--60:12.

Digital Library

[11]

Fu-Chung Huang, Gordon Wetzstein, Brian A. Barsky, and Ramesh Raskar. 2014. Eyeglasses-free Display: Towards Correcting Visual Aberrations with Computational Light Field Displays. ACM Transactions on Graphics (TOG) 33, 4 (2014), 59:1--59:12.

Digital Library

[12]

Masahiko Inami, Naoki Kawakami, Dairoku Sekiguchi, Yasuyuki Yanagida, Taro Maeda, and Susumu Tachi. 2000. Visuo-haptic Display Using Head-mounted Projector. In Proceedings of the IEEE Virtual Reality. 233--240.

[13]

Youngjin Jo, Seungjae Lee, Dongheon Yoo, Suyeon Choi, Dongyeon Kim, and Byoungho Lee. 2019. Tomographic Projector: Large Scale Volumetric Display with Uniform Viewing Experiences. ACM Transactions on Graphics (TOG) 38, 6 (2019), 215:1--215:13.

Digital Library

[14]

Paul V Johnson, Jared AQ Parnell, Joohwan Kim, Christopher D Saunter, Gordon D Love, and Martin S Banks. 2016. Dynamic Lens and Monovision 3D Displays to Improve Viewer Comfort. Optics express 24, 11 (2016), 11808--11827.

[15]

Andrew Jones, Ian McDowall, Hideshi Yamada, Mark Bolas, and Paul Debevec. 2007. Rendering for an Interactive 360° Light Field Display. ACM Transactions on Graphics (TOG) 26, 3 (2007).

Digital Library

[16]

Kiyoshi Kiyokawa, Yoshinori Kurata, and Hiroyuki Ohno. 2000. An Optical See-through Display for Mutual Occlusion of Real and Virtual Environments. In Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000). 60--67.

[17]

Robert Konrad, Emily A Cooper, and Gordon Wetzstein. 2016. Novel Optical Configurations for Virtual Reality: Evaluating User Preference and Performance with Focus-tunable and Monovision Near-eye Displays. In Conference on Human Factors in Computing Systems (CHI). 1211--1220.

[18]

G. A. Koulieris, K. Akşit, M. Stengel, R. K. Mantiuk, K. Mania, and C. Richardt. 2019. Near-Eye Display and Tracking Technologies for Virtual and Augmented Reality. Computer Graphics Forum 38, 2 (2019), 493--519.

[19]

George-Alex Koulieris, Bee Bui, Martin S. Banks, and George Drettakis. 2017. Accommodation and Comfort in Head-mounted Displays. ACM Transactions on Graphics (TOG) 36, 4 (2017), 87:1--87:11.

Digital Library

[20]

Douglas Lanman and David Luebke. 2013. Near-eye Light Field Displays. ACM Transactions on Graphics (TOG) 32, 6, Article 220 (Nov. 2013), 10 pages.

Digital Library

[21]

Vincent Laude. 1998. Twisted-nematic Liquid-crystal Pixelated Active Lens. Optics Communications 153, 1-3 (1998), 134--152.

[22]

Seungjae Lee, Youngjin Jo, Dongheon Yoo, Jaebum Cho, Dukho Lee, and Byoungho Lee. 2019. Tomographic Near-eye Displays. Nature communications 10, 1 (2019), 2497.

[23]

Anat Levin, Haggai Maron, and Michal Yarom. 2016. Passive light and viewpoint sensitive display of 3D content. In IEEE Conference on Computational Photography (ICCP). 1--15.

[24]

Sheng Liu, Dewen Cheng, and Hong Hua. 2008. An Optical See-through Head Mounted Display with Addressable Focal Planes. In IEEE/ACM International Symposium on Mixed and Augmented Reality. 33--42.

[25]

Sheng Liu and Hong Hua. 2009. Time-multiplexed Dual-focal Plane Head-mounted Display with a Liquid Lens. Optics letters 34, 11 (2009), 1642--1644.

[26]

Patrick Llull, Noah Bedard, Wanmin Wu, Ivana Tosic, Kathrin Berkner, and Nikhil Balram. 2015. Design and Optimization of a Near-eye Multifocal Display System for Augmented Reality. In Imaging and Applied Optics. JTH3A.5.

[27]

Gordon D Love, David M Hoffman, Philip JW Hands, James Gao, Andrew K Kirby, and Martin S Banks. 2009. High-speed Switchable Lens Enables the Development of a Volumetric Stereoscopic Display. Optics express 17, 18 (2009), 15716--15725.

[28]

Kevin J MacKenzie, David M Hoffman, and Simon J Watt. 2010. Accommodation to Multiple-focal-plane Displays: Implications for Improving Stereoscopic Displays and for Accommodation Control. Journal of vision 10, 8 (2010), 22--22.

[29]

Andrew Maimone, Andreas Georgiou, and Joel S. Kollin. 2017. Holographic Near-eye Displays for Virtual and Augmented Reality. ACM Transactions on Graphics (TOG) 36, 4 (2017), 85:1--85:16.

Digital Library

[30]

Nathan Matsuda, Alexander Fix, and Douglas Lanman. 2017. Focal Surface Displays. ACM Transactions on Graphics (TOG) 36, 4 (July 2017), 86:1--86:14.

Digital Library

[31]

Olivier Mercier, Yusufu Sulai, Kevin Mackenzie, Marina Zannoli, James Hillis, Derek Nowrouzezahrai, and Douglas Lanman. 2017. Fast Gaze-contingent Optimal Decompositions for Multifocal Displays. ACM Transactions on Graphics (TOG) 36, 6 (2017), 237:1--237:15.

Digital Library

[32]

Jurriaan D Mulder. 2005. Realistic Occlusion Effects in Mirror-based Co-located Augmented Reality Systems. In IEEE Proceedings on Virtual Reality. 203--208.

[33]

Rahul Narain, Rachel A. Albert, Abdullah Bulbul, Gregory J. Ward, Martin S. Banks, and James F. O'Brien. 2015. Optimal Presentation of Imagery with Focus Cues on Multi-plane Displays. ACM Transactions on Graphics (TOG) 34, 4(2015), 59:1--59:12.

Digital Library

[34]

Nitish Padmanaban, Robert Konrad, Tal Stramer, Emily A Cooper, and Gordon Wetzstein. 2017. Optimizing Virtual Reality for All Users Through Gaze-contingent and Adaptive Focus Displays. Proceedings of the National Academy of Sciences 114 (2017), 9.

[35]

Kishore Rathinavel, Hanpeng Wang, Alex Blate, and Henry Fuchs. 2018. An Extended Depth-at-field Volumetric Near-eye Augmented Reality Display. IEEE Transactions on Visualization and Computer Graphics 24, 11 (2018), 2857--2866.

[36]

Kishore Rathinavel, Gordon Wetzstein, and Henry Fuchs. 2019. Varifocal Occlusion-Capable Optical See-through Augmented Reality Display based on Focus-tunable Optics. IEEE Transactions on Visualization and Computer Graphics 25, 11 (2019), 3125--3134.

[37]

Jannick P. Rolland, Myron W. Krueger, and Alexei A. Goon. 1999. Dynamic Focusing in Head-mounted Displays. Proceeding of SPIE 3639 (1999), 3639 - 3639 - 8.

[38]

Neil R Smith, Don C Abeysinghe, Joseph W Haus, and Jason Heikenfeld. 2006. Agile Wide-angle Beam Steering with Electrowetting Microprisms. Optics Express 14, 14 (2006), 6557--6563.

[39]

Nelson V Tabiryan, Svetlana V Serak, David E Roberts, Diane M Steeves, and Brian R Kimball. 2015. Thin Waveplate Lenses of Switchable Focal Length-new Generation in Optics. Optics express 23, 20 (2015), 25783--25794.

[40]

Simon J Watt, Kevin J MacKenzie, and Louise Ryan. 2012. Real-world Stereoscopic Performance in Multiple-focal-plane Displays: How Far Apart Should the Image Planes Be?. In Stereoscopic Displays and Applications XXIII, Vol. 8288. 82881E.

[41]

Gordon Wetzstein, Douglas Lanman, Wolfgang Heidrich, and Ramesh Raskar. 2011. Layered 3D: Tomographic Image Synthesis for Attenuation-based Light Field and High Dynamic Range Displays. ACM Transactions on Graphics (TOG) 30, 4 (2011), 95:1--95:12.

Digital Library

[42]

Lei Xiao, Anton Kaplanyan, Alexander Fix, Matthew Chapman, and Douglas Lanman. 2018. DeepFocus: Learned Image Synthesis for Computational Displays. ACM Transactions on Graphics (TOG) 37, 6 (2018), 200:1--200:13.

Digital Library

[43]

Marina Zannoli, Gordon D Love, Rahul Narain, and Martin S Banks. 2016. Blur and the Perception of Depth at Occlusions. Journal of Vision 16, 6 (2016), 17--17.

Cited By

Ebner CPlopski ASchmalstieg DKalkofen D(2024)Gaze-Contingent Layered Optical See-Through Displays with a Confidence-Driven View VolumeIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345620430:11(7203-7213)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3456204
Xia TLou YHu JWu F(2022)Simplified retinal 3D projection rendering method and systemApplied Optics10.1364/AO.45148261:9(2382)Online publication date: 16-Mar-2022
https://doi.org/10.1364/AO.451482
Batmaz ABarrera Machuca MSun JStuerzlinger W(2022)The Effect of the Vergence-Accommodation Conflict on Virtual Hand Pointing in Immersive DisplaysProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3502067(1-15)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3502067
Show More Cited By

Index Terms

Towards occlusion-aware multifocal displays
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Virtual reality

Recommendations

Towards multifocal displays with dense focal stacks

We present a virtual reality display that is capable of generating a dense collection of depth/focal planes. This is achieved by driving a focus-tunable lens to sweep a range of focal lengths at a high frequency and, subsequently, tracking the focal ...
Focal surface displays

Conventional binocular head-mounted displays (HMDs) vary the stimulus to vergence with the information in the picture, while the stimulus to accommodation remains fixed at the apparent distance of the display, as created by the viewing optics. Sustained ...
Occlusion Detectable Stereo -- Occlusion Patterns in Camera Matrix
CVPR '96: Proceedings of the 1996 Conference on Computer Vision and Pattern Recognition (CVPR '96)

In stereo algorithms with more than two cameras, the improvement of accuracy is often reported since they are robust against noise. However, another important aspect of the polynocular stereo, that is the ability of occlusion detection, has been paid ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 39, Issue 4

August 2020

1732 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3386569

Editor:
Szymon Rusinkiewicz
Princeton University

Issue’s Table of Contents

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 August 2020

Published in TOG Volume 39, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSF Career
ERC

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
529
Total Downloads

Downloads (Last 12 months)116
Downloads (Last 6 weeks)22

Reflects downloads up to 06 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ebner CPlopski ASchmalstieg DKalkofen D(2024)Gaze-Contingent Layered Optical See-Through Displays with a Confidence-Driven View VolumeIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345620430:11(7203-7213)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3456204
Xia TLou YHu JWu F(2022)Simplified retinal 3D projection rendering method and systemApplied Optics10.1364/AO.45148261:9(2382)Online publication date: 16-Mar-2022
https://doi.org/10.1364/AO.451482
Batmaz ABarrera Machuca MSun JStuerzlinger W(2022)The Effect of the Vergence-Accommodation Conflict on Virtual Hand Pointing in Immersive DisplaysProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3502067(1-15)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3502067
Monin SSankaranarayanan ALevin A(2022)Exponentially-wide etendue displays using a tilting cascade2022 IEEE International Conference on Computational Photography (ICCP)10.1109/ICCP54855.2022.9887737(1-12)Online publication date: 1-Aug-2022
https://doi.org/10.1109/ICCP54855.2022.9887737
Kimura SIwai DPunpongsanon PSato K(2021)Multifocal Stereoscopic Projection MappingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2021.310648627:11(4256-4266)Online publication date: 1-Nov-2021
https://dl.acm.org/doi/10.1109/TVCG.2021.3106486

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Figures

Tables

Media

View Issue’s Table of Contents