research-article

The light field stereoscope: immersive computer graphics via factored near-eye light field displays with focus cues

Authors:

Fu-Chung Huang,

Gordon WetzsteinAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 34, Issue 4

Article No.: 60, Pages 1 - 12

https://doi.org/10.1145/2766922

Published: 27 July 2015 Publication History

Abstract

Over the last few years, virtual reality (VR) has re-emerged as a technology that is now feasible at low cost via inexpensive cellphone components. In particular, advances of high-resolution micro displays, low-latency orientation trackers, and modern GPUs facilitate immersive experiences at low cost. One of the remaining challenges to further improve visual comfort in VR experiences is the vergence-accommodation conflict inherent to all stereoscopic displays. Accurate reproduction of all depth cues is crucial for visual comfort. By combining well-known stereoscopic display principles with emerging factored light field technology, we present the first wearable VR display supporting high image resolution as well as focus cues. A light field is presented to each eye, which provides more natural viewing experiences than conventional near-eye displays. Since the eye box is just slightly larger than the pupil size, rank-1 light field factorizations are sufficient to produce correct or nearly-correct focus cues; no time-multiplexed image display or gaze tracking is required. We analyze lens distortions in 4D light field space and correct them using the afforded high-dimensional image formation. We also demonstrate significant improvements in resolution and retinal blur quality over related near-eye displays. Finally, we analyze diffraction limits of these types of displays.

Supplementary Material

ZIP File (a60-huang.zip)

Supplemental files

Download
335.93 MB

MP4 File (a60.mp4)

Download
16.37 MB

References

[1]

Akeley, K., Watt, S. J., Girshick, A. R., and Banks, M. S. 2004. A stereo display prototype with multiple focal distances. ACM Trans. Graph. (SIGGRAPH) 23, 804--813.

Digital Library

[2]

Benton, S., and Bove, V. 2006. Holographic Imaging. John Wiley and Sons.

Digital Library

[3]

Brown, D. 2000. Decentering distortion of lenses. Photogrammetric Engineering 32, 3, 444462.

[4]

Cakmakci, O., and Rolland, J. 2006. Head-worn displays: a review. Journal of Display Technology 2, 3, 199--216.

[5]

Clarberg, P., Toth, R., Hasselgren, J., Nilsson, J., and Akenine-Möller, T. 2014. Amfs: Adaptive multi-frequency shading for future graphics processors. ACM Trans. Graph. 33, 4 (July), 141:1--141:12.

Digital Library

[6]

Cutting, J. E., and Vishton, P. M. 1995. Perceiving layout and knowing distances: The interaction, relative potency, and contextual use of different information about depth. W. Epstein and S. Rogers (Eds.), Perception of space and motion, 69--117.

[7]

Favalora, G. E. 2005. Volumetric 3D displays and application infrastructure. IEEE Computer 38, 37--44.

Digital Library

[8]

Hale, K., and Stanney, K. 2014. Handbook of Virtual Environments. CRC Press.

Digital Library

[9]

Heide, F., Lanman, D., Reddy, D., Kautz, J., Pulli, K., and Luebke, D. 2014. Cascaded displays: Spatiotemporal superresolution using offset pixel layers. ACM Trans. Graph. (SIGGRAPH) 33, 4, 60:1--60:11.

Digital Library

[10]

Held, R., Cooper, E., and Banks, M. 2012. Blur and Disparity Are Complementary Cues to Depth. Current Biology 22, R163--R165.

[11]

Hirsch, M., Wetzstein, G., and Raskar, R. 2014. A Compressive Light Field Projection System. ACM Trans. Graph. (Proc. SIGGRAPH) 33, 4, 1--12.

Digital Library

[12]

Hoffman, D. M., and Banks, M. S. 2010. Focus information is used to interpret binocular images. Journal of Vision 10, 5, 13.

[13]

Hua, H., and Javidi, B. 2014. A 3d integral imaging optical see-through head-mounted display. OSA Opt. Exp. 22, 11, 13484--13491.

[14]

Huang, F.-C., Wetzstein, G., Barsky, B. A., and Raskar, R. 2014. Eyeglasses-free display: Towards correcting visual aberrations with computational light field displays. ACM Trans. Graph. (SIGGRAPH) 33, 4, 59:1--59:12.

Digital Library

[15]

Jones, A., McDowall, I., Yamada, H., Bolas, M., and Debevec, P. 2007. Rendering for an interactive 360° light field display. ACM Trans. Graph. (SIGGRAPH) 26, 40:1--40:10.

Digital Library

[16]

Kelly, D. H. 1979. Motion and vision. ii. stabilized spatiotemporal threshold surface. Journal of the Optical Society of America 69, 1340--1349.

[17]

Kress, B., and Starner, T. 2013. A review of head-mounted displays (hmd) technologies and applications for consumer electronics. In Proc. SPIE, vol. 8720, 87200A--87200A--13.

[18]

Lanman, D., and Luebke, D. 2013. Near-eye light field displays. ACM Trans. Graph. (SIGGRAPH Asia) 32, 6, 220:1--220:10.

Digital Library

[19]

Lanman, D., Hirsch, M., Kim, Y., and Raskar, R. 2010. Content-adaptive parallax barriers: Optimizing dual-layer 3D displays using low-rank light field factorization. ACM Trans. Graph. (SIGGRAPH Asia) 29, 163:1--163:10.

Digital Library

[20]

Levoy, M., and Hanrahan, P. 1996. Light Field Rendering. In ACM SGGRAPH, 31--42.

Digital Library

[21]

Liu, S., Cheng, D., and Hua, H. 2008. An optical see-through head mounted display with addressable focal planes. In Proc. ISMAR, 33--42.

Digital Library

[22]

Love, G. D., Hoffman, D. M., Hands, P. J., Gao, J., Kirby, A. K., and Banks, M. S. 2009. High-speed switchable lens enables the development of a volumetric stereoscopic display. OSA Optics Express 17, 18, 15716--15725.

[23]

MacKenzie, K. J., Hoffman, D. M., and Watt, S. J. 2010. Accommodation to multiplefocalplane displays: Implications for improving stereoscopic displays and for accommodation control. Journal of Vision 10, 8.

[24]

Maimone, A., and Fuchs, H. 2013. Computational augmented reality eyeglasses. In Proc. ISMAR, 29--38.

[25]

Maimone, A., Wetzstein, G., Hirsch, M., Lanman, D., Raskar, R., and Fuchs, H. 2013. Focus 3d: Compressive accommodation display. ACM Trans. Graph. 32, 5, 153:1--153:13.

Digital Library

[26]

Maimone, A., Lanman, D., Rathinavel, K., Keller, K., Luebke, D., and Fuchs, H. 2014. Pinlight displays: Wide field of view augmented reality eyeglasses using defocused point light sources. ACM Trans. Graph. (SIGGRAPH) 33, 4.

Digital Library

[27]

Marshall, J. A., Marshall, J. A., Ariely, D., Burbeck, C. A., Aricly, T. D., Rolland, J. P., and Martin, K. E. 1996. Occlusion edge blur: A cue to relative visual depth. OSA JOSAA 13, 681--688.

[28]

Marwah, K., Wetzstein, G., Bando, Y., and Raskar, R. 2013. Compressive Light Field Photography using Overcomplete Dictionaries and Optimized Projections. ACM Trans. Graph. (Proc. SIGGRAPH).

Digital Library

[29]

Rothbaum, B., Hodges, L., Ready, D., Graap, K., and Alarcon, R. 2001. Virtual reality exposure therapy for vietnam veterans with posttraumatic stress disorder. Ann Surg 62, 8, 617--22.

[30]

Ruch, T., and Fulton, J. F. 1960. Medical physiology and biophysics. W. B. Saunders Company.

[31]

Rushton, S. K., and Riddell, P. M. 1999. Developing visual systems and exposure to virtual reality and stereo displays: some concerns and speculations about the demands on accommodation and vergence. Appl Ergonomics 30, 69--78.

[32]

Ryana, L., MacKenziea, K., and Watta, S. 2012. Multiple-focal-planes 3d displays: A practical solution to the vergence-accommodation conflict? In Proc. 3D Imaging (IC3D), 1--6.

[33]

Schowengerdt, B. T., and Seibel, E. J. 2006. True 3-d scanned voxel displays using single or multiple light sources. Journal of the Society for Information Display 14, 2, 135--143.

[34]

Seetzen, H., Heidrich, W., Stuerzlinger, W., Ward, G., Whitehead, L., Trentacoste, M., Ghosh, A., and Vorozcovs, A. 2004. High dynamic range display systems. ACM Trans. Graph. (SIGGRAPH) 23, 3, 760--768.

Digital Library

[35]

Sullivan, A. 2003. A solid-state multi-planar volumetric display. In SID Digest, vol. 32, 207--211.

[36]

Sutherland, I. 1968. A head-mounted three dimensional display. In Proc. AFIPS Fall Joint Computer Conference.

Digital Library

[37]

Takaki, Y., Tanaka, K., and Nakamura, J. 2011. Super multi-view display with a lower resolution flat-panel display. Opt. Express 19, 5, 4129--4139.

[38]

Takaki, Y. 2006. High-Density Directional Display for Generating Natural Three-Dimensional Images. Proc. IEEE 94, 3.

[39]

Watt, S., Akeley, K., Ernst, M., and Banks, M. 2005. Focus cues affect perceived depth. Journal of Vision 5, 10, 834--862.

[40]

Wetzstein, G., Lanman, D., Heidrich, W., and Raskar, R. 2011. Layered 3D: Tomographic image synthesis for attenuation-based light field and high dynamic range displays. ACM Trans. Graph. (SIGGRAPH) 30, 1--11.

Digital Library

[41]

Wetzstein, G., Lanman, D., Hirsch, M., and Raskar, R. 2012. Tensor Displays: Compressive Light Field Synthesis using Multilayer Displays with Directional Backlighting. ACM Trans. Graph. (SIGGRAPH) 31, 1--11.

Digital Library

[42]

Wheatstone, C. 1838. Contributions to the physiology of vision. part the first. on some remarkable, and hitherto unobserved, phenomena of binocular vision. Philosophical Transactions of the Royal Society of London 128, 371--394.

[43]

Zhang, Z. 2000. A flexible new technique for camera calibration. IEEE Trans. PAMI 22, 11, 1330--1334.

Digital Library

[44]

Zwicker, M., Matusik, W., Durand, F., and Pfister, H. 2006. Antialiasing for automultiscopic 3D displays. In EGSR.

Digital Library

Cited By

Maeda YWatanabe HKoide DSasaki HHanda THisatomi KArai J(2024)Wide field-of-view light-field head-mounted display for virtual reality applicationsOptics Continuum10.1364/OPTCON.5095623:4(574)Online publication date: 25-Mar-2024
https://doi.org/10.1364/OPTCON.509562
Li PZhao JWu JDeng CHan YWang HYu T(2024)OPAL: Occlusion Pattern Aware Loss for Unsupervised Light Field Disparity EstimationIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.329660046:2(681-694)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1109/TPAMI.2023.3296600
Jo YYoo DJoeng YLee B(2023)Three-dimensional projection from multi-layered light fieldUltra-High-Definition Imaging Systems VI10.1117/12.2649569(23)Online publication date: 14-Mar-2023
https://doi.org/10.1117/12.2649569
Show More Cited By

Index Terms

The light field stereoscope: immersive computer graphics via factored near-eye light field displays with focus cues
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Virtual reality

Recommendations

The making of welcome to light fields VR
SIGGRAPH '18: ACM SIGGRAPH 2018 Talks

Light fields can provide transportive immersive experiences with a level of realism unsurpassed by any other imaging technology. Within a limited viewing volume, light fields accurately reproduce stereo parallax, motion parallax, reflections, ...
DeepFocus: learned image synthesis for computational displays

Addressing vergence-accommodation conflict in head-mounted displays (HMDs) requires resolving two interrelated problems. First, the hardware must support viewing sharp imagery over the full accommodation range of the user. Second, HMDs should accurately ...
A system for acquiring, processing, and rendering panoramic light field stills for virtual reality

We present a system for acquiring, processing, and rendering panoramic light field still photography for display in Virtual Reality (VR). We acquire spherical light field datasets with two novel light field camera rigs designed for portable and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 34, Issue 4

August 2015

1307 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2809654

Issue’s Table of Contents

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 July 2015

Published in TOG Volume 34, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

206
Total Citations
View Citations
3,367
Total Downloads

Downloads (Last 12 months)89
Downloads (Last 6 weeks)6

Reflects downloads up to 26 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Maeda YWatanabe HKoide DSasaki HHanda THisatomi KArai J(2024)Wide field-of-view light-field head-mounted display for virtual reality applicationsOptics Continuum10.1364/OPTCON.5095623:4(574)Online publication date: 25-Mar-2024
https://doi.org/10.1364/OPTCON.509562
Li PZhao JWu JDeng CHan YWang HYu T(2024)OPAL: Occlusion Pattern Aware Loss for Unsupervised Light Field Disparity EstimationIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.329660046:2(681-694)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1109/TPAMI.2023.3296600
Jo YYoo DJoeng YLee B(2023)Three-dimensional projection from multi-layered light fieldUltra-High-Definition Imaging Systems VI10.1117/12.2649569(23)Online publication date: 14-Mar-2023
https://doi.org/10.1117/12.2649569
Ebner CMohr PLanglotz TPeng YSchmalstieg DWetzstein GKalkofen D(2023)Off-Axis Layered Displays: Hybrid Direct-View/Near-Eye Mixed Reality with Focus CuesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324707729:5(2816-2825)Online publication date: May-2023
https://doi.org/10.1109/TVCG.2023.3247077
Jin JGuo MHou JLiu HXiong H(2023)Light Field Reconstruction Via Deep Adaptive Fusion of Hybrid LensesIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.3287603(1-18)Online publication date: 2023
https://doi.org/10.1109/TPAMI.2023.3287603
Guo MHou JJin JLiu HZeng HLu J(2023)Content-Aware Warping for View SynthesisIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.324270945:8(9486-9503)Online publication date: Aug-2023
https://doi.org/10.1109/TPAMI.2023.3242709
Liu GYue HWu JYang J(2023)Intra-Inter View Interaction Network for Light Field Image Super-ResolutionIEEE Transactions on Multimedia10.1109/TMM.2021.312438525(256-266)Online publication date: 2023
https://doi.org/10.1109/TMM.2021.3124385
Sheng HWang SYang DCong RCui ZChen R(2023)Cross-View Recurrence-Based Self-Supervised Super-Resolution of Light FieldIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2023.327846233:12(7252-7266)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1109/TCSVT.2023.3278462
Chi TPerng WChen H(2023)Perceptual Tolerance of Split-Up Effect for Near-Eye Light Field Display2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR59233.2023.00032(177-184)Online publication date: 16-Oct-2023
https://doi.org/10.1109/ISMAR59233.2023.00032
Morimoto THirota MSasaki KKato KTakigawa RYoneyama SHayashi TFujikado TTokuhisa S(2023)Light-Field Visual System for the Remote Robot Operation Interface2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)10.1109/IROS55552.2023.10342529(7020-7025)Online publication date: 1-Oct-2023
https://doi.org/10.1109/IROS55552.2023.10342529
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents