research-article

Open access

VR social copresence with light field displays

Authors:

Nathan Matsuda,

Brian Wheelwright,

Douglas LanmanAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 40, Issue 6

Article No.: 244, Pages 1 - 13

https://doi.org/10.1145/3478513.3480481

Published: 10 December 2021 Publication History

Abstract

As virtual reality (VR) devices become increasingly commonplace, asymmetric interactions between people with and without headsets are becoming more frequent. Existing video pass-through VR headsets solve one side of these asymmetric interactions by showing the user a live reconstruction of the outside world. This paper further advocates for reverse pass-through VR, wherein a three-dimensional view of the user's face and eyes is presented to any number of outside viewers in a perspective-correct manner using a light field display. Tying together research in social telepresence and copresence, autostereoscopic displays, and facial capture, reverse pass-through VR enables natural eye contact and other important non-verbal cues in a wider range of interaction scenarios, providing a path to potentially increase the utility and social acceptability of VR headsets in shared and public spaces.

Supplementary Material

MP4 File (a244-matsuda.mp4)

Download
48.45 MB

References

[1]

Samer Al Moubayed, Jonas Beskow, Gabriel Skantze, and Björn Granström. 2012. Furhat: A Back-projected Human-like Robot Head for Multiparty Human-machine Interaction. Springer.

[2]

Oleg Alexander, Mike Rogers, William Lambeth, Matt Chiang, and Paul Debevec. 2009. The Digital Emily Project: Photoreal Facial Modeling and Animation.

[3]

Stuart M. Anstis, John W. Mayhew, and Tania Morley. 1969. The Perception of Where a Face or Television 'Portrait' is Looking. American Journal of Psychology 82, 4 (1969).

[4]

Amit H. Bermano, Markus Billeter, Daisuke Iwai, and Anselm Grundhöfer. 2017. Makeup Lamps: Live Augmentation of Human Faces via Projection. Computer Graphics Forum 36, 2.

Digital Library

[5]

Blender Foundation. 2021. Blender - A 3D Modelling and Rendering Package. Blender Foundation. http://www.blender.org

[6]

Liwei Chan and Kouta Minamizawa. 2017. FrontFace: Facilitating Communication between HMD Users and Outsiders Using Front-Facing-Screen HMDs. ACM Human-Computer Interaction with Mobile Devices and Services.

Digital Library

[7]

Milton Chen. 2002. Leveraging the Asymmetric Sensitivity of Eye Contact for Video-conference. ACM Human Factors in Computing Systems.

Digital Library

[8]

Sebastian Zambal Christoph Heindl, Lukas Brunner and Josef Scharinger. 2020. Blend-Torch: A Real-Time, Adaptive Domain Randomization Library. International Conference on Pattern Recognition Workshops.

[9]

Christian Frueh, Avneesh Sud, and Vivek Kwatra. 2017. Headset Removal for Virtual and Mixed Reality. ACM SIGGRAPH Talks (2017).

Digital Library

[10]

Taichi Furukawa, Daisuke Yamamoto, Moe Sugawa, Roshan Peiris, and Kouta Minamizawa. 2019. TeleSight: Enabling Asymmetric Collaboration in VR between HMD User and Non-HMD Users.

Digital Library

[11]

Ying Geng, Jacques Gollier, Brian Wheelwright, Fenglin Peng, Yusufu Sulai, Brant Lewis, Ning Chan, Wai Sze Tiffany Lam, Alexander Fix, Douglas Lanman, et al. 2018. Viewing Optics for Immersive Near-eye Displays: Pupil Swim/Size and Weight/Stray Light. Digital Optics for Immersive Displays.

[12]

Daniel Gotsch, Xujing Zhang, Timothy Merritt, and Roel Vertegaal. 2018. TeleHuman2: A Cylindrical Light Field Teleconferencing System for Life-size 3D Human Telepresence. ACM Conference on Human Factors in Computing Systems 18.

Digital Library

[13]

Jan Gugenheimer, Christian Mai, Mark McGill, Julie Williamson, Frank Steinicke, and Ken Perlin. 2019. Challenges Using Head-Mounted Displays in Shared and Social Spaces. ACM Human Factors in Computing Systems Extend Abstracts.

Digital Library

[14]

Pieter Hintjens. 2013. ZeroMQ: Messaging for Many Applications. O'Reilly Media, Inc.

[15]

Fu-Chung Huang, David P Luebke, and Gordon Wetzstein. 2015. The Light Field Stereoscope. ACM SIGGRAPH Emerging Technologies.

Digital Library

[16]

Andrew Jones, Magnus Lang, Graham Fyffe, Xueming Yu, Jay Busch, Ian McDowall, Mark Bolas, and Paul Debevec. 2009. Achieving Eye Contact in a One-To-Many 3D Video Teleconferencing System. ACM Transactions on Graphics 28, 3 (2009).

Digital Library

[17]

Andrew V Jones, Koki Nagano, Jing Liu, Jay Busch, Xueming Yu, Mark T Bolas, and Paul Debevec. 2014. Interpolating Vertical Parallax for an Autostereoscopic Three-dimensional Projector Array. Journal of Electronic Imaging 23, 1 (2014).

[18]

Kibum Kim, John Bolton, Audrey Girouard, Jeremy Cooperstock, and Roel Vertegaal. 2012. TeleHuman: Effects of 3D Perspective on Gaze and Pose Estimation with a Life-Size Cylindrical Telepresence Pod. ACM Human Factors in Computing Systems.

Digital Library

[19]

Kopin. 2021. Kopin's 2.6k x 2.6k OLED Display Incorporated in Panasonic's New VR Glasses. (Jan 2021). https://kopin.irpass.com/profiles/investor/NewsPrint.asp?v=6&b=2379&ID=96490&m=rl&g=1207

[20]

Douglas Lanman and David Luebke. 2013. Near-eye Light Field Displays. ACM Transactions on Graphics 32, 6 (2013).

Digital Library

[21]

Douglas Lanman, Gordon Wetzstein, Matthew Hirsch, and Ramesh Raskar. 2013. Depth of Field Analysis for Multilayer Automultiscopic Displays. Journal of Physics: Conference Series 415, 1.

[22]

Hao Li, Laura Trutoiu, Kyle Olszewski, Lingyu Wei, Tristan Trutna, Pei-Lun Hsieh, Aaron Nicholls, and Chongyang Ma. 2015. Facial Performance Sensing Head-Mounted Display. ACM Transactions on Graphics 34, 4 (2015).

Digital Library

[23]

Peter Lincoln, Andrew Nashel, Adrian Ilie, Herman Towles, Gregory Welch, and Henry Fuchs. 2009a. Multi-View Lenticular Display for Group Teleconferencing. Immersive Telecommunications.

Digital Library

[24]

Peter Lincoln, Greg Welch, Andrew Nashel, Adrian Ilie, Andrei State, and Henry Fuchs. 2009b. Animatronic Shader Lamps Avatars. IEEE International Symposium on Mixed and Augmented Reality.

Digital Library

[25]

Stephen Lombardi, Jason Saragih, Tomas Simon, and Yaser Sheikh. 2018. Deep Appearance Models for Face Rendering. ACM Transactions on Graphics 37, 4 (2018).

Digital Library

[26]

Christian Mai and Mohamed Khamis. 2018. Public HMDs: Modeling and Understanding User Behavior around Public Head-Mounted Displays. ACM Pervasive Displays.

Digital Library

[27]

Christian Mai, Lukas Rambold, and Mohamed Khamis. 2017. TransparentHMD: Revealing the HMD User's Face to Bystanders. ACM Mobile and Ubiquitous Multimedia.

Digital Library

[28]

Andrew Maimone, Douglas Lanman, Kishore Rathinavel, Kurtis Keller, David Luebke, and Henry Fuchs. 2014. Pinlight Displays: Wide Field Of View Augmented Reality Eyeglasses Using Defocused Point Light Sources.

Digital Library

[29]

Andrew Maimone and Junren Wang. 2020. Holographic Optics for Thin and Lightweight Virtual Reality. ACM Transactions on Graphics 39, 4 (2020).

Digital Library

[30]

Manuel Martínez-Corral and Bahram Javidi. 2018. Fundamentals of 3D Imaging and Displays: A Tutorial on Integral Imaging, Light-field, and Plenoptic Systems. Advances in Optics and Photonics 10, 3 (2018).

[31]

Nathan Matsuda, Brian Wheelwright, Joel Hegland, and Douglas Lanman. 2021. Reverse Pass-Through VR. ACM SIGGRAPH Emerging Technologies.

Digital Library

[32]

Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. 2016. A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation. IEEE Computer Vision and Pattern Recognition.

[33]

Kana Misawa and Jun Rekimoto. 2015. ChameleonMask: Embodied Physical and Social Telepresence using Human Surrogates. ACM Human Factors in Computing Systems.

Digital Library

[34]

Samer Al Moubayed, Jens Edlund, and Jonas Beskow. 2012. Taming Mona Lisa: Communicating Gaze Faithfully in 2D and 3D Facial Projections. ACM Transactions on Interactive Intelligent Systems 1, 2 (2012).

Digital Library

[35]

Koki Nagano, Andrew Jones, Jing Liu, Jay Busch, Xueming Yu, Mark Bolas, and Paul Debevec. 2013. An Autostereoscopic Projector Array Optimized for 3D Facial Display.

[36]

David Nguyen and John Canny. 2005. MultiView: Spatially Faithful Group Video Conferencing. ACM Human Factors in Computing Systems.

Digital Library

[37]

Kyle Olszewski, Joseph J Lim, Shunsuke Saito, and Hao Li. 2016. High-fidelity Facial and Speech Animation for VR HMDs. ACM Transactions on Graphics 35, 6 (2016).

Digital Library

[38]

Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, Yury Degtyarev, David Kim, Philip L Davidson, Sameh Khamis, Mingsong Dou, et al. 2016. Holoportation: Virtual 3D Teleportation in Real-Time. User Interface Software and Technology.

Digital Library

[39]

Colin J. Palmer, Yumiko Otsuka, and Colin W.G. Clifford. 2020. A sparkle in the eye: Illumination cues and lightness constancy in the perception of eye contact. Cognition 205 (2020).

[40]

Ye Pan and Anthony Steed. 2014. A Gaze-Preserving Situated Multiview Telepresence System. ACM Human Factors in Computing Systems.

Digital Library

[41]

Ye Pan and Anthony Steed. 2016. Effects of 3D perspective on Head Gaze Estimation with a Multiview Autostereoscopic Display. International Journal of Human-Computer Studies 86 (2016).

Digital Library

[42]

Ryan Schubert, Greg Welch, Peter Lincoln, Arjun Nagendran, Remo Pillat, and Henry Fuchs. 2012. Advances in Shader Lamps Avatars for Telepresence. IEEE 3DTV.

[43]

Valentin Schwind, Jens Reinhardt, Rufat Rzayev, Niels Henze, and Katrin Wolf. 2018. Virtual Reality on the Go? A Study on Social Acceptance of VR Glasses. ACM Human-Computer Interaction with Mobile Devices and Services.

Digital Library

[44]

David Sirkin, Gina Venolia, John Tang, George Robertson, Taemie Kim, Kori Inkpen, Mara Sedlins, Bongshin Lee, and Mike Sinclair. 2011. Motion and Attention in a Kinetic Videoconferencing Proxy. IFIP Conference on Human-Computer Interaction.

Digital Library

[45]

Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, and Matthias Nießner. 2018. FaceVR: Real-time Gaze-aware Facial Reenactment in Virtual Reality. ACM Transactions on Graphics 37, 2 (2018).

Digital Library

[46]

Chiu-Hsuan Wang, Seraphina Yong, Hsin-Yu Chen, Yuan-Syun Ye, and Liwei Chan. 2020. HMD Light: Sharing In-VR Experience via Head-mounted Projector for Asymmetric Interaction. User Interface Software and Technology.

Digital Library

[47]

Miao Wang, Xin Wen, and Shi-Min Hu. 2019b. Faithful Face Image Completion for HMD Occlusion Removal. IEEE International Symposium on Mixed and Augmented Reality Adjunct.

[48]

Yan Wang, Zihang Lai, Gao Huang, Brian H Wang, Laurens Van Der Maaten, Mark Campbell, and Kilian Q Weinberger. 2019a. Anytime Stereo Image Depth Estimation on Mobile Devices. IEEE International Conference on Robotics and Automation.

[49]

Shih-En Wei, Jason Saragih, Tomas Simon, Adam W Harley, Stephen Lombardi, Michal Perdoch, Alexander Hypes, Dawei Wang, Hernan Badino, and Yaser Sheikh. 2019. VR Facial Animation via Multiview Image Translation. ACM Transactions on Graphics 38, 4 (2019).

Digital Library

[50]

Gordon Wetzstein, Wolfgang Heidrich, and Ramesh Raskar. 2014. Computational Schlieren Photography with Light Field Probes. International Journal of Computer Vision 110, 2 (2014).

Digital Library

[51]

Austin Wilson and Hong Hua. 2021. Design of a Pupil-Matched Occlusion-Capable Optical See-Through Wearable Display. IEEE TVCG (2021).

[52]

William Hyde Wollaston. 1824. On the Apparent Direction of Eyes in a Portrait. Philosophical Transactions of the Royal Society of London 114 (1824).

[53]

Timothy L Wong, Zhisheng Yun, Gregg Ambur, and Jo Etter. 2017. Folded Optics with Birefringent Reflective Polarizers. Digital Optical Technologies.

[54]

Chris Wyman. 2005. Interactive Image-space Refraction of Nearby Geometry. International Conference on Computer Graphics and Interactive Techniques in Australasia and South East Asia.

Digital Library

[55]

Tatsuo Yotsukura, Frank Nielsen, Kim Binsted, Shigeo Morishima, and Claudio S. Pinhanez. 2002. HyperMask: Talking Head Projected onto Real Object. The Visual Computer 18, 2 (2002).

Digital Library

[56]

Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. 2017. Unpaired Image-to-image Translation Using Cycle-consistent Adversarial Networks. IEEE International Conference on Computer Vision.

Cited By

Du YLi YLi CXu RMeng LBai Y(2025)Optical Adhesives and Screen Sealants for Foldable Displays: Analysis, Progress and TrendsACS Applied Materials & Interfaces10.1021/acsami.4c11678Online publication date: 19-Jan-2025
https://doi.org/10.1021/acsami.4c11678
Wang XHuang WChen KWang ZDeng Y(2024)EAT: epipolar-aware Transformer for low-light light field enhancementMultimedia Tools and Applications10.1007/s11042-024-19271-zOnline publication date: 17-May-2024
https://doi.org/10.1007/s11042-024-19271-z
Chen YJiang GYu MXu HHo Y(2023)Learning to simultaneously enhance field of view and dynamic range for light field imagingInformation Fusion10.1016/j.inffus.2022.10.02191:C(215-229)Online publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1016/j.inffus.2022.10.021

Index Terms

VR social copresence with light field displays
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Virtual reality

Recommendations

Reverse Pass-Through VR
SIGGRAPH '21: ACM SIGGRAPH 2021 Emerging Technologies

We introduce reverse pass-through VR, wherein a three-dimensional view of the wearer’s eyes is presented to multiple outside viewers in a perspective-correct manner, with a prototype headset containing a world-facing light field display. This approach, ...
Near-eye light field displays
SIGGRAPH '13: ACM SIGGRAPH 2013 Emerging Technologies

We propose a light-field-based approach to near-eye display that allows for thin, lightweight head-mounted displays capable of depicting accurate accommodation, convergence, and binocular disparity depth cues. Our near-eye light field displays depict ...
Near-eye light field displays

We propose near-eye light field displays that enable thin, lightweight head-mounted displays (HMDs) capable of presenting nearly correct convergence, accommodation, binocular disparity, and retinal defocus depth cues. Sharp images are depicted by out-of-...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 40, Issue 6

December 2021

1351 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3478513

Issue’s Table of Contents

Copyright © 2021 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 December 2021

Published in TOG Volume 40, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
811
Total Downloads

Downloads (Last 12 months)219
Downloads (Last 6 weeks)29

Reflects downloads up to 18 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Du YLi YLi CXu RMeng LBai Y(2025)Optical Adhesives and Screen Sealants for Foldable Displays: Analysis, Progress and TrendsACS Applied Materials & Interfaces10.1021/acsami.4c11678Online publication date: 19-Jan-2025
https://doi.org/10.1021/acsami.4c11678
Wang XHuang WChen KWang ZDeng Y(2024)EAT: epipolar-aware Transformer for low-light light field enhancementMultimedia Tools and Applications10.1007/s11042-024-19271-zOnline publication date: 17-May-2024
https://doi.org/10.1007/s11042-024-19271-z
Chen YJiang GYu MXu HHo Y(2023)Learning to simultaneously enhance field of view and dynamic range for light field imagingInformation Fusion10.1016/j.inffus.2022.10.02191:C(215-229)Online publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1016/j.inffus.2022.10.021

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Figures

Tables

Media

View Issue’s Table of Contents