abstract

AI-Mediated 3D Video Conferencing

Authors:

David LuebkeAuthors Info & Claims

SIGGRAPH '23: ACM SIGGRAPH 2023 Emerging Technologies

Article No.: 4, Pages 1 - 2

https://doi.org/10.1145/3588037.3595385

Published: 26 July 2023 Publication History

Get Access

Abstract

We present an AI-mediated 3D video conferencing system that can reconstruct and autostereoscopically display a life-sized talking head using consumer-grade compute resources and minimal capture equipment. Our 3D capture uses a novel 3D lifting method that encodes a given 2D input into an efficient triplanar neural representation of the user, which can be rendered from novel viewpoints in real-time. Our AI-based techniques drastically reduce the cost for 3D capture, while providing a high-fidelity 3D representation on the receiver’s end at the cost of traditional 2D video streaming. Additional advantages of our AI-based approach include the ability to accommodate both photorealistic and stylized avatars, and the ability to enable mutual eye contact in multi-directional video conferencing. We demonstrate our system using a tracked stereo display for a personal viewing experience as well as a lightfield display for a room-scale multi-viewer experience.

References

[1]

Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. 2022. Efficient Geometry-aware 3D Generative Adversarial Networks. In CVPR.

Google Scholar

[2]

Andrew Jones, Magnus Lang, Graham Fyffe, Xueming Yu, Jay Busch, Ian McDowall, Mark Bolas, and Paul Debevec. 2009. HeadSPIN: A One-to-Many 3D Video Teleconferencing System. In ACM SIGGRAPH 2009 Emerging Technologies.

Digital Library

Google Scholar

[3]

Andrew Jones, Jonas Unger, Koki Nagano, Jay Busch, Xueming Yu, Hsuan-Yueh Peng, Oleg Alexander, Mark Bolas, and Paul Debevec. 2015. An Automultiscopic Projector Array for Interactive Digital Humans. In ACM SIGGRAPH 2015 Emerging Technologies.

Google Scholar

[4]

Jason Lawrence, Danb Goldman, Supreeth Achar, Gregory Major Blascovich, Joseph G. Desloge, Tommy Fortes, Eric M. Gomez, Sascha Häberling, Hugues Hoppe, Andy Huibers, Claude Knaus, Brian Kuschak, Ricardo Martin-Brualla, Harris Nover, Andrew Ian Russell, Steven M. Seitz, and Kevin Tong. 2021. Project Starline: A High-Fidelity Telepresence System. ACM Trans. Graph. (2021).

Digital Library

Google Scholar

[5]

Koki Nagano, Andrew Jones, Jing Liu, Jay Busch, Xueming Yu, Mark Bolas, and Paul Debevec. 2013. An Autostereoscopic Projector Array Optimized for 3D Facial Display. In ACM SIGGRAPH 2013 Emerging Technologies.

Digital Library

Google Scholar

[6]

Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. 2021. One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing. In CVPR.

Google Scholar

[7]

Yufeng Zheng, Seonwook Park, Xucong Zhang, Shalini De Mello, and Otmar Hilliges. 2020. Self-learning transformations for improving gaze and head redirection. NeurIPS (2020).

Google Scholar

Cited By

View all

Yang ZLiu BSong YYi LXiong YZhang ZYu X(2024)DirectL: Efficient Radiance Fields Rendering for 3D Light Field DisplaysACM Transactions on Graphics10.1145/368789743:6(1-19)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687897
Tu HShao RDong XZheng SZhang HChen LWang MLi WMa SZhang SZhou BLiu Y(2024)Tele-Aloha: A Telepresence System with Low-budget and High-authenticity Using Sparse RGB CamerasACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657491(1-12)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657491
Lawrence JOverbeck RPrives TFortes TRoth NNewman B(2024)Project Starline: A high-fidelity telepresence systemACM SIGGRAPH 2024 Emerging Technologies10.1145/3641517.3664381(1-2)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641517.3664381
Show More Cited By

Recommendations

Marker Tracking and HMD Calibration for a Video-Based Augmented Reality Conferencing System
IWAR '99: Proceedings of the 2nd IEEE and ACM International Workshop on Augmented Reality

We describe an augmented reality conferencing system which uses the overlay of virtual images on the real world. Remote collaborators are represented on Virtual Monitors which can be freely positioned about a user in space. Users can collaboratively ...
Real-time 3D video avatar in mixed reality: an implementation for immersive telecommunication
Symposium: virtual reality simulation

This article presents an implementation of a real-time dynamic 3D avatar from multiview cameras for immersive telecommunication. Immersive telecommunication is a new challenging field that enables a user to share a virtual space with remote participants,...
Virtual reality conferencing: multi-user immersive VR experiences on the web
MMSys '18: Proceedings of the 9th ACM Multimedia Systems Conference

Virtual Reality (VR) and 360-degree video are set to become part of the future social environment, enriching and enhancing the way we share experiences and collaborate remotely. While Social VR applications are getting more momentum, most services ...

Comments

Information & Contributors

Information

Published In

SIGGRAPH '23: ACM SIGGRAPH 2023 Emerging Technologies

July 2023

44 pages

ISBN:9798400701542

DOI:10.1145/3588037

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 July 2023

Check for updates

Qualifiers

Abstract
Research
Refereed limited

Conference

SIGGRAPH '23

Sponsor:

SIGGRAPH

SIGGRAPH '23: Special Interest Group on Computer Graphics and Interactive Techniques Conference

August 6 - 10, 2023

CA, Los Angeles, USA

Acceptance Rates

Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
459
Total Downloads

Downloads (Last 12 months)179
Downloads (Last 6 weeks)22

Reflects downloads up to 15 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Yang ZLiu BSong YYi LXiong YZhang ZYu X(2024)DirectL: Efficient Radiance Fields Rendering for 3D Light Field DisplaysACM Transactions on Graphics10.1145/368789743:6(1-19)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687897
Tu HShao RDong XZheng SZhang HChen LWang MLi WMa SZhang SZhou BLiu Y(2024)Tele-Aloha: A Telepresence System with Low-budget and High-authenticity Using Sparse RGB CamerasACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657491(1-12)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657491
Lawrence JOverbeck RPrives TFortes TRoth NNewman B(2024)Project Starline: A high-fidelity telepresence systemACM SIGGRAPH 2024 Emerging Technologies10.1145/3641517.3664381(1-2)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641517.3664381
Katsumata KVo DLiu BNakayama H(2024)Revisiting Latent Space of GAN Inversion for Robust Real Image Editing2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00523(5301-5310)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00523
Huang HZhou XNing HLu HZhang QLiang YLyu WJia CZhang XZhang LMa S(2024)Low-Complexity 3D-Vision Conferencing System based on Accelerated RIFE Model2024 Picture Coding Symposium (PCS)10.1109/PCS60826.2024.10566458(1-5)Online publication date: 12-Jun-2024
https://doi.org/10.1109/PCS60826.2024.10566458
Xu WShen YLin QAllebach JZhu F(2024)Pose Guided Portrait View Interpolation from Dual Cameras with a Long Baseline2024 IEEE 26th International Workshop on Multimedia Signal Processing (MMSP)10.1109/MMSP61759.2024.10743353(1-6)Online publication date: 2-Oct-2024
https://doi.org/10.1109/MMSP61759.2024.10743353
Annamalai RBharathraj SDharaneesh G(2024)BrMEPS: Review on 3D Body Reconstruction and Measurement Estimation for Personalized ClothingProceedings of International Conference on Recent Innovations in Computing10.1007/978-981-97-3442-9_52(737-750)Online publication date: 23-Oct-2024
https://doi.org/10.1007/978-981-97-3442-9_52

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Abstract

References

Cited By

Recommendations

Marker Tracking and HMD Calibration for a Video-Based Augmented Reality Conferencing System

Real-time 3D video avatar in mixed reality: an implementation for immersive telecommunication

Virtual reality conferencing: multi-user immersive VR experiences on the web

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

HTML Format

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations