other

Intermediated Reality with an AI 3D Printed Character

Authors:

Llogari Casas,

Kenny MitchellAuthors Info & Claims

SIGGRAPH '23: ACM SIGGRAPH 2023 Real-Time Live!

Article No.: 5, Pages 1 - 2

https://doi.org/10.1145/3588430.3597251

Published: 23 July 2023 Publication History

Get Access

Abstract

We introduce live character conversational interactions in Intermediated Reality to bring real-world objects to life in Augmented Reality (AR) and Artificial Intelligence (AI). The AI recognizes live speech and generates short character responses, syncing the character’s facial expressions with speech audio. The Intermediated Reality AR warping technique allows for a high degree of realism in the animated facial expressions and movements reusing live video optical appearance direct from the device’s embedded camera. The proposed applications of Intermediated Reality with AI are exemplified through the captivating fusion of these technologies in toy interactive storytelling broadcasts and social telepresence scenarios. This innovative combination allows individuals to engage with AI characters in a natural and intuitive manner, creating new opportunities for social engagement and entertainment.

Supplementary Material

MP4 File (IntermediatedRealityWithAnAI3DPrintedCharacter.mp4)

Video

Download
99.16 MB

References

[1]

ARKit. 2022. Apple ARKit. https://developer.apple.com/augmented-reality/arkit/

Google Scholar

[2]

Llogari Casas, Matthias Fauconneau, Maggie Kosek, Kieran Mclister, and Kenny Mitchell. 2018. Image Based Proximate Shadow Retargeting. In Computer Graphics and Visual Computing (CGVC), Gary K. L. Tam and Franck Vidal (Eds.). The Eurographics Association, Swansea, UK. https://doi.org/10.2312/cgvc.20181206

Digital Library

Google Scholar

[3]

Llogari Casas, Maggie Kosek, and Kenny Mitchell. 2017. Props Alive: A Framework for Augmented Reality Stop Motion Animation. In 2017 IEEE 10th Workshop on Software Engineering and Architectures for Realtime Interactive Systems (SEARIS). 1–4. https://doi.org/10.1109/SEARIS41720.2017.9183487

Crossref

Google Scholar

[4]

Llogari Casas and Kenny Mitchell. 2019. Intermediated Reality: A Framework for Communication Through Tele-Puppetry. Frontiers in Robotics and AI 6 (2019). https://doi.org/10.3389/frobt.2019.00060

Crossref

Google Scholar

[5]

A.I. Comport, E. Marchand, M. Pressigout, and F. Chaumette. 2006. Real-time markerless tracking for augmented reality: the virtual visual servoing framework. IEEE Transactions on Visualization and Computer Graphics 12, 4 (2006), 615–628. https://doi.org/10.1109/TVCG.2006.78

Digital Library

Google Scholar

[6]

Steve Mann. 1999. Mediated Reality. Linux J. 1999, 59es (mar 1999), 5–es.

Google Scholar

[7]

Mehmet S. Yavas. 2011. Phonetics. John Wiley Sons Ltd, New York, NY, USA, Chapter 1, 1–29. https://doi.org/10.1002/9781444392623.ch1 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/9781444392623.ch1

Crossref

Google Scholar

[8]

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. A Survey of Large Language Models. arxiv:2303.18223 [cs.CL]

Google Scholar

Cited By

View all

Constantinides NConstantinides AKoukopoulos DFidas CBelk MBoratto LGena CMarras MGermanakos PPopescus E(2024)CulturAI: Exploring Mixed Reality Art Exhibitions with Large Language Models for Personalized Immersive ExperiencesAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3664874(102-105)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3631700.3664874
Casas LHannah SMitchell K(2024)MoodFlow: Orchestrating Conversations with Emotionally Intelligent Avatars in Mixed Reality2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00021(86-89)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VRW62533.2024.00021

Index Terms

Intermediated Reality with an AI 3D Printed Character
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Mixed / augmented reality
2. Human-centered computing
  1. Interaction design
    1. Interaction design theory, concepts and paradigms

Recommendations

Intermediated Reality
SA '19: SIGGRAPH Asia 2019 Doctoral Consortium

This thesis explores technical solutions to address the gap between the virtual and physical worlds towards photo-realistic interactive Augmented Reality (AR). As mobile network bandwidth increases, latencies reduce and graphics processing power becomes ...
Initial lessons from AR Façade, an interactive augmented reality drama
ACE '06: Proceedings of the 2006 ACM SIGCHI international conference on Advances in computer entertainment technology

In this paper, we describe an augmented reality version of the acclaimed desktop-based interactive drama, Façade [18]. Few entertainment experiences combine interactive virtual characters, non-linear narrative, and unconstrained embodied interaction. In ...
Augmented reality as perceptual reality
VSMM'06: Proceedings of the 12th international conference on Interactive Technologies and Sociotechnical Systems

As shown in Paul Milgram et al’s Reality-Virtuality Continuum (1994), Augmented Reality occupies a very unique status in the spectrum of Mixed Reality. Unlike Virtual Reality, which is completely made up of the virtual and has been the most important ...

Comments

Information & Contributors

Information

Published In

SIGGRAPH '23: ACM SIGGRAPH 2023 Real-Time Live!

July 2023

29 pages

ISBN:9798400701580

DOI:10.1145/3588430

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 July 2023

Check for updates

Author Tags

Qualifiers

Other
Research
Refereed limited

Funding Sources

Innovate UK

Conference

SIGGRAPH '23

Sponsor:

SIGCHI

SIGGRAPH '23: Special Interest Group on Computer Graphics and Interactive Techniques Conference

August 6 - 10, 2023

CA, Los Angeles, USA

Acceptance Rates

Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
126
Total Downloads

Downloads (Last 12 months)44
Downloads (Last 6 weeks)3

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Constantinides NConstantinides AKoukopoulos DFidas CBelk MBoratto LGena CMarras MGermanakos PPopescus E(2024)CulturAI: Exploring Mixed Reality Art Exhibitions with Large Language Models for Personalized Immersive ExperiencesAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3664874(102-105)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3631700.3664874
Casas LHannah SMitchell K(2024)MoodFlow: Orchestrating Conversations with Emotionally Intelligent Avatars in Mixed Reality2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00021(86-89)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VRW62533.2024.00021

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Abstract

Supplementary Material

References

Cited By

Index Terms

Recommendations