extended-abstract

Public Access

Embodied Human-Computer Interactions through Situated Grounding

Authors:

James Pustejovsky,

Nikhil KrishnaswamyAuthors Info & Claims

IVA '20: Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents

Article No.: 48, Pages 1 - 3

https://doi.org/10.1145/3383652.3423910

Published: 19 October 2020 Publication History

PDF eReader

Abstract

In this paper, we introduce a simulation platform for modeling and building Embodied Human-Computer Interactions (EHCI). This system, VoxWorld, is a multimodal dialogue system enabling communication through language, gesture, action, facial expressions, and gaze tracking, in the context of task-oriented interactions. A multimodal simulation is an embodied 3D virtual realization of both the situational environment and the co-situated agents, as well as the most salient content denoted by communicative acts in a discourse. It is built on the modeling language VoxML [7], which encodes objects with rich semantic typing and action affordances, and actions themselves as multimodal programs, enabling contextu-ally salient inferences and decisions in the environment. VoxWorld enables an embodied HCI by situating both human and computational agents within the same virtual simulation environment, where they share perceptual and epistemic common ground.

References

[1]

Yiannis Gatsoulis, Muhannad Alomari, Chris Burbridge, Christian Dondrup, Paul Duckworth, Peter Lightbody, Marc Hanheide, Nick Hawes, DC Hogg, AG Cohn, et al. 2016. QSRlib: a software library for online acquisition of Qualitative Spatial Relations from Video. (2016).

Google Scholar

[2]

Nikhil Krishnaswamy, Scott Friedman, and James Pustejovsky. 2019. Combining Deep Learning and Qualitative Spatial Reasoning to Learn Complex Structures from Sparse Examples with Noise. In 33rd AAAI Conference on Artificial Intelligence. AAAI.

Digital Library

Google Scholar

[3]

Nikhil Krishnaswamy and James Pustejovsky. 2016. VoxSim: A Visual Platform for Modeling Motion Language. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. ACL.

Google Scholar

[4]

David G McNeely-White, Francisco R Ortega, J Ross Beveridge, Bruce A Draper, Rahul Bangar, Dhruva Patil, James Pustejovsky, Nikhil Krishnaswamy, Kyeong-min Rim, Jaime Ruiz, and Isaac Wang. 2019. User-Aware Shared Perception for Embodied Agents. In 2019 IEEE International Conference on Humanized Computing and Communication (HCC). IEEE, 46--51.

Google Scholar

[5]

Pradyumna Narayana, Nikhil Krishnaswamy, Isaac Wang, Rahul Bangar, Dhruva Patil, Gururaj Mulay, Kyeongmin Rim, Ross Beveridge, Jaime Ruiz, James Pustejovsky, and Bruce Draper. 2018. Cooperating with Avatars Through Gesture, Language and Action. In Intelligent Systems Conference (IntelliSys).

Google Scholar

[6]

James Pustejovsky. 1995. The Generative Lexicon. MIT Press, Cambridge, MA.

Google Scholar

[7]

James Pustejovsky and Nikhil Krishnaswamy. 2016. VoxML: A Visualization Modeling Language. Proceedings of LREC (2016).

Google Scholar

Cited By

View all

Alalyani NKrishnaswamy N(2024)Multimodal Referring Expression Generation for Human-Computer InteractionHCI International 2024 – Late Breaking Papers10.1007/978-3-031-76827-9_1(3-22)Online publication date: 31-Dec-2024
https://doi.org/10.1007/978-3-031-76827-9_1
VanderHoeven HBlanchard NKrishnaswamy N(2024)Point Target Detection for Multimodal CommunicationDigital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management10.1007/978-3-031-61060-8_25(356-373)Online publication date: 1-Jun-2024
https://doi.org/10.1007/978-3-031-61060-8_25
Alalyani NKrishnaswamy N(2023)A Methodology for Evaluating Multimodal Referring Expression Generation for Embodied Virtual AgentsCompanion Publication of the 25th International Conference on Multimodal Interaction10.1145/3610661.3616548(164-173)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3610661.3616548
Show More Cited By

Index Terms

Embodied Human-Computer Interactions through Situated Grounding
1. Human-centered computing

Recommendations

Authoring Communicative Behaviors for Situated, Embodied Characters
ICMI '14: Proceedings of the 16th International Conference on Multimodal Interaction

Embodied conversational agents hold great potential as multimodal interfaces due to their ability to communicate naturally using speech and nonverbal cues. The goal of my research is to enable animators and designers to endow ECAs with interactive ...
Impression Detection and Management Using an Embodied Conversational Agent
Human-Computer Interaction. Multimodal and Natural Interaction
Abstract
Embodied Conversational Agents (ECAs) are a promising medium for human-computer interaction, since they are capable of engaging users in real-time face-to-face interaction [1, 2]. Users’ formed impressions of an ECA (e.g. favour or dislike) could ...
Modeling believable human-computer interaction with an embodied conversational agent: face-to-face communication of uncertainty

Comments

Information & Contributors

Information

Published In

IVA '20: Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents

October 2020

394 pages

ISBN:9781450375863

DOI:10.1145/3383652

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 October 2020

Check for updates

Author Tags

Qualifiers

Extended-abstract
Research
Refereed limited

Funding Sources

Defense Advanced Research Projects Agency

Conference

IVA '20

Sponsor:

SIGAI

IVA '20: ACM International Conference on Intelligent Virtual Agents

October 20 - 22, 2020

Scotland, Virtual Event, UK

Acceptance Rates

Overall Acceptance Rate 53 of 196 submissions, 27%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
350
Total Downloads

Downloads (Last 12 months)95
Downloads (Last 6 weeks)17

Reflects downloads up to 02 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Alalyani NKrishnaswamy N(2024)Multimodal Referring Expression Generation for Human-Computer InteractionHCI International 2024 – Late Breaking Papers10.1007/978-3-031-76827-9_1(3-22)Online publication date: 31-Dec-2024
https://doi.org/10.1007/978-3-031-76827-9_1
VanderHoeven HBlanchard NKrishnaswamy N(2024)Point Target Detection for Multimodal CommunicationDigital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management10.1007/978-3-031-61060-8_25(356-373)Online publication date: 1-Jun-2024
https://doi.org/10.1007/978-3-031-61060-8_25
Alalyani NKrishnaswamy N(2023)A Methodology for Evaluating Multimodal Referring Expression Generation for Embodied Virtual AgentsCompanion Publication of the 25th International Conference on Multimodal Interaction10.1145/3610661.3616548(164-173)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3610661.3616548
Eman S(2021)Shifting from face-to-face learning to Zoom online teaching, research, and internship supervision in a technologically developing ‘female students’ university in Pakistan: A psychology teacher’s and students’ perspectivePsychology Teaching Review10.53841/bpsptr.2021.27.1.4227:1(42-55)Online publication date: 1-Jan-2021
https://doi.org/10.53841/bpsptr.2021.27.1.42
Pustejovsky JKrishnaswamy N(2021)Embodied Human Computer InteractionKI - Künstliche Intelligenz10.1007/s13218-021-00727-535:3-4(307-327)Online publication date: 16-Sep-2021
https://doi.org/10.1007/s13218-021-00727-5
Pustejovsky JKrishnaswamy N(2021)The Role of Embodiment and Simulation in Evaluating HCI: Theory and FrameworkDigital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Human Body, Motion and Behavior10.1007/978-3-030-77817-0_21(288-303)Online publication date: 24-Jul-2021
https://dl.acm.org/doi/10.1007/978-3-030-77817-0_21

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Authoring Communicative Behaviors for Situated, Embodied Characters

Impression Detection and Management Using an Embodied Conversational Agent

Modeling believable human-computer interaction with an embodied conversational agent: face-to-face communication of uncertainty

Comments

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Funding Sources

Conference

Acceptance Rates

Other Metrics

Article Metrics

Other Metrics

Cited By

PDF

eReader

Login options

Full Access

Abstract

References

Cited By

Index Terms

Recommendations

Authoring Communicative Behaviors for Situated, Embodied Characters

Impression Detection and Management Using an Embodied Conversational Agent

Modeling believable human-computer interaction with an embodied conversational agent: face-to-face communication of uncertainty

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Funding Sources

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

Login options

Full Access

Share

Share this Publication link

Share on social media

Affiliations