Estuary: A Framework For Building Multimodal Low-Latency Real-Time Socially Interactive Agents

Abstract

The rise in capability and ubiquity of generative artificial intelligence (AI) technologies has enabled its application to the field of Socially Interactive Agents (SIAs). Despite rising interest in modern AI-powered components used for real-time SIA research, substantial friction remains due to the absence of a standardized and universal SIA framework. To target this absence, we developed Estuary: a multimodal (text, audio, and soon video) framework which facilitates the development of low-latency, real-time SIAs. Estuary seeks to reduce repeat work between studies and to provide a flexible platform that can be run entirely off-cloud to maximize configurability, controllability, reproducibility of studies, and speed of agent response times. We are able to do this by constructing a robust multimodal framework which incorporates current and future components seamlessly into a modular and interoperable architecture.

Figure 1: Estuary’s various capabilities in Augmented Reality and Virtual Reality environments.

1 Background

Rapid advancements in AI have catalyzed the development of SIAs, which integrate complex technologies to facilitate nuanced human-computer interactions across various domains, producing affective virtual agents [15], multi-agent simulations of human behavior [20, 27, 13], and more [16, 17, 25]. To build effective SIAs, multiple microservices and components need to be integrated and managed, including Automatic Speech Recognition (ASR) and Text-To-Speech (TTS). Implementing these features require significant efforts. A modern, comprehensive SIA framework is useful to streamline efforts and reduce redundant work [18] and to provide a scalable and interoperable standard that can support new models and microservices.

1.1 SIA & Conversational AI Frameworks.

Several existing toolkits help streamline the process of building SIAs. The Virtual Human Toolkit (VHToolkit) [14] and Greta [22] are such examples, however, they do not support generative AI models such as Large Language Models (LLMs). Moreover, a recent wave of lightweight, high performing models (Speech-To-Text (STT) [24, 21], LLM [5], and TTS [9]) can be run on-edge to overcome privacy concerns evident in cloud-based services. These gaps and improvements in current tools add additional motivation for the creation of a cohesive framework tailored for SIAs.

Recent projects such as Pipecat [12] and NVIDIA ACE [3] provide frameworks for building conversational agents by connecting AI microservices. However, Pipecat is not tailored for building SIAs and lacks integration with game engines, which SIA research heavily relies upon. NVIDIA ACE, at the time of writing, is not open-source and restricts users to only running NVIDIA-approved AI microservices. This may be a critical limitation for research involving custom AI microservices and pipelines. Furthermore, NVIDIA ACE requires a cost-prohibitive enterprise plan if developers would like to host their microservices off-cloud or on-prem.

1.2 Limitations of Current Approaches.

Currently, several factors hinder SIA research: 1) computational limitations of devices (e.g., head-mounted displays (HMDs)) which prohibit running advanced AI models on a standalone device, 2) hardware architecture incompatibilities with AI models which necessitate an inferencing server, and 3) high latency of cloud-based microservices. To address these shortcomings, we developed Estuary, a distributed framework for low-latency real-time SIAs. Estuary simplifies the development process by seamlessly integrating user-defined modules for ASR, TTS, dialogue management, and LLMs with the purpose of constructing pipelines for SIAs. This allows it to support low-latency real-time voice interactions, understand the physical world, and to facilitate reproducible and highly configurable experiments.

Figure 2: Estuary leveraging mesh classification capabilities into multimodal interaction.

2 System

Estuary brings five core value-propositions: 1) an interoperable microservice architecture, 2) multi-platform support, 3) off-cloud capabilities, 4) support for multimodal input and analysis, and 5) an open-source nature that opens it to community contributions.

2.1 Microservice Architecture.

We use a modular design to universally wrap local model or online API service within a Stage, which are asynchronous and parallelizable as denoted in Figure 3. A Stage component wraps the ML inference logic and hosts it on a child process, aggregating inputs and dispatching outputs. A selection of Stages (e.g., Whisper [23], GPT-3.5 [7], gTTS [11]) are connected into a $Pipeline$ according to the flow of choice. The $Pipeline$ internally orchestrates the flow of a standardized data type $DataWindow$, which consists of one or more $DataPacket$(s) (e.g., $AudioPacket$, $VisionPacket$, $TextPacket$, etc.). A $DataPacket$, with its source and creation timestamp, acts as identifiable placeholders for the Stage outcomes. This micro-service architecture allows us to plug and play new models rapidly by implementing them as Stage(s). A $Pipeline$ out of the box runs as a background process on a server that can communicate with any client through SocketIO protocol [4].

Figure 3: System diagram of the server and client

2.2 Multi-platform.

Estuary is a distributed framework that uses the SocketIO protocol to establish a connection between a client device and a host device running our framework. It can support any platform, including those supported by Unity game engine given that the hardware has a microphone, speaker, and can communicate using SocketIO. This hardware-agnostic support makes available HMDs and other devices by overcoming compute limitations or hardware architecture incompatibilities.

2.3 Off-Cloud.

In addition to leveraging cloud APIs, Estuary can be hosted locally and/or entirely with off-cloud microservices. This reduces latency by eliminating multiple cloud endpoints, improves reproducibility, and adds privacy and security as no data ever enters the cloud. As shown in our video demo, it takes approximately 1.2$\sim$2.5 seconds (compared to an average 2.8 second latency from ChatGPT-4o  [19]) from the end of a user’s dialogue to the first utterance from the SIA’s TTS module through a FasterWhisperBase.EN [2] $\rightarrow$ GPT-3.5 API [7] $\rightarrow$ XTTS [9] pipeline on a desktop with a RTX 4090 graphics card. This is made possible through several optimizations such as simultaneously streaming the LLM and TTS response. Furthermore, off-cloud microservices ensure reproducibility, which is of utmost importance especially in fields relating to psychology [10]. In Estuary, the exact same versions of LLM, ASR, TTS, and other microservices can be maintained and loaded for future use, whereas cloud-based services do not have guarantees to remain unchanged overtime.

2.4 Multimodal Input And Analysis.

Multimodal input  [19] is critical for agents to understand the physical world and multiple modalities to produce a better cognitive model [6, 8]. Estuary integrates with Unity and packages like ARKit  [26] to empower embodied agents with basic semantic understanding of the physical world and pathfinding capabilities in Augmented Reality (AR). Currently, Estuary supports text and audio datastreams. We plan to expand to video as well which will shift reliance away from hardware-specific packages.

2.5 Open-Source.

Estuary is open-source and promotes growth from community contribution. By nature, Estuary is flexible and hugely extensible to support integration of microservices now and into the future. Researchers and developers have the freedom to choose what models they use and what data is collected from anywhere in the pipeline, all without limitations imposed by paid services. Estuary is built as a robust, modern framework that can empower SIA research for years to come.

3 Demonstration

Our first scenario consists of a computer hosting Estuary interfaced to an HMD over a local network to demonstrate an advanced embodied conversational agent that can classify objects in its physical surroundings and interact accordingly. The second scenario consists of a computer hosting both Estuary and a desktop frontend to demonstrate the versatility of our framework. Estuary’s source code can be found in our GitHub  and a demo video here on our website .

4 Acknowledgments

We would like to thank our teammates and advisors from the 2022 and 2023 NASA SUITS Challenge whose efforts contributed to the success of this project. We would also like to thank all open-source developers and creatives whose work we respectfully used.