Fostering Engagement Through a Latency-Optimized LLM-Based Dialogue System for Multimodal ECA Responses
IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR), pp. 85–97
Abstract
Interactions with Embodied Conversational Agents (ECAs) are an integral part of many social Virtual Reality (VR) applications, increasing the need for free, context-sensitive conversations characterized by latency-optimized and multimodal ECA responses. Our presented methodology consists of three interdependent steps: We first present a holistic framework driven by a Large Language Model (LLM), which integrates existing technologies into a modular and extendable system that is developer-friendly and suitable for diverse use-cases. Building on this foundation, our second step comprises streaming-based optimizations that effectively reduce measured response latency, thereby facilitating real-time conversations. Finally, we conduct a comparative analysis between our latency optimized LLM-driven ECA and a conventional button-based Wizard-of-Oz (WoZ) system to evaluate performance differences in user engagement. Our insights reveal that users perceive our LLM-driven ECA as significantly more natural, competent, and trustworthy than their WoZ counterparts, despite objective measures indicating slightly higher latency in technical performance. These findings underscore the potential of LLMs to enhance engagement in ECAs within VR environments.
Authors 4
-
Affiliation as printed
RWTH Aachen University,Visual Computing Institute,Aachen,Germany
-
Affiliation as printed
RWTH Aachen University,Visual Computing Institute,Aachen,Germany
-
Affiliation as printed
RWTH Aachen University,Visual Computing Institute,Aachen,Germany
-
Affiliation as printed
RWTH Aachen University,Visual Computing Institute,Aachen,Germany
Cited by 2 stored of 2
2 results
No patents citing this paper on Lens.org (checked 2026-10-06).