Objectifying Social Presence: Evaluating Multimodal Degraders in ECAs Using the Heard Text Recall Paradigm
IEEE Transactions on Visualization and Computer Graphics, vol. 32, pp. 2312–2325
Abstract
Embodied conversational agents (ECAs) are key social interaction partners in various virtual reality (VR) applications, with their perceived social presence significantly influencing the quality and effectiveness of user-ECA interactions. This article investigates the potential of a novel indirect objective proxy for evaluating social presence, which is traditionally assessed through subjective questionnaires. As proxy we use the Heard Text Recall (HTR) paradigm, originally designed to assess memory performance in listening tasks. In the task participants have to recall information from family stories being told. When combined with a secondary task in a dual-task paradigm, the HTR can also be used to evaluate cognitive spare capacity, which is then correlated with subjectively rated social presence. As a prerequisite for this investigation, we introduce various co-verbal gesture modification techniques and assess their impact on the perceived naturalness of the presenting ECA, a crucial aspect fostering social presence. The main study then explores the applicability of HTR as a proxy for social presence by examining its effectiveness under different multimodal degraders of ECA behavior, including degraded co-verbal gestures, omitted lip synchronization, and the use of synthetic voices. The findings suggest that while HTR shows potential as an objective measure of social presence, its effectiveness is primarily evident in response to substantial changes in ECA behavior. Additionally, the study also highlights the negative effects of synthetic voices and inadequate lip synchronization on perceived social presence, emphasizing the need for careful consideration of these elements in ECA design.
Authors 8
-
Affiliation as printed
Visual Computing Institute, RWTH Aachen University, Aachen, Germany
Visual Computing Institute, RWTH Aachen University, Germany
-
Affiliation as printed
Faculty of Computer Science, RWTH Aachen University, Aachen, Germany
Faculty of Computer Science, RWTH Aachen University, Germany
-
Affiliation as printed
Work and Engineering Psychology, RWTH Aachen University, Aachen, Germany
Work and Engineering Psychology, RWTH Aachen University, Germany
-
Affiliation as printed
Institute for Hearing Technology and Acoustics, RWTH Aachen University, Aachen, Germany
Institute for Hearing Technology and Acoustics, RWTH Aachen University, Germany
-
Affiliation as printed
Institute for Hearing Technology and Acoustics, RWTH Aachen University, Aachen, Germany
Institute for Hearing Technology and Acoustics, RWTH Aachen University, Germany
-
Affiliation as printed
Work and Engineering Psychology, RWTH Aachen University, Aachen, Germany
Work and Engineering Psychology, RWTH Aachen University, Germany
-
Affiliation as printed
Visual Computing Institute, RWTH Aachen University, Aachen, Germany
Visual Computing Institute, RWTH Aachen University, Germany
-
Affiliation as printed
Visual Computing Institute, RWTH Aachen University, Aachen, Germany
Visual Computing Institute, RWTH Aachen University, Germany
Cited by 0 stored of 0
No patents citing this paper on Lens.org (checked 2026-10-06).