Simulating complex listening scenarios in audiovisual virtual reality : the impact of audiovisual cues, spatial noise, and room acoustics on listening performance
RWTH Publications (RWTH Aachen)
Abstract
Real-life listening is inherently complex; scenarios often contain multiple partially masking sound sources, varying spatial configurations, and room acoustic reflections. Moreover, listeners typically rely not only on auditory but also on visual cues, such as lip movements, gestures, gaze, and embodiment. Laboratory listening experiments often fail to capture this complexity. For example, many studies employ simplified environments when examining listening performance, which may result in findings that do not generalise to real-life listening. Recent technological advances enable the simulation of complex auditory and visual environments, e.g., through virtual reality (VR).By utilising these techniques, listening experiments can be designed to imitate real-life listening more closely and to increase complexity. Consequently, the primary objective of this thesis was to assess how increasing complexity influences listening performance, conceptualised in terms of working memory and listening effort. In seven experiments, reported in five publications, systematic increments in auditory and audiovisual complexity were evaluated. More precisely, the experiments introduced spatial audio reproduction and spatial target and background position changes, varied the semantic content and temporal structure of background noise, examined the influence of display device and audiovisual incongruences in virtual environments with embodied conversational agents (ECAs), and finally investigated the influence of room acoustic conditions in a simulated audiovisually complex environment. Furthermore, throughout the thesis, task complexity was increased from the classical serial recall task with isolated digits as target items towards a joint analysis of memory and listening effort by employing the heard-text-recall (HTR) task in a dual-task design. The findings suggest a task-dependent relevance of these complexity increments and highlight the interdependence of environment, target source, task, and receiver. In conclusion, the thesis proposes a framework of audiovisual complexity that extends existing acoustic models by explicitly incorporating visual, receiver-related, and technical dimensions. In addition, a conceptual distinction between descriptive and generative approaches to complexity is introduced, which might facilitate more differentiated discussions and interpretations of future findings. In this context, this thesis highlights that current simulation-based approaches and experimental paradigms may not capture real-life listening to a sufficient degree, and argues for a more holistic perspective on complex listening scenarios that incorporates behaviour- and experience-based dependent variables. Together, these contributions aim to support the design and interpretation of future listening experiments in complex audiovisual environments and to move towards models that are better suited to predict everyday listening. In this way, listening experience can ultimately be improved, particularly for vulnerable listener groups, such as people with hearing impairment.
Authors 1
-
Affiliation as printed
RWTH Aachen