Temporal Mismatch Effects in Short-term Memory of Audio-visually Presented Spoken Digits

Convention of the European Acoustics Association Forum Acusticum, pp. 5017–5020

Abstract

In everyday life, humans are confronted with stimuli from multiple sensory modalities which must be associated with, or segregated from each other to form meaningful object-based percepts.A key physical attribute of these stimuli is their spatiotemporal coincidence, especially in the auditory and visual domains.For example, maintaining the synchrony of audio and video streams is one of the main requirements for online conferencing tools.In a previous study, spoken digits (auditory input) were presented with their corresponding graphemes (visual input) at various stimulus onset asynchronies (SOAs).When SOAs were noticeable, the results indicated a tendency to ignore one of the input streams among participants.Consequently, it is assumed that the lack of temporal correspondence between these stimuli reduced the detectability of SOAs and that SOAs were easy to ignore, when detected.Here, we investigate distracting effects of SOAs on verbal serial recall capability using audio-visual speech recordings.The temporal correspondence of these stimuli is increased by the natural cooccurrence of speech and lip movements.Key questions address whether potential stimulus onset asynchrony effects occur with audio-visual speech stimuli and if these effects reflect the perceptual asymmetry of simultaneity judgments.

Authors 2

1. Cosima A. Ermert Aachen Institute for Hearing Technology and Acoustics

   RWTH Aachen University

   Affiliation as printed

   Institute for Hearing Technology and Acoustics, RWTH Aachen

   Institute for Hearing Technology and Acoustics, RWTH Aachen University, Germany

2. Janina Fels Aachen Institute for Hearing Technology and Acoustics

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University

   Institute for Hearing Technology and Acoustics, RWTH Aachen University, Germany