Cross-modal congruency modulates evidence accumulation, not decision thresholds

Frontiers in Neuroscience, vol. 19, pp. 1513083

Abstract

Audiovisual cross-modal correspondences (CMCs) refer to the brain's inherent ability to subconsciously connect auditory and visual information. These correspondences reveal essential aspects of multisensory perception and influence behavioral performance, enhancing reaction times and accuracy. However, the impact of different types of CMCs-arising from statistical co-occurrences or shaped by semantic associations-on information processing and decision-making remains underexplored. This study utilizes the Implicit Association Test, where unisensory stimuli are sequentially presented and linked via CMCs within an experimental block by the specific response instructions (either congruent or incongruent). Behavioral data are integrated with EEG measurements through neurally informed drift-diffusion modeling to examine how neural activity across both auditory and visual trials is modulated by CMCs. Our findings reveal distinct neural components that differentiate between congruent and incongruent stimuli regardless of modality, offering new insights into the role of congruency in shaping multisensory perceptual decision-making. Two key neural stages were identified: an Early component enhancing sensory encoding in congruent trials and a Late component affecting evidence accumulation, particularly in incongruent trials. These results suggest that cross-modal congruency primarily influences the processing and accumulation of sensory information rather than altering decision thresholds.

Authors 4

1. Natálie Brožová corresponding Aachen Institute for Hearing Technology and Acoustics

   RWTH Aachen University

   Affiliation as printed

   Institute for Hearing Technology and Acoustics, RWTH Aachen University, Aachen, Germany

2. Björn M. Kampa Aachen Systems Neurophysiology Department Institute of Zoology (Biology 2)

   RWTH Aachen University

   Affiliation as printed

   Systems Neurophysiology Department, Institute of Zoology, RWTH Aachen University, Aachen, Germany

3. Christoph Kayser

   Bielefeld University

   Affiliation as printed

   Department of Cognitive Neuroscience, Universität Bielefeld, Bielefeld, Germany

4. Janina Fels Aachen Institute for Hearing Technology and Acoustics

   RWTH Aachen University

   Affiliation as printed

   Institute for Hearing Technology and Acoustics, RWTH Aachen University, Aachen, Germany