Leveraging spatial cues from cochlear implant microphones to efficiently enhance speech separation in naturalistic listening scenes

Scientific Reports, vol. 16, pp. 2255

Abstract

Despite the success of speech separation approaches for dry (non-reverb) speech mixtures, speech separation in naturalistic, spatial, and reverberant acoustic environments remains challenging. This limits the effectiveness of current speech separation methods for assistive hearing devices as well as neuroprosthetic devices such as cochlear implants (CIs). Here, we investigate whether a deep neural network model for speech separation can utilize the spatial information in naturalistic listening scenes as captured by a CI's microphones to improve separation performance. We examined the impact of latent spatial cues (inherently present in two-channel speech mixtures, but need to be learned from these mixtures), as well as pre-computed spatial cues added to the speech mixtures as auxiliary input features (inter-channel level and phase differences, ILDs and IPDs). Specifically, we introduce a two-channel version of the SuDoRM-RF speech separation model, which takes as input speech mixtures recorded with two CI microphones and shows that latent spatial cues enhance separation performance without affecting model efficiency in terms of model complexity and inference latency. Pre-computed spatial cues - especially IPDs - enhanced separation performance even more, but simultaneously reduced model efficiency. Finally, simulating a CI user's listening experience with a vocoder showed that the beneficial effect of spatial cues on DNN speech separation persists even if the separated speech streams are spectrotemporally degraded as in the output of a CI.

Authors 6

1. Feyisayo Olalere corresponding

   Radboud University Nijmegen · Donders Institute for Brain, Cognition and Behaviour

   Affiliation as printed

   Donders Institute for Brain, Cognition and Behavior, Radboud University, Nijmegen, The Netherlands. feyisayo.olalere@donders.ru.nl

   Donders Institute for Brain, Cognition and Behavior, Radboud University, Nijmegen, The Netherlands

2. Kiki van der Heijden

   Radboud University Nijmegen · Donders Institute for Brain, Cognition and Behaviour · Mortimer B. Zuckerman Mind Brain Behavior Institute · Columbia University

   Affiliation as printed

   Donders Institute for Brain, Cognition and Behavior, Radboud University, Nijmegen, The Netherlands

   Mortimer B. Zuckerman Mind, Brain, Behavior Institute, Columbia University, New York, USA

3. Hendrik Christiaan Stronks Aachen Department of Otorhinolaryngology, Leiden University Medical Centre

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Otorhinolaryngology, Leiden University Medical Centre, Leiden, The Netherlands

4. Jeroen Johannes Briaire Aachen Department of Otorhinolaryngology, Leiden University Medical Centre

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Department of Otorhinolaryngology, Leiden University Medical Centre, Leiden, The Netherlands

5. Johan H. M. Frijns Aachen Department of Otorhinolaryngology, Leiden University Medical Centre Leiden Institute for Brain and Cognition

   Leiden University · Leiden University Medical Center · Delft University of Technology

   Affiliation as printed

   Department of Bioelectronics, Delft University of Technology, Delft, The Netherlands

   Department of Otorhinolaryngology, Leiden University Medical Centre, Leiden, The Netherlands

   Leiden Institute for Brain and Cognition, Leiden, The Netherlands

6. Marcel A. J. van Gerven

   Radboud University Nijmegen · Donders Institute for Brain, Cognition and Behaviour

   Affiliation as printed

   Donders Institute for Brain, Cognition and Behavior, Radboud University, Nijmegen, The Netherlands