Enhancing and Adversarial: Improve ASR with Speaker Labels

IEEE International Conference on Acoustics Speech and Signal Processing, pp. 1–5

Abstract

ASR can be improved by multi-task learning (MTL) with domain enhancing or domain adversarial training, which are two opposite objectives with the aim to increase/decrease domain variance towards domain-aware/agnostic ASR, respectively. In this work, we study how to best apply these two opposite objectives with speaker labels to improve conformer-based ASR. We also propose a novel adaptive gradient reversal layer for stable and effective adversarial training without tuning effort. Detailed analysis and experimental verification are conducted to show the optimal positions in the ASR neural network (NN) to apply speaker enhancing and adversarial training. We also explore their combination for further improvement, achieving the same performance as i-vectors plus adversarial training. Our best speaker-based MTL achieves 7% relative improvement on the Switchboard Hub5’00 set. We also investigate the effect of such speaker-based MTL w.r.t. cleaner dataset and weaker ASR NN.

Authors 7

1. Wei Guo Zhou Aachen Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Human Language Technology and Pattern Recognition,Computer Science Department,Aachen,Germany,52074

   AppTek GmbH, Aachen, Germany

2. Haotian Wu Aachen Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Human Language Technology and Pattern Recognition,Computer Science Department,Aachen,Germany,52074

3. Jingjing Xu Aachen Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Human Language Technology and Pattern Recognition,Computer Science Department,Aachen,Germany,52074

4. Mohammad Zeineldeen Aachen Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Human Language Technology and Pattern Recognition,Computer Science Department,Aachen,Germany,52074

   AppTek GmbH, Aachen, Germany

5. Christoph Lüscher Aachen Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Human Language Technology and Pattern Recognition,Computer Science Department,Aachen,Germany,52074

   AppTek GmbH, Aachen, Germany

6. Ralf Schlüter Aachen Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Human Language Technology and Pattern Recognition,Computer Science Department,Aachen,Germany,52074

   AppTek GmbH, Aachen, Germany

7. Hermann Ney Aachen Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Human Language Technology and Pattern Recognition,Computer Science Department,Aachen,Germany,52074

   AppTek GmbH, Aachen, Germany