Robust Knowledge Distillation from RNN-T Models with Noisy Training Labels Using Full-Sum Loss

IEEE International Conference on Acoustics Speech and Signal Processing, pp. 1–5

Abstract

This work studies knowledge distillation (KD) and addresses its constraints for recurrent neural network transducer (RNN-T) models. In hard distillation, a teacher model transcribes large amounts of unlabelled speech to train a student model. Soft distillation is another popular KD method that distills the output logits of the teacher model. Due to the nature of RNN-T alignments, applying soft distillation between RNNT architectures having different posterior distributions is challenging. In addition, bad teachers having high word-error-rate (WER) reduce the efficacy of KD. We investigate how to effectively distill knowledge from variable quality ASR teachers, which has not been studied before to the best of our knowledge. We show that a sequence-level KD, full-sum distillation, outperforms other distillation methods for RNN-T models, especially for bad teachers. We also propose a variant of full-sum distillation that distills the sequence discriminative knowledge of the teacher leading to further improvement in WER. We conduct experiments on public datasets namely SpeechStew and LibriSpeech, and on in-house production data.

Authors 4

1. Mohammad Zeineldeen Aachen Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Human Language Technology and Pattern Recognition,Computer Science Department,Aachen,Germany,52074

2. Kartik Audhkhasi

   Google (United States)

   Affiliation as printed

   Google LLC,New York

   Google LLC, New York

3. Murali Karthick Baskar

   Google (United States)

   Affiliation as printed

   Google LLC,New York

   Google LLC, New York

4. Bhuvana Ramabhadran

   Google (United States)

   Affiliation as printed

   Google LLC,New York

   Google LLC, New York