Enhancing BEST-RQ Pseudo-Label Quality through Online Refinement for Automatic Speech Recognition

arXiv (Cornell University)

Abstract

BEST-RQ is a simple and effective self-supervised training method for speech representation learning that performs well on automatic speech recognition (ASR) tasks. It generates pseudolabels using a fixed online quantization scheme, which simplifies training but provides weaker supervision than HuBERT-style models that iteratively refine pseudo-labels. In this work, we improve online pseudo-label generation while preserving simplicity. We propose three modifications: replacing the quantizer's linear projection with Principal Component Analysis (PCA), updating the codebook via iterative codebook refinement, and introducing an additional codebook updated via codebook distillation. We pre-train on the LibriSpeech 960-hour dataset and fine-tune using 100 hours of supervised LibriSpeech data. With all three modifications enabled, we achieve a 12% relative reduction in word error rate (WER) on the LibriSpeech test-other set, improving from 10.1% to 8.8%.

Authors 6

1. Jingjing Xu Aachen Machine Learning and Human Language Technology Group

   RWTH Aachen University

   Affiliation as printed

   Apptek GmbH , Aachen , Germany

   Machine Learning and Human Language Technology Group , RWTH Aachen University , Germany

2. Zijian Yang Aachen Machine Learning and Human Language Technology Group

   RWTH Aachen University

   Affiliation as printed

   Machine Learning and Human Language Technology Group , RWTH Aachen University , Germany

3. Mohammad Zeineldeen Aachen Machine Learning and Human Language Technology Group

   RWTH Aachen University

   Affiliation as printed

   Apptek GmbH , Aachen , Germany

   Machine Learning and Human Language Technology Group , RWTH Aachen University , Germany

4. Eugen Beck Aachen Machine Learning and Human Language Technology Group

   RWTH Aachen University

   Affiliation as printed

   Apptek GmbH , Aachen , Germany

   Machine Learning and Human Language Technology Group , RWTH Aachen University , Germany

5. R. Schlueter
6. Hermann Ney Aachen Machine Learning and Human Language Technology Group

   RWTH Aachen University

   Affiliation as printed

   Apptek GmbH , Aachen , Germany

   Machine Learning and Human Language Technology Group , RWTH Aachen University , Germany