On Sampling-Based Training Criteria for Neural Language Modeling

Interspeech (USB), pp. 1877–1881

Abstract

As the vocabulary size of modern word-based language models becomes ever larger, many sampling-based training criteria are proposed and investigated.The essence of these sampling methods is that the softmax-related traversal over the entire vocabulary can be simplified, giving speedups compared to the baseline.A problem we notice about the current landscape of such sampling methods is the lack of a systematic comparison and some myths about preferring one over another.In this work, we consider Monte Carlo sampling, importance sampling, a novel method we call compensated partial summation, and noise contrastive estimation.Linking back to the three traditional criteria, namely mean squared error, binary cross-entropy, and crossentropy, we derive the theoretical solutions to the training problems.Contrary to some common belief, we show that all these sampling methods can perform equally well, as long as we correct for the intended class posterior probabilities.Experimental results in language modeling and automatic speech recognition on Switchboard and LibriSpeech support our claim, with all sampling-based methods showing similar perplexities and word error rates while giving the expected speedups.

Authors 5

1. Yingbo Gao Aachen Human Language Technology and Pattern Recognition Group Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   AppTek GmbH, 52062 Aachen, Germany

   Human Language Technology and Pattern Recognition Group, Computer Science Department RWTH Aachen University, 52074 Aachen, Germany

2. David Thulke Aachen Human Language Technology and Pattern Recognition Group Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   AppTek GmbH, 52062 Aachen, Germany

   Human Language Technology and Pattern Recognition Group, Computer Science Department RWTH Aachen University, 52074 Aachen, Germany

3. Alexander Gerstenberger corresponding Aachen Human Language Technology and Pattern Recognition Group Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   Human Language Technology and Pattern Recognition Group, Computer Science Department RWTH Aachen University, 52074 Aachen, Germany

4. Ralf Schlüter Aachen Human Language Technology and Pattern Recognition Group Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   AppTek GmbH, 52062 Aachen, Germany

   Human Language Technology and Pattern Recognition Group, Computer Science Department RWTH Aachen University, 52074 Aachen, Germany

5. Hermann Ney Aachen Human Language Technology and Pattern Recognition Group Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   AppTek GmbH, 52062 Aachen, Germany

   Human Language Technology and Pattern Recognition Group, Computer Science Department RWTH Aachen University, 52074 Aachen, Germany