Self-Normalized Importance Sampling for Neural Language Modeling

arXiv (Cornell University)

Abstract

To mitigate the problem of having to traverse over the full vocabulary in the softmax normalization of a neural language model, sampling-based training criteria are proposed and investigated in the context of large vocabulary word-based neural language models. These training criteria typically enjoy the benefit of faster training and testing, at a cost of slightly degraded performance in terms of perplexity and almost no visible drop in word error rate. While noise contrastive estimation is one of the most popular choices, recently we show that other sampling-based criteria can also perform well, as long as an extra correction step is done, where the intended class posterior probability is recovered from the raw model outputs. In this work, we propose self-normalized importance sampling. Compared to our previous work, the criteria considered in this work are self-normalized and there is no need to further conduct a correction step. Through self-normalized language model training as well as lattice rescoring experiments, we show that our proposed self-normalized importance sampling is competitive in both research-oriented and production-oriented automatic speech recognition tasks.

Authors 6

1. Zijian Yang Aachen Human Language Technology and Pattern Recognition Group Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   Human Language Technology and Pattern Recognition Group , Computer Science Department RWTH Aachen University , 52074 Aachen , Germany

2. Yingbo Gao Aachen AppTek GmbH Human Language Technology and Pattern Recognition Group Computer Science Department

   RWTH Aachen University · FH Aachen

   Affiliation as printed

   AppTek GmbH , 52062 Aachen , Germany

   Human Language Technology and Pattern Recognition Group , Computer Science Department RWTH Aachen University , 52074 Aachen , Germany

3. Alexander Gerstenberger Aachen Human Language Technology and Pattern Recognition Group Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   Human Language Technology and Pattern Recognition Group , Computer Science Department RWTH Aachen University , 52074 Aachen , Germany

4. Jintao Jiang Aachen AppTek GmbH

   FH Aachen

   Affiliation as printed

   AppTek GmbH , 52062 Aachen , Germany

5. Ralf Schlüter
6. Hermann Ney Aachen AppTek GmbH Human Language Technology and Pattern Recognition Group Computer Science Department

   RWTH Aachen University · FH Aachen

   Affiliation as printed

   AppTek GmbH , 52062 Aachen , Germany

   Human Language Technology and Pattern Recognition Group , Computer Science Department RWTH Aachen University , 52074 Aachen , Germany