The Conformer Encoder May Reverse the Time Dimension

IEEE International Conference on Acoustics Speech and Signal Processing, pp. 1–5

Abstract

We sometimes observe monotonically decreasing cross-attention weights in our Conformer-based global attention-based encoder-decoder (AED) models, negatively affecting performance compared to monotonically increasing attention weights. Further investigation shows that the Conformer encoder reverses the sequence in the time dimension. We analyze the initial behavior of the decoder cross-attention mechanism and find that it encourages the Conformer encoder self-attention to build a connection between the initial frames and all other informative frames. Furthermore, we show that, at some point in training, the self-attention module of the Conformer starts dominating the output over the preceding feed-forward module, which then only allows the reversed information to pass through. We propose methods and ideas of how this flipping can be avoided and investigate a novel method to obtain label-frame-position alignments by using the gradients of the label log probabilities w.r.t. the encoder input frames.

Authors 5

1. Robin Schmitt Aachen Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Human Language Technology and Pattern Recognition,Computer Science Department,Aachen,Germany

2. Albert Zeyer Aachen Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Human Language Technology and Pattern Recognition,Computer Science Department,Aachen,Germany

3. Mohammad Zeineldeen Aachen Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Human Language Technology and Pattern Recognition,Computer Science Department,Aachen,Germany

4. Ralf Schlüter Aachen Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Human Language Technology and Pattern Recognition,Computer Science Department,Aachen,Germany

5. Hermann Ney Aachen Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Human Language Technology and Pattern Recognition,Computer Science Department,Aachen,Germany