Recurrent Attention for the Transformer

Abstract

In this work, we conduct a comprehensive investigation on one of the centerpieces of modern machine translation systems: the encoderdecoder attention mechanism.Motivated by the concept of first-order alignments, we extend the (cross-)attention mechanism by a recurrent connection, allowing direct access to previous attention/alignment decisions.We propose several ways to include such a recurrency into the attention mechanism.Verifying their performance across different translation tasks we conclude that these extensions and dependencies are not beneficial for the translation performance of the Transformer architecture.

Authors 4

1. Jan Rosendahl Aachen Human Language Technology and Pattern Recognition Group Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   Human Language Technology and Pattern Recognition Group Computer Science Department RWTH Aachen University D-52056 Aachen , Germany

2. Christian Herold Aachen Human Language Technology and Pattern Recognition Group Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   Human Language Technology and Pattern Recognition Group Computer Science Department RWTH Aachen University D-52056 Aachen , Germany

3. Frithjof Petrick Aachen Human Language Technology and Pattern Recognition Group Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   Human Language Technology and Pattern Recognition Group Computer Science Department RWTH Aachen University D-52056 Aachen , Germany

4. Hermann Ney Aachen Human Language Technology and Pattern Recognition Group Computer Science Department

   RWTH Aachen University

   Affiliation as printed

   Human Language Technology and Pattern Recognition Group Computer Science Department RWTH Aachen University D-52056 Aachen , Germany