Faster Convergence for Transformer Fine-tuning with Line Search Methods

Abstract

Recent works have shown that line search methods greatly increase performance of traditional stochastic gradient descent methods on a variety of datasets and architectures [1], [2]. In this work we succeed in extending line search methods to the novel and highly popular Transformer architecture and dataset domains in natural language processing. More specifically, we combine the Armijo line search with the Adam optimizer and extend it by subdividing the networks architecture into sensible units and perform the line search separately on these local units. Our optimization method outperforms the traditional Adam optimizer and achieves significant performance improvements for small data sets or small training budgets, while performing equal or better for other tested cases. Our work is publicly available as a python package, which provides a hyperparameter-free pytorch optimizer that is compatible with arbitrary network architectures.

Authors 4

1. Philip Kenneweg

   Bielefeld University

   Affiliation as printed

   University Bielefeld,AG Machine Learning,Bielefeld,Germany

   AG Machine Learning, University Bielefeld, Bielefeld, Germany

2. Leonardo Galli Aachen MIP Chair

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,MIP Chair,Aachen,Germany

   MIP Chair, RWTH Aachen University, Aachen, Germany

3. Tristan Kenneweg

   Bielefeld University

   Affiliation as printed

   University Bielefeld,AG Machine Learning,Bielefeld,Germany

   AG Machine Learning, University Bielefeld, Bielefeld, Germany

4. Barbara Hammer

   Bielefeld University

   Affiliation as printed

   University Bielefeld,AG Machine Learning,Bielefeld,Germany

   AG Machine Learning, University Bielefeld, Bielefeld, Germany