Convergence of gradient descent for learning linear neural networks

Advances in Continuous and Discrete Models, vol. 2024

Abstract

Abstract We study the convergence properties of gradient descent for training deep linear neural networks, i.e., deep matrix factorizations, by extending a previous analysis for the related gradient flow. We show that under suitable conditions on the stepsizes gradient descent converges to a critical point of the loss function, i.e., the square loss in this article. Furthermore, we demonstrate that for almost all initializations gradient descent converges to a global minimum in the case of two layers. In the case of three or more layers, we show that gradient descent converges to a global minimum on the manifold matrices of some fixed rank, where the rank cannot be determined a priori.

Authors 3

1. Gabin Maxime Nguegnang corresponding

   Ludwig-Maximilians-Universität München

   Affiliation as printed

   Department of Mathematics, Ludwig-Maximilians-Universität München, Theresienstr. 39, 80333, München, Germany

2. Holger Rauhut

   Ludwig-Maximilians-Universität München · Munich Center for Machine Learning

   Affiliation as printed

   Department of Mathematics, Ludwig-Maximilians-Universität München, Theresienstr. 39, 80333, München, Germany

   Munich Center for Machine Learning, Ludwig-Maximilians-Universität München, München, Germany

3. Ulrich Terstiege Aachen Chair for Mathematics of Information Processing

   RWTH Aachen University

   Affiliation as printed

   Chair for Mathematics of Information Processing, RWTH Aachen University, Pontdriesch 10, 52062, Aachen, Germany