Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers
Information and Inference A Journal of the IMA, vol. 11, pp. 307–353
Abstract
Abstract We study the convergence of gradient flows related to learning deep linear neural networks (where the activation function is the identity map) from data. In this case, the composition of the network layers amounts to simply multiplying the weight matrices of all layers together, resulting in an overparameterized problem. The gradient flow with respect to these factors can be re-interpreted as a Riemannian gradient flow on the manifold of rank-$r$ matrices endowed with a suitable Riemannian metric. We show that the flow always converges to a critical point of the underlying functional. Moreover, we establish that, for almost all initializations, the flow converges to a global minimum on the manifold of rank $k$ matrices for some $k\leq r$.
Authors 4
-
African Institute for Mathematical Sciences · Stellenbosch University
Affiliation as printed
Research Centre, African Institute for Mathematical Sciences (AIMS) South Africa, Department of Mathematical Sciences, Stellenbosch University, Cape Town, Western Cape, South Africa
-
Affiliation as printed
Chair for Mathematics of Information Processing, RWTH Aachen University, Pontdriesch 10, 52062 Aachen, Germany
Chair for Mathematics of Information Processing, RWTH Aachen University, Pontdriesch 10, 52062 Aachen (Germany)
-
Affiliation as printed
Chair for Mathematics of Information Processing, RWTH Aachen University, Pontdriesch 10, 52062 Aachen, Germany
Chair for Mathematics of Information Processing, RWTH Aachen University, Pontdriesch 10, 52062 Aachen (Germany)
-
Affiliation as printed
Institute for Mathematics, RWTH Aachen University, Templergraben 55, 52062 Aachen, Germany
Cited by 29 stored of 29
No patents citing this paper on Lens.org (checked 2026-10-06).