Convergence guarantees for forward gradient descent in the linear regression model

Journal of Statistical Planning and Inference, vol. 233, pp. 106174

Abstract

Renewed interest in the relationship between artificial and biological neural networks motivates the study of gradient-free methods. Considering the linear regression model with random design, we theoretically analyze in this work the biologically motivated (weight-perturbed) forward gradient scheme that is based on random linear combination of the gradient. If d denotes the number of parameters and k the number of samples, we prove that the mean squared error of this method converges for k≳d2log(d) with rate d2log(d)/k. Compared to the dimension dependence d for stochastic gradient descent, an additional factor dlog(d) occurs.

Authors 2

1. Thijs Bos Aachen

   Leiden University

   Affiliation as printed

   Leiden University, The Netherlands

2. Johannes Schmidt-Hieber corresponding

   University of Twente

   Affiliation as printed

   University of Twente, The Netherlands