Overparameterization and generalization error: weighted trigonometric interpolation

arXiv (Cornell University)

Abstract

Motivated by surprisingly good generalization properties of learned deep neural networks in overparameterized scenarios and by the related double descent phenomenon, this paper analyzes the relation between smoothness and low generalization error in an overparameterized linear learning problem. We study a random Fourier series model, where the task is to estimate the unknown Fourier coefficients from equidistant samples. We derive exact expressions for the generalization error of both plain and weighted least squares estimators. We show precisely how a bias towards smooth interpolants, in the form of weighted trigonometric interpolation, can lead to smaller generalization error in the overparameterized regime compared to the underparameterized regime. This provides insight into the power of overparameterization, which is common in modern machine learning.

Authors 4

1. Yuege Xie

   The University of Texas at Austin

   Affiliation as printed

   Oden Institute , University of Texas at Austin , Austin TX 78712 USA

2. Hung-Hsu Chou Aachen Chair for Mathematics of Information Processing

   RWTH Aachen University

   Affiliation as printed

   Chair for Mathematics of Information Processing , RWTH Aachen University , Pontdriesch 10 , 52056 Aachen , Germany

3. Holger Rauhut Aachen Chair for Mathematics of Information Processing

   RWTH Aachen University

   Affiliation as printed

   Chair for Mathematics of Information Processing , RWTH Aachen University , Pontdriesch 10 , 52056 Aachen , Germany

4. Rachel Ward

   The University of Texas at Austin

   Affiliation as printed

   Mathematics Department , University of Texas at Austin , Austin TX 78712 USA

   Oden Institute , University of Texas at Austin , Austin TX 78712 USA