From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning

JuSER (Forschungszentrum Jülich)

Abstract

Feature learning in neural networks is crucial for their expressive power and inductive biases, motivating various theoretical approaches. Some approaches describe network behavior after training through a change in kernel scale from initialization, resulting in a generalization power comparable to a Gaussian process. Conversely, in other approaches training results in the adaptation of the kernel to the data, involving directional changes to the kernel. The relationship and respective strengths of these two views have so far remained unresolved. This work presents a theoretical framework of multi-scale adaptive feature learning bridging these two views. Using methods from statistical mechanics, we derive analytical expressions for network output statistics which are valid across scaling regimes and in the continuum between them. A systematic expansion of the network's probability distribution reveals that mean-field scaling requires only a saddle-point approximation, while standard scaling necessitates additional correction terms. Remarkably, we find across regimes that kernel adaptation can be reduced to an effective kernel rescaling when predicting the mean network output in the special case of a linear network. However, for linear and non-linear networks, the multi-scale adaptive approach captures directional feature learning effects, providing richer insights than what could be recovered from a rescaling of the kernel alone.

Authors 7

1. Noa Rubin

   Hebrew University of Jerusalem

   Affiliation as printed

   The Racah Institute of Physics , The He- brew University of Jerusalem , Jerusalem , Israel

2. Javed Lindner
3. David Dahmen

   Forschungszentrum Jülich

   Affiliation as printed

   Institute for Advanced Simulation (IAS-6) , Computational and Systems Neuro- science , Jülich Research Centre , Jülich , Germany

4. Inbar Seroussi

   Tel Aviv University

   Affiliation as printed

   Department of Applied Mathematics , School of Mathematical Sciences , Tel-Aviv University , Tel-Aviv , Israel

5. Zohar Ringel

   Hebrew University of Jerusalem

   Affiliation as printed

   The Racah Institute of Physics , The He- brew University of Jerusalem , Jerusalem , Israel

6. M. Krämer Aachen Institute for Theoreti- cal Particle Physics and Cosmology

   RWTH Aachen University

   Affiliation as printed

   Institute for Theoreti- cal Particle Physics and Cosmology , RWTH Aachen University , Aachen , Germany

7. Moritz Helias Aachen Department of Physics

   Forschungszentrum Jülich · RWTH Aachen University

   Affiliation as printed

   Department of Physics , RWTH Aachen University , Aachen , Germany

   Institute for Advanced Simulation (IAS-6) , Computational and Systems Neuro- science , Jülich Research Centre , Jülich , Germany