Lecture notes: From Gaussian processes to feature learning

arXiv (Cornell University)

Abstract

These lecture notes develop the theory of learning in deep and recurrent neuronal networks from the point of view of Bayesian inference. The aim is to enable the reader to understand typical computations found in the literature in this field. Initial chapters develop the theoretical tools, such as probabilities, moment and cumulant-generating functions, and some notions of large deviation theory, as far as they are needed to understand collective network behavior with large numbers of parameters. The main part of the notes derives the theory of Bayesian inference for deep and recurrent networks, starting with the neural network Gaussian process (lazy-learning) limit, which is subsequently extended to study feature learning from the point of view of adaptive kernels. The notes also expose the link between the adaptive kernel approach and approaches of kernel rescaling.

Authors 4

1. Moritz Helias Aachen Department of Physics Faculty 1

   Forschungszentrum Jülich · RWTH Aachen University

   Affiliation as printed

   Department of Physics , Faculty 1 , RWTH Aachen University , Aachen , Germany

   Institute for Advanced Simulation (IAS-6) , Jülich Research Centre , Jülich , Germany

2. Javed Lindner Aachen

   Forschungszentrum Jülich · RWTH Aachen University

   Affiliation as printed

   Institute for Advanced Simulation (IAS-6) , Jülich Research Centre , Jülich , Germany

   RWTH Aachen University , Aachen , Germany

3. Lars Schutzeichel Aachen

   Forschungszentrum Jülich · RWTH Aachen University

   Affiliation as printed

   Institute for Advanced Simulation (IAS-6) , Jülich Research Centre , Jülich , Germany

   RWTH Aachen University , Aachen , Germany

4. Zohar Ringel

   Hebrew University of Jerusalem

   Affiliation as printed

   The Racah Institute of Physics , The Hebrew University of Jerusalem , Jerusalem , Israel