Adaptive random Fourier features training stabilized by resampling with applications in image regression

Foundations of Data Science, vol. 0, pp. 0

Abstract

This paper presents an enhanced adaptive random Fourier features (ARFF) training algorithm for shallow neural networks, building upon the work introduced in 'Adaptive Random Fourier Features with Metropolis Sampling', Kammonen et al., Foundations of Data Science, 2(3):309–332, 2020. This improved method uses a particle filter-type resampling technique to stabilize the training process and reduce the sensitivity to parameter choices. The Metropolis test can also be omitted when resampling is used, reducing the number of hyperparameters by one and reducing the computational cost per iteration compared to the ARFF method. We present comprehensive numerical experiments demonstrating the efficacy of the proposed algorithm in function regression tasks as a stand-alone method and as a pretraining step before gradient-based optimization, using the Adam optimizer. Furthermore, we apply the proposed algorithm to a simple image regression problem, illustrating its utility in sampling frequencies for the random Fourier features (RFF) layer of coordinate-based multilayer perceptrons. In this context, we use the proposed algorithm to sample the parameters of the RFF layer in an automated manner.

Authors 4

1. Aku Kammonen
2. Anamika Pandey corresponding Aachen Chair of Mathematics for Uncertainty Quantification

   RWTH Aachen University

   Affiliation as printed

   Chair of Mathematics for Uncertainty Quantification, RWTH Aachen University, 52062 Aachen, Germany

3. Erik von Schwerin

   King Abdullah University of Science and Technology

   Affiliation as printed

   Computer, Electrical and Mathematical Sciences and Engineering, 4700 King Abdullah University of Science and Technology (KAUST), Thuwal 23955-6900, Kingdom of Saudi Arabia

4. Raul F. Tempone

   King Abdullah University of Science and Technology

   Affiliation as printed

   Computer, Electrical and Mathematical Sciences and Engineering, 4700 King Abdullah University of Science and Technology (KAUST), Thuwal 23955-6900, Kingdom of Saudi Arabia