Preprint: Norm Loss: An efficient yet effective regularization method for deep neural networks

arXiv (Cornell University)

Abstract

Convolutional neural network training can suffer from diverse issues like exploding or vanishing gradients, scaling-based weight space symmetry and covariant-shift. In order to address these issues, researchers develop weight regularization methods and activation normalization methods. In this work we propose a weight soft-regularization method based on the Oblique manifold. The proposed method uses a loss function which pushes each weight vector to have a norm close to one, i.e. the weight matrix is smoothly steered toward the so-called Oblique manifold. We evaluate our method on the very popular CIFAR-10, CIFAR-100 and ImageNet 2012 datasets using two state-of-the-art architectures, namely the ResNet and wide-ResNet. Our method introduces negligible computational overhead and the results show that it is competitive to the state-of-the-art and in some cases superior to it. Additionally, the results are less sensitive to hyperparameter settings such as batch size and regularization factor.

Authors 4

1. Sebastian Schmitt

   Honda (Germany)

   Affiliation as printed

   Honda Research Institute Europe GmbH Offenbach , Germany

2. Thomas Bäck Aachen

   Leiden University

   Affiliation as printed

   Leiden University Leiden Institute of Advanced Computer Science , Leiden , the Netherlands

3. Wei Chen Aachen

   Leiden University

   Affiliation as printed

   Leiden University Leiden Institute of Advanced Computer Science , Leiden , the Netherlands

4. Michael S. Lew Aachen

   Leiden University

   Affiliation as printed

   Leiden University Leiden Institute of Advanced Computer Science , Leiden , the Netherlands