Norm Loss: An efficient yet effective regularization method for deep neural networks

Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition, pp. 8812–8818

Abstract

Convolutional neural network training can suffer from diverse issues like exploding or vanishing gradients, scaling-based weight space symmetry and covariant-shift. In order to address these issues, researchers develop weight regularization methods and activation normalization methods. In this work we propose a weight soft-regularization method based on the Oblique manifold. The proposed method uses a loss function which pushes each weight vector to have a norm close to one, i.e. the weight matrix is smoothly steered toward the so-called Oblique manifold. We evaluate our method on the very popular CIFAR-10, CIFAR-100 and ImageNet 2012 datasets using two state-of-the-art architectures, namely the ResNet and wide-ResNet. Our method introduces negligible computational overhead and the results show that it is competitive to the state-of-the-art and in some cases superior to it. Additionally, the results are less sensitive to hyperparameter settings such as batch size and regularization factor.

Authors 4

1. Sebastian Schmitt
   Affiliation as printed

   Honda Research Institute Europe GmbH, Offenbach, Germany

2. Thomas Bäck Aachen

   Leiden University

   Affiliation as printed

   Leiden University Leiden Institute of Advanced Computer Science, Leiden, the Netherlands

3. Wei Chen Aachen

   Leiden University

   Affiliation as printed

   Leiden University Leiden Institute of Advanced Computer Science, Leiden, the Netherlands

4. Michael S. Lew Aachen

   Leiden University

   Affiliation as printed

   Leiden University Leiden Institute of Advanced Computer Science, Leiden, the Netherlands