Focal Sampling: SGD biased towards early important samples for efficient image classification with augmentation selection

Knowledge and Information Systems, vol. 67, pp. 11161–11191

Abstract

Abstract In deep learning, using larger training datasets usually leads to more accurate models. However, simply adding more but redundant data may be inefficient, as some training samples may be more informative than others. We propose Focal Sampling, a method that biases SGD (Stochastic Gradient Descent) towards samples that are found to be more important after a few training epochs, by sampling them more often for the rest of the training. In contrast to state-of-the-art, our approach requires less computational overhead to estimate sample importance, as it computes estimates once during training using the prediction probabilities, and does not require restarting training. In the experimental evaluation, we see that our learning technique trains faster than state-of-the-art and can achieve higher test accuracy, especially when datasets are not well balanced or when using multiple data augmentations. Lastly, results suggest that our approach has intrinsic balancing properties and that balancing datasets based on class importance, rather than by number of samples, can achieve higher test accuracy. Code is available at https://jugit.fz-juelich.de/ias-8/sgd_biased .

Authors 4

1. Alessio Quercia corresponding Aachen Department of Computer Science

   Forschungszentrum Jülich · RWTH Aachen University

   Affiliation as printed

   Department of Computer Science, RWTH Aachen University, Aachen, Germany

   Institute for Advanced Simulation, IAS-8: Data Analytics and Machine Learning, Forschungszentrum Juelich, Juelich, Germany

2. Abigail Morrison Aachen Department of Computer Science

   Forschungszentrum Jülich · RWTH Aachen University

   Affiliation as printed

   Department of Computer Science, RWTH Aachen University, Aachen, Germany

   Institute for Advanced Simulation, IAS-6: Computational and Systems Neuroscience, Forschungszentrum Juelich, Juelich, Germany

3. Hanno Scharr

   Forschungszentrum Jülich

   Affiliation as printed

   Institute for Advanced Simulation, IAS-8: Data Analytics and Machine Learning, Forschungszentrum Juelich, Juelich, Germany

4. Ira Assent

   Forschungszentrum Jülich · Aarhus University

   Affiliation as printed

   Department of Computer Science, Aarhus University, Aarhus, Denmark

   Institute for Advanced Simulation, IAS-8: Data Analytics and Machine Learning, Forschungszentrum Juelich, Juelich, Germany