Learning multiple defaults for machine learning algorithms

Genetic and Evolutionary Computation Conference Companion (GECCO Companion), pp. 241–242

Abstract

Modern machine learning methods highly depend on their hyper-parameter configurations for optimal performance. A widely used approach to selecting a configuration is using default settings, often proposed along with the publication of a new algorithm. Those default values are usually chosen in an ad-hoc manner to work on a wide variety of datasets. Different automatic hyperparameter configuration algorithms which select an optimal configuration per dataset have been proposed, but despite its importance, tuning is often skipped in applications because of additional run time, complexity, and experimental design questions. Instead, the learner is often applied in its defaults. This principled approach usually improves performance but adds additional algorithmic complexity and computational costs to the training procedure. We propose and study using a set of complementary default values, learned from a large database of prior empirical results as an alternative. Selecting an appropriate configuration on a new dataset then requires only a simple, efficient, and embarrassingly parallel search over this set. To demonstrate the effectiveness and efficiency of the approach, we compare learned sets of configurations to random search and Bayesian optimization. We show that sets of defaults can improve performance while being easy to deploy in comparison to more complex methods.

Authors 5

1. Florian Pfisterer

   Ludwig-Maximilians-Universität München

   Affiliation as printed

   Ludwig-Maximilians-University, Munich, Germany

   Ludwig-Maximilians-University Munich, Germany#TAB#

2. Jan N. van Rijn Aachen

   Leiden University

   Affiliation as printed

   Leiden University, Leiden, Netherlands

   Leiden University Leiden Netherlands

3. Philipp Probst
   Affiliation as printed

   Benediktbeuern, Germany

4. Andreas Christian Müller

   Microsoft (United States)

   Affiliation as printed

   Microsoft

   (Microsoft)

5. Bernd Bischl

   Ludwig-Maximilians-Universität München

   Affiliation as printed

   Ludwig-Maximilians-University, Munich, Germany

   Ludwig-Maximilians-University Munich, Germany#TAB#