Meta-learning for symbolic hyperparameter defaults

Genetic and Evolutionary Computation Conference Companion (GECCO Companion), pp. 151–152

Abstract

Hyperparameter optimization in machine learning (ML) deals with the problem of empirically learning an optimal algorithm configuration from data, usually formulated as a black-box optimization problem. In this work, we propose a zero-shot method to meta-learn symbolic default hyperparameter configurations that are expressed in terms of the properties of the dataset. This enables a much faster, but still data-dependent, configuration of the ML algorithm, compared to standard hyperparameter optimization approaches. In the past, symbolic and static default values have usually been obtained as hand-crafted heuristics. We propose an approach of learning such symbolic configurations as formulas of dataset properties from a large set of prior evaluations on multiple datasets by optimizing over a grammar of expressions using an evolutionary algorithm. We evaluate our method on surrogate empirical performance models as well as on real data across 6 ML algorithms on more than 100 datasets and demonstrate that our method indeed finds viable symbolic defaults.

Authors 5

1. Pieter Gijsbers
   Affiliation as printed

   University of Eindhoven, Eindhoven, Netherlands

2. Florian Pfisterer

   Ludwig-Maximilians-Universität München

   Affiliation as printed

   Ludwig-Maximilians-University, Munich, Germany

   Ludwig-Maximilians-University Munich, Germany#TAB#

3. Jan N. van Rijn Aachen

   Leiden University

   Affiliation as printed

   Leiden University, Leiden, Netherlands

   Leiden University Leiden Netherlands

4. Bernd Bischl

   Ludwig-Maximilians-Universität München

   Affiliation as printed

   Ludwig-Maximilians-University, Munich, Germany

   Ludwig-Maximilians-University Munich, Germany#TAB#

5. Joaquin Vanschoren
   Affiliation as printed

   University of Eindhoven, Eindhoven, Netherlands