Metalearning Approaches for Algorithm Selection I (Exploiting Rankings)

Cognitive technologies, pp. 19–37

Abstract

Summary This chapter discusses an approach to the problem of algorithm selection, which exploits the performance metadata of algorithms (workflows) on prior tasks to generate recommendations for a given target dataset. The recommendations are in the form of rankings of candidate algorithms. The methodology involves two phases. In the first one, rankings of algorithms/workflows are elaborated on the basis of historical performance data on different datasets. These are subsequently aggregated into a single ranking (e.g. average ranking). In the second phase, the average ranking is used to schedule tests on the target dataset with the objective of identifying the best performing algorithm. This approach requires that an appropriate evaluation measure, such as accuracy, is set beforehand. In this chapter we also describe a method that builds this ranking based on a combination of accuracy and runtime, yielding good anytime performance. While this approach is rather simple, it can still provide good recommendations to the user. Although the examples in this chapter are from the classification domain, this approach can be applied to other tasks besides algorithm selection, namely hyperparameter optimization (HPO), as well as the combined algorithm selection and hyperparameter optimization (CASH) problem. As this approach works with discrete data, continuous hyperparameters need to be discretized first.

Authors 4

1. Pavel Brazdil

   Universidade do Porto

   Affiliation as printed

   Laboratory of Artificial Intelligence and Decision Support, University of Porto, Porto, Portugal

2. Jan N. van Rijn Aachen Leiden Institute of Advanced Computer Science

   Leiden University

   Affiliation as printed

   Leiden Institute of Advanced Computer Science, Leiden University, Leiden, The Netherlands

3. Carlos Soares

   Universidade do Porto

   Affiliation as printed

   Porto Business School, Porto, Portugal

4. Joaquin Vanschoren

   Eindhoven University of Technology

   Affiliation as printed

   Department of Mathematics and Computer Science, Technische Universiteit Eindhoven, Eindhoven, The Netherlands