Setting Up Configuration Spaces and Experiments

Cognitive technologies, pp. 143–168

Abstract

Summary This chapter discusses the issues relative to so-called configuration spaces that need to be set up before initiating the search for a solution. It starts by introducing some basic concepts, such as discrete and continuous subspaces. Then it discusses certain criteria that help us to determine whether the given configuration space is (or is not) adequate for the tasks at hand. One important topic which is addressed here ishyperparameter importance, as it helps us to determine which hyperparameters have a high influence on the performance and should therefore be optimized. This chapter also discusses some methods for reducing the configuration space. This is important as it can speed up the process of finding the potentially best workflow for the new task. One problem that current systems face nowadays is that the number of alternatives in a given configuration space can be so large that it is virtually impossible to gather complete metadata. This chapter discusses the issue of whether the system can still function satisfactorily even when the metadata is incomplete. The final part of this chapter discusses some strategies that can be used for gathering metadata that originated in the area of multi-armed bandits, including, for instance, SoftMax, upper confidence bound (UCB) and pricing strategies.

Authors 4

1. Pavel Brazdil corresponding

   Universidade do Porto

   Affiliation as printed

   Laboratory of Artificial Intelligence and Decision Support, University of Porto, Porto, Portugal

2. Jan N. van Rijn Aachen Leiden Institute of Advanced Computer Science

   Leiden University

   Affiliation as printed

   Leiden Institute of Advanced Computer Science, Leiden University, Leiden, The Netherlands

3. Carlos Soares

   Universidade do Porto

   Affiliation as printed

   Porto Business School, Porto, Portugal

4. Joaquin Vanschoren

   Eindhoven University of Technology

   Affiliation as printed

   Department of Mathematics and Computer Science, Technische Universiteit Eindhoven, Eindhoven, The Netherlands