Using Machine Learning Methods to Assess Module Performance Contribution in Modular Optimization Frameworks

Evolutionary Computation, vol. 33, pp. 485–512

Abstract

Modular algorithm frameworks not only allow for combinations never tested in manually selected algorithm portfolios, but they also provide a structured approach to assess which algorithmic ideas are crucial for the observed performance of algorithms. In this paper, we propose a methodology for analyzing the impact of the different modules on the overall performance. We consider modular frameworks for two widely used families of derivative-free, black-box optimization algorithms, the covariance matrix adaptation evolution strategy (CMA-ES) and differential evolution (DE). More specifically, we use performance data of 324 modCMA-ES and 576 modDE algorithm variants (with each variant corresponding to a specific configuration of modules) obtained on the 24 BBOB problems for six different runtime budgets in two dimensions. Our analysis of these data reveals that the impact of individual modules on overall algorithm performance varies significantly. Notably, among the examined modules, the elitism module in CMA-ES and the linear population size reduction module in DE exhibit the most significant impact on performance. Furthermore, our exploratory data analysis of problem landscape data suggests that the most relevant landscape features remain consistent regardless of the configuration of individual modules, but the influence that these features have on regression accuracy varies. In addition, we apply classifiers that exploit feature importance with respect to the trained models for performance prediction and performance data, to predict the modular configurations of CMA-ES and DE algorithm variants. The results show that the predicted configurations do not exhibit a statistically significant difference in performance compared to the true configurations, with the percentage varying depending on the setup (from 49.1% to 95.5% for modCMA and 21.7% to 77.1% for DE).

Authors 6

1. Ana Kostovska corresponding

   Jožef Stefan Institute · Jožef Stefan International Postgraduate School

   Affiliation as printed

   Department of Knowledge Technologies, Jožef Stefan Institute, Ljubljana, 1000, Slovenia ana.kostovska@ijs.si

2. Diederick Vermetten corresponding Aachen Leiden Institute for Advanced Computer Science

   Leiden University

   Affiliation as printed

   Leiden Institute for Advanced Computer Science, Leiden, The Netherlands d.l.vermetten@liacs.leidenuniv.nl

   LIACS - Leiden Institute of Advanced Computer Science [Leiden] (Niels Bohrweg 1 2333 CA Leiden - Netherlands)

3. Peter Korošec corresponding

   Jožef Stefan Institute · Jožef Stefan International Postgraduate School

   Affiliation as printed

   Computer Systems Department, Jožef Stefan Institute, Ljubljana, 1000, Slovenia peter.korosec@ijs.si

4. Sašo Džeroski corresponding

   Jožef Stefan Institute · Jožef Stefan International Postgraduate School

   Affiliation as printed

   Department of Knowledge Technologies, Jožef Stefan Institute, Ljubljana, 1000, Slovenia saso.dzeroski@ijs.si

5. Carola Doerr corresponding

   Centre National de la Recherche Scientifique · Sorbonne Université · LIP6

   Affiliation as printed

   Sorbonne Université, CNRS, LIP6, Paris, France carola.doerr@lip6.fr

6. Tome Eftimov corresponding

   Jožef Stefan Institute · Jožef Stefan International Postgraduate School

   Affiliation as printed

   Computer Systems Department, Jožef Stefan Institute, Ljubljana, 1000, Slovenia tome.eftimov@ijs.si