Assessing the Generalizability of a Performance Predictive Model

arXiv (Cornell University)

Abstract

A key component of automated algorithm selection and configuration, which in most cases are performed using supervised machine learning (ML) methods is a good-performing predictive model. The predictive model uses the feature representation of a set of problem instances as input data and predicts the algorithm performance achieved on them. Common machine learning models struggle to make predictions for instances with feature representations not covered by the training data, resulting in poor generalization to unseen problems. In this study, we propose a workflow to estimate the generalizability of a predictive model for algorithm performance, trained on one benchmark suite to another. The workflow has been tested by training predictive models across benchmark suites and the results show that generalizability patterns in the landscape feature space are reflected in the performance space.

Authors 9

1. Ana Nikolikj

   Jožef Stefan International Postgraduate School

   Affiliation as printed

   Institute & Jožef Stefan International Postgraduate School Ljubljana , Slovenia

2. Gjorgjina Cenikj

   Jožef Stefan Institute · Jožef Stefan International Postgraduate School

   Affiliation as printed

   Jožef Stefan Institute & Jožef Stefan International Postgraduate School Ljubljana , Slovenia

3. Gordana Ispirova

   Jožef Stefan Institute

   Affiliation as printed

   Jožef Stefan Institute Ljubljana , Slovenia

4. Diederick Vermetten Aachen

   Leiden University

   Affiliation as printed

   LIACS , Leiden University The Netherlands

5. Ryan Dieter Lang

   Stellenbosch University

   Affiliation as printed

   Stellenbosch University South Africa

6. Andries Petrus Engelbrecht

   Stellenbosch University

   Affiliation as printed

   Stellenbosch University South Africa

7. Carola Doerr

   Centre National de la Recherche Scientifique · Sorbonne Université

   Affiliation as printed

   Sorbonne Université , CNRS , LIP Paris , France

8. Peter Korošec
9. Tome Eftimov

   Jožef Stefan Institute

   Affiliation as printed

   Jožef Stefan Institute Slovenia