Optimally Weighted Ensembles of Regression Models: Exact Weight Optimization and Applications

arXiv (Cornell University)

Abstract

Automated model selection is often proposed to users to choose which machine learning model (or method) to apply to a given regression task. In this paper, we show that combining different regression models can yield better results than selecting a single ('best') regression model, and outline an efficient method that obtains optimally weighted convex linear combination from a heterogeneous set of regression models. More specifically, in this paper, a heuristic weight optimization, used in a preceding conference paper, is replaced by an exact optimization algorithm using convex quadratic programming. We prove convexity of the quadratic programming formulation for the straightforward formulation and for a formulation with weighted data points. The novel weight optimization is not only (more) exact but also more efficient. The methods we develop in this paper are implemented and made available via github-open source. They can be executed on commonly available hardware and offer a transparent and easy to interpret interface. The results indicate that the approach outperforms model selection methods on a range of data sets, including data sets with mixed variable type from drug discovery applications.

Authors 6

1. Patrick Echtenbruck Aachen

   Leiden University

   Affiliation as printed

   LIACS , Leiden University , Niels Bohrweg 1 ,

2. Martina Echtenbruck Aachen

   Leiden University

   Affiliation as printed

   LIACS , Leiden University , Niels Bohrweg 1 ,

3. Joost Batenburg Aachen

   Leiden University

   Affiliation as printed

   LIACS , Leiden University , Niels Bohrweg 1 ,

4. Thomas Bäck Aachen

   Leiden University

   Affiliation as printed

   LIACS , Leiden University , Niels Bohrweg 1 ,

5. Boris Naujoks

   TH Köln - University of Applied Sciences · Kreiskrankenhaus Gummersbach

   Affiliation as printed

   TH Cologne , 51643 Cologne Gummersbach , Germany

6. Michael Emmerich Aachen

   Leiden University

   Affiliation as printed

   LIACS , Leiden University , Niels Bohrweg 1 ,

   The Netherlands , + IDE+A ,