Multi-objective Ranking using Bootstrap Resampling

Genetic and Evolutionary Computation Conference Companion (GECCO Companion), pp. 155–158

Abstract

Benchmarking and related competitions are widely used for assessing and comparing solver performances. However, underlying uncertainty due to the composition of the instance set and bias induced by choosing specific performance criteria is often not sufficiently addressed. Moreover, performance assessment is almost always multi-objective in nature and no objective, totally neutral approach to it exists. We build on recent work of robust ranking for single-objective solver performance assessment based on bootstrap resampling and introduce a multi-objective robust ranking extension shown to provide new and promising perspectives onto existing competition rankings.

Authors 3

1. Jeroen Rook

   University of Twente

   Affiliation as printed

   University of Twente, Enschede, Netherlands

2. Holger H. Hoos Aachen

   RWTH Aachen University · Leiden University

   Affiliation as printed

   Leiden University, Leiden, Netherlands

   RTWH Aachen University, Aachen, Germany

3. Heike Trautmann

   University of Twente · Paderborn University

   Affiliation as printed

   University of Paderborn, Paderborn, Germany

   University of Twente, Enschede, Netherlands