Using the Empirical Attainment Function for Analyzing Single-Objective Black-Box Optimization Algorithms

IEEE Transactions on Evolutionary Computation, vol. 29, pp. 1774–1782

Abstract

A widely accepted way to assess the performance of iterative black-box optimizers is to analyze their empirical cumulative distribution function (ECDF) of predefined quality targets achieved not later than a given runtime. In this work, we consider an alternative approach, based on the empirical attainment function (EAF) and we show that the target-based ECDF is an approximation of the EAF. We argue that the EAF has several advantages over the target-based ECDF. In particular, it does not require defining a priori quality targets per function, captures performance differences more precisely, and enables the use of additional summary statistics that enrich the analysis. We also show that the average area over the convergence curves is a simpler-to-calculate, but equivalent, measure of anytime performance. To facilitate the accessibility of the EAF, we integrate a module to compute it into the IOHanalyzer platform. Finally, we illustrate the use of the EAF via synthetic examples and via the data available for the black-box optimization benchmark suite.

Authors 4

1. Manuel López‐Ibáñez

   University of Manchester

   Affiliation as printed

   Alliance Manchester Business School, University of Manchester, Manchester, U.K

   University of Manchester [Manchester] (Oxford Rd, Manchester M13 9PL - United Kingdom)

2. Diederick Vermetten Aachen Leiden Institute for Advanced Computer Science

   Leiden University

   Affiliation as printed

   Leiden Institute for Advanced Computer Science, Leiden University, Leiden, CA, The Netherlands

   LIACS - Leiden Institute of Advanced Computer Science [Leiden] (Niels Bohrweg 1 2333 CA Leiden - Netherlands)

3. Johann Dréo

   Institut Pasteur · Université Paris Cité

   Affiliation as printed

   Department of Computational Biology, Systems Biology Group, Pasteur Institute, Université Paris Cité, Bioinformatics and Biostatistics Hub, Paris, France

   Department of Computational Biology, Pasteur Institute, Université Paris Cité, Bioinformatics and Biostatistics Hub, Systems Biology Group, Paris, France

4. Carola Doerr

   Centre National de la Recherche Scientifique · Sorbonne Université · LIP6

   Affiliation as printed

   Sorbonne Université, CNRS, LIP6, Paris, France

   LIP6 (4 Place JUSSIEU 75252 PARIS CEDEX 05 - France)