Towards Realistic Optimization Benchmarks: A Questionnaire on the Properties of Real-World Problems

arXiv (Cornell University)

Abstract

Benchmarks are a useful tool for empirical performance comparisons. However, one of the main shortcomings of existing benchmarks is that it remains largely unclear how they relate to real-world problems. What does an algorithm's performance on a benchmark say about its potential on a specific real-world problem? This work aims to identify properties of real-world problems through a questionnaire on real-world single-, multi-, and many-objective optimization problems. Based on initial responses, a few challenges that have to be considered in the design of realistic benchmarks can already be identified. A key point for future work is to gather more responses to the questionnaire to allow an analysis of common combinations of properties. In turn, such common combinations can then be included in improved benchmark suites. To gather more data, the reader is invited to participate in the questionnaire at: https://tinyurl.com/opt-survey

Authors 8

1. Koen van der Blom Aachen

   Leiden University

   Affiliation as printed

   Leiden University

2. Timo M. Deist

   Centrum Wiskunde & Informatica

   Affiliation as printed

   [Centrum Wiskunde & Informatica]

3. Tea Tušar

   Jožef Stefan Institute

   Affiliation as printed

   Jozef Stefan Inst., Ljubljana, Slovenia#TAB#

4. Mariapia Marchi
   Affiliation as printed

   ESTECO SpA, Trieste, Italy

5. Yusuke Nojima

   Osaka Prefecture University

   Affiliation as printed

   Osaka Prefecture University

6. Akira Oyama

   Japan Aerospace Exploration Agency

   Affiliation as printed

   Japan Aerospace Exploration Agency

7. Vanessa Volz
   Affiliation as printed

   modl.ai, Copenhagen, Denmark

8. Boris Naujoks

   TH Köln - University of Applied Sciences

   Affiliation as printed

   TH Köln, Gummersbach, Germany