Sustainable Benchmarking Tool (Tool Paper)

Repository KITopen (Karlsruhe Institute of Technology)

Abstract

Solvers for NP-hard problems from areas such as automated reasoning or optimisation are complex systems in which many different components interact. The performance of these solvers is the result of an intricate interplay between implementation details, algorithmic concepts and heuristics. This, alongside the complexity of the problem instances to be solved, makes it challenging to assess the effect of a single idea on the overall performance of a given solver. It is therefore not only crucial, but also challenging to evaluate the performance impact of new ideas. Existing reliable evaluation methods require large sets of diverse benchmark instances and considerable amounts of computing resources. This makes empirical evaluation a bottleneck for solver development, as it is time-consuming and energy-intensive, often requiring several CPU years of computation to evaluate the impact of a single idea. In recent years, this bottleneck has led to the development of data-driven approaches that can dynamically select a smaller number of instances that provide sufficient statistical evidence to evaluate the relative performance of a given set of solvers. However, these methods are typically not easily accessible. In this work, we present a tool that implements these methods and makes them readily accessible to solver developers, thus enabling them to obtain swifter feedback on their ideas.

Authors 7

1. Dipl.-Inform. Markus Iser

   Karlsruhe Institute of Technology

2. Marie Anastacio Aachen

   RWTH Aachen University

3. Théo Matricon

   Centre National de la Recherche Scientifique · Institut national de recherche en sciences et technologies du numérique · Institut de Recherche en Informatique et Systèmes Aléatoires · Université de Rennes

4. Laurent Simon

   Centre National de la Recherche Scientifique · Université de Bordeaux · Laboratoire Bordelais de Recherche en Informatique · Institut Polytechnique de Bordeaux

5. Holger H. Hoos Aachen

   Leiden University · University of British Columbia · RWTH Aachen University

6. Alexey Ignatiev
7. Stefan Szeider