Continuous benchmarking: Keeping pace with an evolving ecosystem of models and technologies

arXiv (Cornell University)

Abstract

Drawing on ideas from continuous integration, we present concepts of an automated benchmarking pipeline for high performance applications. Customization and collaboration have been key design goals owing to the requirements of research-software development as a continuous community effort. We have extended our previous conceptual work on systematic benchmarking workflows with the functionality of user-agnostic operations as well as continuous benchmarking. This fosters reproducibility and re-use of benchmarking results to ensure sustainable technological progress. We provide software-engineering solutions to keep pace with the rapid evolution of both large-scale models and high-performance computing systems with a view towards the scientific domains of neuroscience and artificial intelligence.

Authors 10

1. Jan Vogelsang
2. Melissa Lober Aachen

   Forschungszentrum Jülich · Jülich Aachen Research Alliance · RWTH Aachen University

   Affiliation as printed

   Institute for Advanced Simulation (IAS-6) , Jülich Research Centre , Jülich , Germany

   RWTH Aachen University , Aachen , Germany

3. Catherine Mia Schöfmann
4. José Villamar Aachen

   Forschungszentrum Jülich · Jülich Aachen Research Alliance · RWTH Aachen University

   Affiliation as printed

   Institute for Advanced Simulation (IAS-6) , Jülich Research Centre , Jülich , Germany

   RWTH Aachen University , Aachen , Germany

5. D. Terhorst

   Forschungszentrum Jülich · Jülich Aachen Research Alliance

   Affiliation as printed

   Institute for Advanced Simulation (IAS-6) , Jülich Research Centre , Jülich , Germany

6. Johanna Senk

   University of Sussex · Forschungszentrum Jülich · Jülich Aachen Research Alliance

   Affiliation as printed

   Institute for Advanced Simulation (IAS-6) , Jülich Research Centre , Jülich , Germany

   Sussex AI , School of Engineering and Informatics , University of Sussex , Brighton , United Kingdom

7. Hans Ekkehard Pleßer

   Forschungszentrum Jülich · Jülich Aachen Research Alliance · Norwegian University of Life Sciences

   Affiliation as printed

   Faculty of Science and Technology , Norwegian University of Life Sciences , Ås , Norway

   Institute for Advanced Simulation (IAS-6) , Jülich Research Centre , Jülich , Germany

8. Markus Diesmann
9. Susanne Kunkel Aachen & Department of Psychiatry, Psychotherapy, and Psychosomatics Medical School

   Forschungszentrum Jülich · Universitätsklinikum Aachen · RWTH Aachen University

   Affiliation as printed

   & Department of Psychiatry, Psychotherapy, and Psychosomatics , Medical School , RWTH Aachen University , Aachen , Germany

   Neuromorphic Software Ecosystems (PGI-15) , Jülich Research Centre , Jülich , Germany

10. Anno C. Kurth

    Forschungszentrum Jülich · RIKEN Center for Brain Science · Jülich Aachen Research Alliance

    Affiliation as printed

    Current address: Hierarchical Neural Computation RIKEN ECL Research Unit , RIKEN Center for Brain Science , Wako , Japan

    Institute for Advanced Simulation (IAS-6) , Jülich Research Centre , Jülich , Germany