Metadata practices for simulation workflows

Scientific Data, vol. 12, pp. 942

Abstract

Computer simulations are an essential pillar of knowledge generation in science. Exploring, understanding, reproducing, and sharing the results of simulations relies on tracking and organizing the metadata describing the numerical experiments. The models used to understand real-world systems, and the computational machinery required to simulate them, are typically complex, and produce large amounts of heterogeneous metadata. Here, we present general practices for acquiring and handling metadata that are agnostic to software and hardware, and highly flexible for the user. These consist of two steps: 1) recording and storing raw metadata, and 2) selecting and structuring metadata. As a proof of concept, we develop the Archivist, a Python tool to help with the second step, and use it to apply our practices to distinct high-performance computing use cases from neuroscience and hydrology. Our practices and the Archivist can readily be applied to existing workflows without the need for substantial restructuring. They support sustainable numerical workflows, fostering replicability, reproducibility, data exploration, and data sharing in simulation-based research.

Authors 6

1. José Villamar corresponding Aachen

   Forschungszentrum Jülich · RWTH Aachen University

   Affiliation as printed

   Institute for Advanced Simulation (IAS-6), Jülich Research Centre, Jülich, Germany. j.villamar@fz-juelich.de

   RWTH Aachen University, Aachen, Germany. j.villamar@fz-juelich.de

2. Matthias Kelbling

   Helmholtz Centre for Environmental Research

   Affiliation as printed

   Department of Computational Hydrosystems, Helmholtz-Centre for Environmental Research, Leipzig, Germany

3. Michael Denker

   Forschungszentrum Jülich

   Affiliation as printed

   Institute for Advanced Simulation (IAS-6), Jülich Research Centre, Jülich, Germany

4. Tom Tetzlaff

   Forschungszentrum Jülich

   Affiliation as printed

   Institute for Advanced Simulation (IAS-6), Jülich Research Centre, Jülich, Germany

5. Johanna Senk

   University of Sussex · Forschungszentrum Jülich

   Affiliation as printed

   Institute for Advanced Simulation (IAS-6), Jülich Research Centre, Jülich, Germany

   Sussex AI, School of Engineering and Informatics, University of Sussex, Brighton, United Kingdom

6. Stephan Thober

   Helmholtz Centre for Environmental Research

   Affiliation as printed

   Department of Computational Hydrosystems, Helmholtz-Centre for Environmental Research, Leipzig, Germany