Assessing Reproducibility in Evolutionary Computation

Genetic and Evolutionary Computation Conference (GECCO), pp. 13–22

Abstract

Reproducibility is an important requirement in evolutionary computation, where results largely depend on computational experiments. In practice, reproducibility relies on how algorithms, experimental protocols, and artifacts are documented and shared. Despite growing awareness, there is still limited empirical evidence on the actual reproducibility levels of published work in the field. In this paper, we study the reproducibility practices in papers published in the Evolutionary Combinatorial Optimization and Metaheuristics track of the Genetic and Evolutionary Computation Conference over a ten-year period. We introduce a structured reproducibility checklist and apply it through a systematic manual assessment of the selected corpus. In addition, we propose RECAP (REproducibility Checklist Automation Pipeline), an LLM-based system that automatically evaluates reproducibility signals from paper text and associated code repositories. Our analysis shows that papers achieve an average completeness score of 0.62, and that 36.90% of them provide additional material beyond the manuscript itself. We demonstrate that automated assessment is feasible: RECAP achieves substantial agreement with human evaluators (Cohen's κ of 0.67). Together, these results highlight persistent gaps in reproducibility reporting and suggest that automated tools can effectively support large-scale, systematic monitoring of reproducibility practices.

Authors 5

1. Francesca Da Ros

   University of Udine

   Affiliation as printed

   DPIA, University of Udine, Udine, Italy

2. Tarik Začiragić Aachen

   Leiden University

   Affiliation as printed

   LIACS, Leiden University, Leiden, Netherlands

3. Aske Plaat Aachen

   Leiden University

   Affiliation as printed

   LIACS, Leiden University, Leiden, Netherlands

4. Thomas Bäck Aachen

   Leiden University

   Affiliation as printed

   LIACS, Leiden University, Leiden, Netherlands

5. Bas van Stein Aachen

   Leiden University

   Affiliation as printed

   LIACS, Leiden University, Leiden, Netherlands