Editorial for the Special Issue on Reproducibility

Evolutionary Computation, vol. 32, pp. 1–2

Abstract

Experimental research is an essential component in the field of evolutionary computation (EC). The scientific method requires that empirical results are reproducible. Reproducibility of experiments also helps later researchers build upon the work of previous researchers. Interest in improving reproducibility in computer science and other empirical sciences has grown in recent years and there is a growing number of works analyzing current and best practices, obstacles and guidelines, effectiveness of journal policies, etc. Reproducibility issues in the context of EC have been a topic of discussion for a long time in the context of best practices for empirical research, but there are few studies analyzing reproducibility in EC research, and reproducibility studies themselves are extremely rare. There is room for improvement to attain the minimum standards for reproducibility encouraged in other scientific fields. Reproducibility goes beyond making implementation of algorithms publicly available. Challenges for reproducibility in EC research arise from the stochastic nature of the algorithms and, sometimes, the problems, which require multiple runs to analyze expected behavior and variance; sensitivity of the results to the computational environment, parameter settings, or implementation details; and the generalizability of conclusions to different instances of the same or related problems.This special issue of Evolutionary Computation on reproducibility features three exceptional papers that highlight different aspects of reproducibility and how to achieve it in practice.In “Using Decomposed Error for Reproducing Implicit Understanding of Algorithms” (10.1162/evco_a_00321), Caitlin A. Owen, Grant Dick, and Peter A. Whigham propose an error decomposition framework to improve the reproducibility of experiments in evolutionary machine learning. This framework takes into account information about bias, variance due to internal algorithmic choices, and variance due to training data, from multiple runs. The authors examine the behavior of three evolutionary machine learning approaches with this framework, which provides a fine-grained analysis on the decomposition of errors, and allow them to pinpoint mismatched expectations about algorithm behavior.In “The Importance of Being Constrained: Dealing with Infeasible Solutions in Differential Evolution and Beyond” (10.1162/evco_a_00333), Anna V. Kononova, Diederick Vermetten, Fabio Caraffini, Madalina-A. Mitran, and Daniela Zaharie argue that the strategy for dealing with infeasible solutions in constrained optimization problems has a significant impact on the reproducibility of experiments in heuristic optimization, and this impact grows with the dimensionality of the problem.In “A Practical Methodology for Reproducible Experimentation: An Application to the Double-Row Facility Layout Problem” (10.1162/evco_a_00317), Raúl Martín-Santamaría, Sergio Cavero, Alberto Herrán, Abraham Duarte, and J. Manuel Colmenar provide a methodology, and the software implementing it, for carrying out experiments with stochastic optimization methods. They illustrate the methodology on the double-row facility layout problem, reproducing previous results and ensuring that their own new results are fully reproducible.We believe that there is an ongoing cultural shift within computer science in general and within EC in particular, with both reviewers and funding agencies expecting and rewarding reproducibility efforts. The submission guidelines of Evolutionary Computation encourage authors to “ensure reproducibility.” Other journals have adopted “reproducibility boards” and “reproducibility badges.” Some conferences and journals have already gone a step further and require that experiments are reproducible by reviewers before publication. As a result of these efforts, we expect that the good practice standards in EC regarding reproducibility will improve in the next decade.

Authors 3

1. Manuel López‐Ibáñez corresponding

   University of Manchester

   Affiliation as printed

   Alliance Manchester Business School, University of Manchester, Manchester, UK manuel.lopez-ibanez@manchester.ac.uk

2. Luís Paquete corresponding

   University of Coimbra

   Affiliation as printed

   CISUC, Department of Informatics Engineering, University of Coimbra, Coimbra, Portugal paquete@dei.uc.pt

3. Mike Preuß corresponding Aachen

   Leiden University

   Affiliation as printed

   LIACS, Universiteit Leiden, Leiden, The Netherlands m.preuss@liacs.leidenuniv.nl