The Shift Toward Open and Reproducible AI Research

arXiv (Cornell University)

Abstract

The reproducibility crisis has directed the AI research community toward improving documentation practices. Several studies have identified methodological issues, and in response, the most impactful venues in the field have introduced reproducibility checklists. We seek to understand whether documentation practices have changed over time by assessing all published papers at five leading AI conferences over the past decade. Seven reproducibility variables were identified, quality-assured and used to analyse 56 800 publications. Our analysis reveals that in the period 2014 to 2024, documentation practices have improved; papers sharing both code and data increased nearly sixfold, from 11% to 64% Building on empirical reproducibility rates from a prior study, we estimate - inferred from documentation practices, not direct testing - that reproducibility increased from 28% in 2014 to 64% in 2024. Improvements in documentation practices predate the introduction of reproducibility checklists, suggesting these changes reflect a broader movement toward open science rather than a direct response to formal requirements.

Authors 4

1. Kevin Coakley

   University of California San Diego · Norwegian University of Science and Technology · San Diego Supercomputer Center

   Affiliation as printed

   Department of Computer Science , Norwegian University of Science and Technology , Sem Sælands Vei 7-9 , Trondheim , 7031 , Trønderlag , Norway

   San Diego Supercomputer Center , University of California San Diego , 9500 Gilman Dr , La Jolla , 92093 , California , USA

2. Thijs Snelleman Aachen Chair of AI Methodology (AIM)

   RWTH Aachen University

   Affiliation as printed

   Chair of AI Methodology , RWTH Aachen University , Theaterstrasse 35-39 , Aachen , 52062 , North Rhine-Westphalia , Germany

3. Holger H. Hoos Aachen Chair of AI Methodology (AIM)

   RWTH Aachen University · Leiden University · University of Applied Sciences Leiden

   Affiliation as printed

   Chair of AI Methodology , RWTH Aachen University , Theaterstrasse 35-39 , Aachen , 52062 , North Rhine-Westphalia , Germany

   Leiden Institute of Advanced Computer Science , Leiden University , Einsteinweg 55 , Leiden , 2333 CC , South Holland , The Netherlands

4. Odd Erik Gundersen

   Norwegian University of Science and Technology

   Affiliation as printed

   Department of Computer Science , Norwegian University of Science and Technology , Sem Sælands Vei 7-9 , Trondheim , 7031 , Trønderlag , Norway