Towards verifiable Benchmarks for Reinforcement Learning

IEEE Conference on Games (CoG), vol. 10, pp. 159–166

Abstract

Reinforcement Learning (RL) is one of the most dynamic research areas in Game AI and AI as a whole, and a wide variety of games are used as its prominent test problems. However, it is subject to the replicability crisis that currently affects most algorithmic AI research. Benchmarking in Reinforcement Learning could be improved through verifiable results. There are numerous benchmark environments whose scores are used to compare different algorithms, such as Atari. Nevertheless, reviewers must trust that figures represent truthful values, as it is difficult to reproduce an exact training curve. We propose improving this situation by providing access to the original evaluation data to validate study results. To that end, we rely on the concept of replay traces. These allow re-simulation of action sequences in deterministic RL environments and, in turn, enable reviewers to verify, re-use, and manually inspect evaluation results without needing large compute clusters. It also permits validation of presented reward graphs, an inspection of individual episodes, and re-use of result data (baselines) for proper comparison in follow-up papers. We offer plug-and-play code that works with Gym so that our measures fit well in the existing RL and reproducibility eco-system. Our approach is freely available, easy to use, and adds minimal overhead, as replay traces allow a data compression ratio of up to $\approx 10^{4}$: 1 (94 GB to 8 MB for Atari Pong) compared to a regular MDP trace used in offline RL datasets. The paper presents proof-of-concept results for a variety of games.

Authors 3

1. Matthias Müller-Brockhausen Aachen Leiden Institute of Advanced Computer Science

   Leiden University

   Affiliation as printed

   Leiden Institute of Advanced Computer Science (LIACS), Leiden University,The Netherlands

   Leiden Institute of Advanced Computer Science (LIACS), Leiden University, The Netherlands

2. Aske Plaat Aachen Leiden Institute of Advanced Computer Science

   Leiden University

   Affiliation as printed

   Leiden Institute of Advanced Computer Science (LIACS), Leiden University,The Netherlands

   Leiden Institute of Advanced Computer Science (LIACS), Leiden University, The Netherlands

3. Mike Preuß Aachen Leiden Institute of Advanced Computer Science

   Leiden University

   Affiliation as printed

   Leiden Institute of Advanced Computer Science (LIACS), Leiden University,The Netherlands

   Leiden Institute of Advanced Computer Science (LIACS), Leiden University, The Netherlands