A Tutorial on Safe Anytime-Valid Inference: Practical Maximally Flexible Sampling Designs for Experiments Based on e-Values

Abstract

We demonstrate how e-values simplify both experimental design and the inference process. With e-values researchers can perform anytime-valid tests and construct confidence intervals that maintain type I error control regardless of the sample size. This enables real-time monitoring of evidence as data are collected, permitting early termination of experiments without intolerably inflating the risk of false discoveries. Early stopping not only con- serves resources, but also mitigates risk for participants in clinical settings. Anytime-valid tests allow for optional continuation, that is, the extension of an experiment, for instance if more funds become available, or even if the evidence looks promising and the funding agency, a reviewer, or an editor urges the experimenter to collect more data. Analogously, a researcher can be assured that a 95% anytime-valid confidence interval will, with at least 95% probability, cover the true effect size regardless of how, or even if, data collection is stopped. We use the free and open-source software package safestats implemented in R to illustrate the practical benefits of this novel inference framework.

Authors 5

1. Alexander Ly

   Centrum Wiskunde & Informatica

   Affiliation as printed

   Centrum Wiskunde & Informatica

2. Udo Boehm

   Centrum Wiskunde & Informatica · Tilburg University

   Affiliation as printed

   Centrum Wiskunde & Informatica

   Tilburg University

3. Peter Grünwald Aachen

   Leiden University · Centrum Wiskunde & Informatica

   Affiliation as printed

   Centrum Wiskunde & Informatica

   Leiden University

4. Aaditya Ramdas

   Carnegie Mellon University

   Affiliation as printed

   Carnegie Mellon

5. Don van Ravenzwaaij

   University of Groningen · Tilburg University

   Affiliation as printed

   Tilburg University , Department Methodology and Statistics , PO Box 90153 , 5000 LE Tilburg , The Netherlands ,

   University of Groningen