Safe Anytime-Valid Inference: Practical Maximally Flexible Sampling Designs for Experiments Based on e-Values

Abstract

We show how e-values simplify the design and the conduct of experiments. These e-values form anytime-valid tests and confidence intervals that preserve type I error guarantee regardless of the sample size. This allows for real-time monitoring of evidence through e-values as data are collected, permitting early termination of experiments without over-inflating the risk of making a false discovery. Early stopping not only preserves resources but also mitigates risk for participants in clinical settings. Anytime-valid tests always allow for optional continuation, that is, the extension of an experiment regardless of the motivation. For instance, if more funds become available, or if the evidence looks promising and the funding agency, a reviewer, or an editor urges the experimenter to collect more data. Analogously, unlike classical 95% (frequentist) confidence intervals, or 95% (Bayesian) credible intervals, a researcher can be assured that a 95% anytime-valid confidence interval will, with at least 95% chance, cover the true effect size regardless of how, or even if, data collection is stopped. We use the free and open-source software library safestats implemented in R to illustrate the practical benefits of this novel inference framework.

Authors 5

1. Alexander Ly

   Centrum Wiskunde & Informatica

   Affiliation as printed

   Centrum Wiskunde & Informatica,

2. Udo Boehm

   Centrum Wiskunde & Informatica

   Affiliation as printed

   Centrum Wiskunde & Informatica,

3. Peter Grünwald Aachen

   Leiden University · Centrum Wiskunde & Informatica

   Affiliation as printed

   Centrum Wiskunde & Informatica,

   Leiden University,

4. Aaditya Ramdas

   Carnegie Mellon University

   Affiliation as printed

   Carnegie Mellon,

5. Don van Ravenzwaaij

   University of Groningen

   Affiliation as printed

   University of Groningen