Generic E-variables for exact sequential k -sample tests that allow for optional stopping

Journal of Statistical Planning and Inference, vol. 230, pp. 106116

Abstract

We develop E-variables for testing whether two or more data streams come from the same source or not, and more generally, whether the difference between the sources is larger than some minimal effect size. These E-variables lead to exact, nonasymptotic tests that remain safe, i.e., keep their type-I error guarantees, under flexible sampling scenarios such as optional stopping and continuation. In special cases our E-variables also have an optimal ‘growth’ property under the alternative. While the construction is generic, we illustrate it through the special case of k×2 contingency tables, i.e. k Bernoulli streams, allowing for the incorporation of different restrictions on the composite alternative. Comparison to p-value analysis in simulations and a real-world 2 × 2 contingency table example show that E-variables, through their flexibility, often allow for early stopping of data collection — thereby retaining similar power as classical methods — while also retaining the option of extending or combining data afterwards.

Authors 3

1. Rosanne Turner corresponding

   University Medical Center Utrecht · Centrum Wiskunde & Informatica

   Affiliation as printed

   CWI, part of NWO-I, Amsterdam, The Netherlands

   University Medical Center Utrecht, Brain Center, Utrecht, The Netherlands

2. Alexander Ly

   University of Amsterdam · Centrum Wiskunde & Informatica

   Affiliation as printed

   CWI, part of NWO-I, Amsterdam, The Netherlands

   University of Amsterdam, Department of Psychology, Amsterdam, The Netherlands

3. Peter Grünwald Aachen Department of Mathematics, Leiden

   Leiden University · Centrum Wiskunde & Informatica

   Affiliation as printed

   CWI, part of NWO-I, Amsterdam, The Netherlands

   Leiden University, Department of Mathematics, Leiden, The Netherlands