Carefree multiple testing with e-processes

Electronic Journal of Statistics, vol. 20, pp. 3178–3189

Abstract

E-processes enable hypothesis testing with ongoing data collection while maintaining Type I error control. However, when testing multiple hypotheses simultaneously, current e-value based multiple testing methods such as e-BH are not invariant to the order in which data are gathered for the different e-processes. This can lead to undesirable situations, e.g., where a hypothesis rejected at time t is no longer rejected at time t+1 after choosing to gather more data for one or more e-processes unrelated to that hypothesis. We argue for multiple testing methods working with suprema of e-processes. We provide an example to illustrate that e-BH does not control the FDR, at level α when applied to suprema of e-processes. From the same example we see that the FWER is not controlled with averaging, and also closed e-BH does not control the FDR. We show that adjusters can be used to ensure FDR-sup control with e-BH under arbitrary dependence.

Authors 3

1. Yury Tavyrikov

   Vrije Universiteit Amsterdam

   Affiliation as printed

   Vrije Universiteit Amsterdam

2. Jelle J. Goeman Aachen

   Leiden University Medical Center

   Affiliation as printed

   Leiden University Medical Center

3. Rianne de Heide

   University of Twente · Centrum Wiskunde & Informatica

   Affiliation as printed

   University of Twente and Centrum Wiskunde & Informatica Amsterdam