Bringing Closure to False Discovery Rate Control: A General Principle for Multiple Testing

arXiv (Cornell University)

Abstract

We present a novel necessary and sufficient principle for multiple testing methods controlling an expected loss. This principle asserts that every such multiple testing method is a special case of a general closed testing procedure based on e-values. It generalizes the Closure Principle, known to underlie all methods controlling familywise error and tail probabilities of false discovery proportions, to a large class of error rates -- in particular to the false discovery rate (FDR). By writing existing methods as special cases of this procedure, we can achieve uniform improvements, as we demonstrate for the e-Benjamini-Hochberg and the Benjamini-Yekutieli procedures, and the self-consistent method of Su (2018). We also show that methods derived using our novel e-Closure Principle generally control their error rate not just for one rejected set, but simultaneously over many, allowing post hoc flexibility for the researcher. Moreover, because all multiple testing methods for the expected loss error metrics covered by our framework are derived from the same procedure, researchers may even choose the error metric post hoc. Under certain conditions, this flexibility even extends to post hoc choice of the nominal error rate.

Authors 6

1. Ziyu Xu

   Carnegie Mellon University

   Affiliation as printed

   Department of Statistics and Data Science , Carnegie Mellon University , USA

2. Aldo Solari

   Ca' Foscari University of Venice

   Affiliation as printed

   Department of Economics , Ca' Foscari University of Venice , Italy

3. Lasse Fischer

   University of Bremen

   Affiliation as printed

   Competence Center for Clinical Trials Bremen , University of Bremen , Germany

4. Rianne de Heide

   Centrum Wiskunde & Informatica · University of Twente

   Affiliation as printed

   Department of Applied Mathematics , University of Twente , and Centrum Wiskunde & Informatica , Amsterdam , The Netherlands

5. Aaditya Ramdas

   Carnegie Mellon University

   Affiliation as printed

   Departments of Statistics and Data Science, and Machine Learning , Carnegie Mellon University , USA

6. Jelle J. Goeman Aachen Department of Biomedical Data Sciences

   Leiden University Medical Center

   Affiliation as printed

   || Department of Biomedical Data Sciences , Leiden University Medical Center , the Netherlands