Safe testing

Journal of the Royal Statistical Society Series B (Statistical Methodology), vol. 86, pp. 1091–1128

Abstract

Abstract We develop the theory of hypothesis testing based on the e-value, a notion of evidence that, unlike the p-value, allows for effortlessly combining results from several studies in the common scenario where the decision to perform a new study may depend on previous outcomes. Tests based on e-values are safe, i.e. they preserve type-I error guarantees, under such optional continuation. We define growth rate optimality (GRO) as an analogue of power in an optional continuation context, and we show how to construct GRO e-variables for general testing problems with composite null and alternative, emphasizing models with nuisance parameters. GRO e-values take the form of Bayes factors with special priors. We illustrate the theory using several classic examples including a 1-sample safe t-test and the 2×2 contingency table. Sharing Fisherian, Neymanian, and Jeffreys–Bayesian interpretations, e-values may provide a methodology acceptable to adherents of all three schools.

Authors 3

1. Peter Grünwald corresponding Aachen Mathematical Institute

   Leiden University · Centrum Wiskunde & Informatica

   Affiliation as printed

   Machine Learning Group, Centrum Wiskunde & Informatica , Amsterdam ,

   Mathematical Institute, Leiden University , Leiden ,

   Centrum Wiskunde & Informatica, Science Park 123, 1098 XG Amsterdam, The Netherlands

2. Rianne de Heide

   Vrije Universiteit Amsterdam

   Affiliation as printed

   Department of Mathematics, Vrije Universiteit Amsterdam , Amsterdam ,

3. Wouter M. Koolen

   University of Twente · Centrum Wiskunde & Informatica

   Affiliation as printed

   Machine Learning Group, Centrum Wiskunde & Informatica , Amsterdam ,

   Statistics Group, University of Twente , Enschede ,

   Centrum Wiskunde & Informatica, Science Park 123, 1098 XG Amsterdam, The Netherlands