Safe Testing

Information Theory and Applications Workshop (ITA), pp. 1–54

Abstract

We present a new theory of hypothesis testing. The main concept is the s-value, a notion of evidence which, unlike p-values, allows for effortlessly combining evidence from several tests, even in the common scenario where the decision to perform a new test depends on the previous test outcome: safe tests based on s-values generally preserve Type-I error guarantees under such `optional continuation'. S-values exist for completely general testing problems with composite null and alternatives. Their prime interpretation is in terms of gambling or investing, each S-value corresponding to a particular investment. Surprisingly, optimal "GROW" S-values, which lead to fastest capital growth, are fully characterized by the joint information projection (JIPr) between the set of all Bayes marginal distributions on H0and H1. Thus, optimal s-values also have an interpretation as Bayes factors, with priors given by the JIPr. We illustrate the theory using two classical testing scenarios: the one-sample t-test and the 2 × 2-contingency table. In the t-test setting, GROW S-values correspond to adopting the right Haar prior on the variance, like in Jeffreys' Bayesian t-test. However, unlike Jeffreys', the default safe t-test puts a discrete 2-point prior on the effect size, leading to better behaviour in terms of statistical power. Sharing Fisherian, Neymanian and Jeffreys-Bayesian interpretations, S-values and safe tests may provide a methodology acceptable to adherents of all three schools.

Authors 3

1. Peter Grünwald Aachen

   Leiden University · Centrum Wiskunde & Informatica

   Affiliation as printed

   Centrum Wiskunde & Informatica, Amsterdam, The Netherlands

   Leiden University, Leiden, The Netherlands

2. Rianne de Heide Aachen

   Leiden University · Centrum Wiskunde & Informatica

   Affiliation as printed

   Centrum Wiskunde & Informatica, Amsterdam, The Netherlands

   Leiden University, Leiden, The Netherlands

3. Wouter M. Koolen

   Centrum Wiskunde & Informatica

   Affiliation as printed

   Centrum Wiskunde & Informatica, Amsterdam, The Netherlands