Beyond Neyman–Pearson: E-values enable hypothesis testing with a data-driven alpha

Proceedings of the National Academy of Sciences, vol. 121, pp. e2302098121

Abstract

-values, it is not clear how to use an extreme observation (e.g. [Formula: see text]) for getting better frequentist decisions. With e-values it is straightforward, since they provide Type-I risk control in a generalized Neyman-Pearson setting with the decision task (a general loss function) determined post hoc, after observation of the data-thereby providing a handle on "roving [Formula: see text]'s." When Type-II risks are taken into consideration, the only admissible decision rules in the post hoc setting turn out to be e-value-based. Similarly, if the loss incurred when specifying a faulty confidence interval is not fixed in advance, standard confidence intervals and distributions may fail, whereas e-confidence sets and e-posteriors still provide valid risk guarantees. Sufficiently powerful e-values have by now been developed for a range of classical testing problems. We discuss the main challenges for wider development and deployment.

Authors 1

1. Peter Grünwald corresponding Aachen Mathematical Institute

   Leiden University · Centrum Wiskunde & Informatica

   Affiliation as printed

   Machine Learning Group, National research institute for mathematics and computer science in the Netherlands (Centrum Wiskunde & Informatica), Amsterdam 1098 XG, The Netherlands

   Mathematical Institute, Leiden University, Leiden 2333 CC, The Netherlands