Selecting fitted models under epistemic uncertainty using a stochastic process on quantile functions
Nature Communications, vol. 16, pp. 9393
Abstract
Fitting models to data is an important part of the practice of science. Advances in machine learning have made it possible to fit more-and more complex-models, but have also exacerbated a problem: when multiple models fit the data equally well, which one(s) should we pick? The answer depends entirely on the modelling goal. In the scientific context, the essential goal is replicability: if a model works well to describe one experiment, it should continue to do so when that experiment is replicated tomorrow, or in another laboratory. The selection criterion must therefore be robust to the variations inherent to the replication process. In this work we develop a nonparametric method for estimating uncertainty on a model's empirical risk when replications are non-stationary, thus ensuring that a model is only rejected when another is reproducibly better. We illustrate the method with two examples: one a more classical setting, where the models are structurally distinct, and a machine learning-inspired setting, where they differ only in the value of their parameters. We show how, in this context of replicability or "epistemic uncertainty", it compares favourably to existing model selection criteria, and has more satisfactory behaviour with large experimental datasets.
Authors 2
-
RWTH Aachen University · University of Ottawa
Affiliation as printed
Fakultät 1, RWTH Aachen, Physik, Aachen, Germany. a.rene@physik.rwth-aachen.de
Fakultät für Informatik, RWTH Aachen, Chair of Computational Network Science, Aachen, Germany. a.rene@physik.rwth-aachen.de
University of Ottawa, Department of Physics, Ottawa, Canada. a.rene@physik.rwth-aachen.de
-
Affiliation as printed
Center for Neural Dynamics, Ottawa, Canada
University of Ottawa, Department of Cellular and Molecular Medicine, Ottawa, Canada
University of Ottawa, Department of Physics, Ottawa, Canada
Cited by 2 stored of 2
2 results
No patents citing this paper on Lens.org (checked 2026-10-06).