Decision curve analysis: confidence intervals and hypothesis testing for net benefit

Diagnostic and Prognostic Research, vol. 7, pp. 11

Abstract

BACKGROUND: A number of recent papers have proposed methods to calculate confidence intervals and p values for net benefit used in decision curve analysis. These papers are sparse on the rationale for doing so. We aim to assess the relation between sampling variability, inference, and decision-analytic concepts. METHODS AND RESULTS: We review the underlying theory of decision analysis. When we are forced into a decision, we should choose the option with the highest expected utility, irrespective of p values or uncertainty. This is in some distinction to traditional hypothesis testing, where a decision such as whether to reject a given hypothesis can be postponed. Application of inference for net benefit would generally be harmful. In particular, insisting that differences in net benefit be statistically significant would dramatically change the criteria by which we consider a prediction model to be of value. We argue instead that uncertainty related to sampling variation for net benefit should be thought of in terms of the value of further research. Decision analysis tells us which decision to make for now, but we may also want to know how much confidence we should have in that decision. If we are insufficiently confident that we are right, further research is warranted. CONCLUSION: Null hypothesis testing or simple consideration of confidence intervals are of questionable value for decision curve analysis, and methods such as value of information analysis or approaches to assess the probability of benefit should be considered instead.

Authors 4

1. Andrew Julian Vickers corresponding

   Memorial Sloan Kettering Cancer Center

   Affiliation as printed

   Department of Epidemiology and Biostatistics, Memorial Sloan Kettering Cancer Center, New York, NY, USA. vickersa@mskcc.org

   Department of Epidemiology and Biostatistics, Memorial Sloan Kettering Cancer Center, New York, NY, USA

2. Ben Van Claster Aachen Department of Biomedical Data Sciences

   Leiden University Medical Center · KU Leuven

   Affiliation as printed

   Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands

   Department of Development and Regeneration, KU Leuven, Louvain, Belgium

   EPI-Centre, KU Leuven, Louvain, Belgium

3. Laure Wynants

   Maastricht University · Care and Public Health Research Institute · KU Leuven

   Affiliation as printed

   Department of Development and Regeneration, KU Leuven, Louvain, Belgium

   Department of Epidemiology, CAPHRI Care and Public Health Research Institute, Maastricht University, Maastricht, Netherlands

   EPI-Centre, KU Leuven, Louvain, Belgium

4. Ewout Willem Steyerberg Aachen Department of Biomedical Data Sciences

   Leiden University Medical Center

   Affiliation as printed

   Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands