Population-Level Decision Curve Analysis May Mislead the Evaluation of Prediction Model Usefulness under Subgroup Utility Heterogeneity

arXiv (Cornell University)

Abstract

Background Decision curve analysis (DCA) evaluates prediction model usefulness using net benefit (NB). Population-level NB is often interpreted as a proxy for population-level expected utility, but this assumes comparability of utility across subgroups. We aimed to characterize when subgroup utility heterogeneity may invalidate population-level DCA conclusions. Methods We compared prediction-driven treatment with default strategies in populations containing subgroups with different utility values. We derived an inconsistency region: combinations of subgroup-specific ΔNB values for which population-level NB and utility favor different strategies. We also developed a practical robustness framework. Results Opposite signs of subgroup-specific ΔNB provide a warning signal for possible inconsistency. The inconsistency region is larger when subgroup sizes are more similar and subgroup-specific \(a-c\) values are more different, where \(a-c\) is the incremental utility of a true positive relative to a false negative. When \(a-c\) differs across subgroups, population-level NB combines quantities on different implicit utility scales and may conflict with population utility. A real-world case study illustrates the problem. Conclusions Using population-level NB as a proxy for population utility implicitly assumes homogeneous \(a-c\) across subgroups. Subgroup DCA and our framework can identify and assess when utility heterogeneity may invalidate population-level conclusions.

Authors 6

1. Junfeng Wang

   Maastricht University · Department of Health

   Affiliation as printed

   . Department of Epidemiology , CAPHRI Care and Public Health Research Institute , Maastricht University , Maastricht , The Netherlands

2. Kim Zhipei Wang

   Maastricht University · Department of Health

   Affiliation as printed

   . Department of Epidemiology , CAPHRI Care and Public Health Research Institute , Maastricht University , Maastricht , The Netherlands

3. Ben Van Calster
4. Nan van Geloven Aachen Department of Biomedical Data Sciences

   Leiden University Medical Center

   Affiliation as printed

   . Department of Biomedical Data Sciences , Leiden University Medical Center , Leiden , the Netherlands

5. Ewout Willem Steyerberg

   University Medical Center Utrecht · Utrecht University

   Affiliation as printed

   . Julius Center for Health Sciences and Primary Care , University Medical Center Utrecht , Utrecht University , Utrecht , The Netherlands

6. Laure Wynants