External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature
Circulation Cardiovascular Quality and Outcomes, vol. 14, pp. e007858
Abstract
Background: There are many clinical prediction models (CPMs) available to inform treatment decisions for patients with cardiovascular disease. However, the extent to which they have been externally tested, and how well they generally perform has not been broadly evaluated. Methods: A SCOPUS citation search was run on March 22, 2017 to identify external validations of cardiovascular CPMs in the Tufts Predictive Analytics and Comparative Effectiveness CPM Registry. We assessed the extent of external validation, performance heterogeneity across databases, and explored factors associated with model performance, including a global assessment of the clinical relatedness between the derivation and validation data. Results: We identified 2030 external validations of 1382 CPMs. Eight hundred seven (58%) of the CPMs in the Registry have never been externally validated. On average, there were 1.5 validations per CPM (range, 0–94). The median external validation area under the receiver operating characteristic curve was 0.73 (25th–75th percentile [interquartile range (IQR)], 0.66–0.79), representing a median percent decrease in discrimination of −11.1% (IQR, −32.4% to +2.7%) compared with performance on derivation data. 81% (n=1333) of validations reporting area under the receiver operating characteristic curve showed discrimination below that reported in the derivation dataset. 53% (n=983) of the validations report some measure of CPM calibration. For CPMs evaluated more than once, there was typically a large range of performance. Of 1702 validations classified by relatedness, the percent change in discrimination was −3.7% (IQR, −13.2 to 3.1) for closely related validations (n=123), −9.0 (IQR, −27.6 to 3.9) for related validations (n=862), and −17.2% (IQR, −42.3 to 0) for distantly related validations (n=717; P <0.001). Conclusions: Many published cardiovascular CPMs have never been externally validated, and for those that have, apparent performance during development is often overly optimistic. A single external validation appears insufficient to broadly understand the performance heterogeneity across different settings.
Authors 12
-
Affiliation as printed
Division of Cardiology (B.S.W., G.G.), Tufts Medical Center, Boston, MA
Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA
-
Affiliation as printed
Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA
Jason Nelson https://orcid.org/0000-0003-1113-9326 Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA
-
Affiliation as printed
Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA
Jinny G. Park https://orcid.org/0000-0001-8537-5749 Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA
-
Affiliation as printed
Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA
Hannah McGinnes Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA
-
Affiliation as printed
Division of Cardiology (B.S.W., G.G.), Tufts Medical Center, Boston, MA
Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA
Gaurav Gulati https://orcid.org/0000-0002-7701-0878 Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA. Division of Cardiology (B.S.W., G.G.), Tufts Medical Center, Boston, MA
-
Affiliation as printed
Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA
Riley Brazil https://orcid.org/0000-0003-4806-6271 Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA
-
Affiliation as printed
KU Leuven, Department of Development and Regeneration, Belgium (B.V.C.)
Ben Van Calster https://orcid.org/0000-0003-1613-7450 KU Leuven, Department of Development and Regeneration, Belgium (B.V.C.)
-
David V. Klaveren Aachen Department of Biomedical Data Sciences Tufts Medical Center MA. Department of Biomedical Data Sciences
Leiden University · Tufts Medical Center · Leiden University Medical Center
Affiliation as printed
Department of Biomedical Data Sciences (D.v.K.), Leiden University Medical Centre, Netherlands
Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA
David van Klaveren Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA. Department of Biomedical Data Sciences (D.v.K.), Leiden University Medical Centre, Netherlands
-
Affiliation as printed
Department of Neurology (E.V.), Erasmus MC University Medical Center, Rotterdam, the Netherlands
Department of Public Health (E.V., E.S.), Erasmus MC University Medical Center, Rotterdam, the Netherlands
Esmee Venema Department of Public Health (E.V., E.S.), Erasmus MC University Medical Center, Rotterdam, the Netherlands. Department of Neurology (E.V.), Erasmus MC University Medical Center, Rotterdam, the Netherlands
-
Ewout Willem Steyerberg Aachen Department of Biomedical Data Sciences Netherlands. Department of Public Health
Leiden University · Leiden University Medical Center · Erasmus MC
Affiliation as printed
Department of Biomedical Data Sciences (E.S.), Leiden University Medical Centre, Netherlands
Department of Public Health (E.V., E.S.), Erasmus MC University Medical Center, Rotterdam, the Netherlands
Ewout Steyerberg https://orcid.org/0000-0002-7787-0122 Department of Biomedical Data Sciences (E.S.), Leiden University Medical Centre, Netherlands. Department of Public Health (E.V., E.S.), Erasmus MC University Medical Center, Rotterdam, the Netherlands
-
Affiliation as printed
Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA
Jessica K. Paulus Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA
-
Affiliation as printed
Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA
David M. Kent https://orcid.org/0000-0002-9205-5070 Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA
Cited by 79 stored of 82
No patents citing this paper on Lens.org (checked 2026-10-11).