Seek COVER: using a disease proxy to rapidly develop and validate a personalized risk calculator for COVID-19 outcomes in an international network

BMC Medical Research Methodology, vol. 22, pp. 35

Abstract

BACKGROUND: We investigated whether we could use influenza data to develop prediction models for COVID-19 to increase the speed at which prediction models can reliably be developed and validated early in a pandemic. We developed COVID-19 Estimated Risk (COVER) scores that quantify a patient's risk of hospital admission with pneumonia (COVER-H), hospitalization with pneumonia requiring intensive services or death (COVER-I), or fatality (COVER-F) in the 30-days following COVID-19 diagnosis using historical data from patients with influenza or flu-like symptoms and tested this in COVID-19 patients. METHODS: We analyzed a federated network of electronic medical records and administrative claims data from 14 data sources and 6 countries containing data collected on or before 4/27/2020. We used a 2-step process to develop 3 scores using historical data from patients with influenza or flu-like symptoms any time prior to 2020. The first step was to create a data-driven model using LASSO regularized logistic regression, the covariates of which were used to develop aggregate covariates for the second step where the COVER scores were developed using a smaller set of features. These 3 COVER scores were then externally validated on patients with 1) influenza or flu-like symptoms and 2) confirmed or suspected COVID-19 diagnosis across 5 databases from South Korea, Spain, and the United States. Outcomes included i) hospitalization with pneumonia, ii) hospitalization with pneumonia requiring intensive services or death, and iii) death in the 30 days after index date. RESULTS: Overall, 44,507 COVID-19 patients were included for model validation. We identified 7 predictors (history of cancer, chronic obstructive pulmonary disease, diabetes, heart disease, hypertension, hyperlipidemia, kidney disease) which combined with age and sex discriminated which patients would experience any of our three outcomes. The models achieved good performance in influenza and COVID-19 cohorts. For COVID-19 the AUC ranges were, COVER-H: 0.69-0.81, COVER-I: 0.73-0.91, and COVER-F: 0.72-0.90. Calibration varied across the validations with some of the COVID-19 validations being less well calibrated than the influenza validations. CONCLUSIONS: This research demonstrated the utility of using a proxy disease to develop a prediction model. The 3 COVER models with 9-predictors that were developed using influenza data perform well for COVID-19 patients for predicting hospitalization, intensive services, and fatality. The scores showed good discriminatory performance which transferred well to the COVID-19 population. There was some miscalibration in the COVID-19 validations, which is potentially due to the difference in symptom severity between the two diseases. A possible solution for this is to recalibrate the models in each location before use.

Authors 48

1. Ross D. Williams

   Erasmus MC · Erasmus University Rotterdam

   Affiliation as printed

   Department of Medical Informatics, Erasmus University Medical Center, Doctor Molewaterplein, 403015, GD, Rotterdam, The Netherlands

2. Aniek F. Markus

   Erasmus MC · Erasmus University Rotterdam

   Affiliation as printed

   Department of Medical Informatics, Erasmus University Medical Center, Doctor Molewaterplein, 403015, GD, Rotterdam, The Netherlands

3. Cynthia Barcelon Yang

   Erasmus MC · Erasmus University Rotterdam

   Affiliation as printed

   Department of Medical Informatics, Erasmus University Medical Center, Doctor Molewaterplein, 403015, GD, Rotterdam, The Netherlands

4. Talita Duarte‐Salles

   Institut Universitari d'Investigació en Atenció Primària Jordi Gol

   Affiliation as printed

   Fundacio Institut Universitari per a la recerca a l'Atencio Primaria de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain

5. Scott L. DuVall

   University of Utah

   Affiliation as printed

   Department of Veterans Affairs, University of Utah, Salt Lake City, UT, USA

6. Thomas Falconer

   Columbia University

   Affiliation as printed

   Department of Biomedical Informatics, Columbia University, New York, NY, USA

7. Jitendra Jonnagaddala

   UNSW Sydney

   Affiliation as printed

   School of Public Health and Community Medicine, UNSW, Sydney, Australia

8. Chungsoo Kim

   Ajou University

   Affiliation as printed

   Department of Biomedical Sciences, Ajou University Graduate School of Medicine, Suwon, Republic of Korea

9. Yeunsook Rho

   National Health Insurance Service

   Affiliation as printed

   Department of Big Data Strategy, National Health Insurance Service, Wonju, Republic of Korea

10. Andrew E. Williams

    Tufts University

    Affiliation as printed

    Tufts University School of Medicine, Institute for Clinical Research and Health Policy Studies, Boston, MA, 02111, USA

11. Amanda Alberga Machado
    Affiliation as printed

    Independent Epidemiologist, OHDSI, Rotterdam, The Netherlands

12. Min Ho An
    Affiliation as printed

    So Ahn Public Health Center, Wando County Health Center and Hospital, Wando, Republic of Korea

13. María Aragón

    Institut Universitari d'Investigació en Atenció Primària Jordi Gol

    Affiliation as printed

    Fundacio Institut Universitari per a la recerca a l'Atencio Primaria de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain

14. Carlos Morgado Areia

    University of Oxford

    Affiliation as printed

    Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK

15. Edward Burn

    University of Oxford · Institut Universitari d'Investigació en Atenció Primària Jordi Gol

    Affiliation as printed

    Centre for Statistics in Medicine, NDORMS, University of Oxford, Oxford, UK

    Fundacio Institut Universitari per a la recerca a l'Atencio Primaria de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain

16. Young Hwa Choi

    Ajou University

    Affiliation as printed

    Department of Infectious Diseases, Ajou University School of Medicine, Suwon, Republic of Korea

17. Iannis Drakos
    Affiliation as printed

    Center for Surgical Science, Koege, Denmark

18. Maria Tereza Fernandes Abrahão

    Universidade de São Paulo

    Affiliation as printed

    Faculty of Medicine, University of Sao Paulo, Sao Paulo, Brazil

19. Sergio Fernández‐Bertolín

    Institut Universitari d'Investigació en Atenció Primària Jordi Gol

    Affiliation as printed

    Fundacio Institut Universitari per a la recerca a l'Atencio Primaria de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain

20. George M. Hripcsak

    Columbia University

    Affiliation as printed

    Department of Biomedical Informatics, Columbia University, New York, NY, USA

21. Benjamin Skov Kaas‐Hansen

    University of Copenhagen · Zealand University Hospital

    Affiliation as printed

    Clinical Pharmacology Unit, Zealand University Hospital, Roskilde, Denmark

    NNF Centre for Protein Research, University of Copenhagen, Copenhagen, Denmark

22. Prasanna L. Kandukuri

    AbbVie (United States)

    Affiliation as printed

    Abbvie, Chicago, USA

23. Jan A. Kors

    Erasmus MC · Erasmus University Rotterdam

    Affiliation as printed

    Department of Medical Informatics, Erasmus University Medical Center, Doctor Molewaterplein, 403015, GD, Rotterdam, The Netherlands

24. Kristin Kostka
    Affiliation as printed

    Real World Solutions, IQVIA, Cambridge, MA, USA

25. Siaw‐Teng Liaw

    UNSW Sydney

    Affiliation as printed

    School of Public Health and Community Medicine, UNSW, Sydney, Australia

26. Kristine E. Lynch

    University of Utah

    Affiliation as printed

    Department of Veterans Affairs, University of Utah, Salt Lake City, UT, USA

27. Gerardo Machnicki
    Affiliation as printed

    Janssen Latin America, Buenos Aires, Argentina

28. Michael Edwin Matheny

    United States Department of Veterans Affairs · Vanderbilt University · Veterans Health Administration

    Affiliation as printed

    Department of Veterans Affairs, Washington D. C, USA

    Vanderbilt University, Nashville, USA

29. Daniel R. Morales

    University of Dundee

    Affiliation as printed

    Division of Population Health and Genomics, University of Dundee, Dundee, UK

30. Fredrik Nyberg

    University of Gothenburg

    Affiliation as printed

    School of Public Health and Community Medicine, Institute of Medicine, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden

31. Rae Woong Park

    Ajou University

    Affiliation as printed

    Department of Biomedical Informatics, Ajou University School of Medicine, Suwon, Republic of Korea

32. Albert Prats‐Uribe

    University of Oxford

    Affiliation as printed

    Centre for Statistics in Medicine, NDORMS, University of Oxford, Oxford, UK

33. Nicole L. Pratt

    University of South Australia

    Affiliation as printed

    Quality Use of Medicines and Pharmacy Research Centre, University of South Australia, Adelaide, Australia

34. Gowtham Rao
    Affiliation as printed

    Janssen Research & Development, Titusville, NJ, USA

35. Christian Reich
    Affiliation as printed

    Real World Solutions, IQVIA, Cambridge, MA, USA

36. Marcela Rivera
    Affiliation as printed

    Bayer Pharmaceuticals, Bayer Hispania, S.L., Barcelona, Spain

37. Tom M Seinen

    Erasmus MC · Erasmus University Rotterdam

    Affiliation as printed

    Department of Medical Informatics, Erasmus University Medical Center, Doctor Molewaterplein, 403015, GD, Rotterdam, The Netherlands

38. Azza Shoaibi
    Affiliation as printed

    Janssen Research & Development, Titusville, NJ, USA

39. Henry M. Spotnitz

    Columbia University

    Affiliation as printed

    Department of Biomedical Informatics, Columbia University, New York, NY, USA

40. Ewout Willem Steyerberg Aachen Department of Biomedical Data Sciences

    Leiden University Medical Center · Erasmus MC

    Affiliation as printed

    Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands

    Department of Public Health, Erasmus University Medical Center, Rotterdam, The Netherlands

41. Marc A. Suchard

    University of California, Los Angeles

    Affiliation as printed

    Department of Biostatistics, UCLA Fielding School of Public Health, University of California, Los Angeles, CA, USA

42. Seng Chan You

    Ajou University

    Affiliation as printed

    Department of Biomedical Informatics, Ajou University School of Medicine, Suwon, Republic of Korea

43. Lin Zhang

    The University of Melbourne · Chinese Academy of Medical Sciences & Peking Union Medical College

    Affiliation as printed

    Melbourne School of Public Health, The University of Melbourne, Melbourne, Victoria, Australia

    School of Public Health, Peking Union Medical College, Beijing, China

44. Lili Zhou

    AbbVie (United States)

    Affiliation as printed

    Abbvie, Chicago, USA

45. Patrick B. Ryan
    Affiliation as printed

    Janssen Research & Development, Titusville, NJ, USA

46. Daniel Prieto‐Alhambra

    University of Oxford

    Affiliation as printed

    Centre for Statistics in Medicine, NDORMS, University of Oxford, Oxford, UK

47. Jenna Reps
    Affiliation as printed

    Janssen Research & Development, Titusville, NJ, USA

48. Peter R. Rijnbeek corresponding

    Erasmus MC · Erasmus University Rotterdam

    Affiliation as printed

    Department of Medical Informatics, Erasmus University Medical Center, Doctor Molewaterplein, 403015, GD, Rotterdam, The Netherlands. p.rijnbeek@erasmusmc.nl

    Department of Medical Informatics, Erasmus University Medical Center, Doctor Molewaterplein, 403015, GD, Rotterdam, The Netherlands