Abstract 4147439: Accuracy of Rule-based Natural Language Processing Models for Identification of Pulmonary Embolism

Circulation, vol. 150

Abstract

Background: Some rule-based natural language processing (NLP) tools can identify pulmonary embolism (PE) from imaging reports. The external validity of these tools remains to be determined. Methods: We evaluated data for hospitalized patients from the Mass General Brigham (MGB) Health System (2016-2021). The NLP algorithms developed by Verma et al. and Johnson et al. were implemented on radiology reports to identify PE among 1,712 patients (mean age 60.6 years, 52.3% female). Previously, these two models had demonstrated high diagnostic accuracy for PE internally within the derivation cohorts (Figure). Manual chart review by two independent physicians using pre-specified criteria was set as the reference standard. Weighted estimates were determined based on the total number of hospitalizations at MGB (n=381,642). Results: The prevalence of PE in the weighted sample was 2.0% (7,708/ 381,642). The weighted estimates indicated high sensitivity (77.7% and 88.1%) and specificity (98.0% and 97.2%) but low positive predictive values (44.3% and 39.7%) for the models by Verma et al. and Johnson et al. (Figure). The most frequent reason for false positive findings was the inability of the tools to properly detect negation (such as “no PE”) from the radiology reports. Most false negative findings were related to the patients diagnosed with PE at another hospital and transferred to MGB; subsequently, they did not have an available imaging report from the index hospitalization. Therefore, the NLP algorithms led to false negative results (Figure). Conclusions: The rule-based NLP models showed high sensitivity and specificity for PE detection. However, they resulted in low positive predictive value in external validation, contrary to the derivation cohorts, limiting their utilization. Modifying the existing NLP tools and using large language models should be considered in upcoming studies to improve the positive predictive value.

Authors 26

1. Sina Rashedi

   Brigham and Women's Hospital · Harvard University

   Affiliation as printed

   Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts, United States

2. Darsiya Krishnathasan

   Brigham and Women's Hospital · Harvard University

   Affiliation as printed

   Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts, United States

3. Candrika D. Khairani

   Brigham and Women's Hospital · Harvard University

   Affiliation as printed

   Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts, United States

4. Antoine Bejjani

   Brigham and Women's Hospital · Harvard University

   Affiliation as printed

   Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts, United States

5. Ying-Chih Lo

   Brigham and Women's Hospital · Harvard University

   Affiliation as printed

   Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States

6. Mehrdad Zarghami

   Brigham and Women's Hospital · Harvard University

   Affiliation as printed

   Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States

7. Shiwani Mahajan

   Yale University

   Affiliation as printed

   Yale School of Medicine, New Haven, Connecticut, United States

8. César Caraballo

   Yale University

   Affiliation as printed

   Yale School of Medicine, New Haven, Connecticut, United States

9. jose victor jimenez ceja

   Yale New Haven Hospital · Yale University

   Affiliation as printed

   YNHH/ Yale Center for Outcomes Research and Evaluation (CORE), New Haven, Connecticut, United States

10. David Jiménez

    Universidad de Alcalá · Centro de Investigación Biomédica en Red de Enfermedades Respiratorias

    Affiliation as printed

    Universidad de Alcalá (Instituto de Ramón y Cajal de Investigación Sanitaria), Centro de Investigación Biomédica en Red de Enfermedades Respiratorias, Madrid, Spain

11. Manuel Monreal

    Universidad Católica de Murcia

    Affiliation as printed

    Universidad Católica de Murcia, Barcelona, Spain

12. Eric Alexander Secemsky

    Beth Israel Deaconess Medical Center

    Affiliation as printed

    Beth Israel Deaconess Medical Center, Boston, Massachusetts, United States

13. Frederikus Albertus Klok Aachen

    Leiden University

    Affiliation as printed

    Leiden University, Leiden, Netherlands

14. Andetta R. Hunsaker

    Brigham and Women's Hospital · Harvard University

    Affiliation as printed

    Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts, United States

15. Ayaz Aghayev

    Brigham and Women's Hospital · Harvard University

    Affiliation as printed

    Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts, United States

16. Alfonso Muriel

    Universidad de Alcalá · Instituto Ramón y Cajal de Investigación Sanitaria · Hospital Universitario Ramón y Cajal

    Affiliation as printed

    Hospital Ramón y Cajal, and Universidad de Alcalá (IRYCIS), Madrid, Spain

17. Mohamad Anas Hussain

    Brigham and Women's Hospital · Harvard University

    Affiliation as printed

    Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States

18. Abena Appah-Sampong

    Brigham and Women's Hospital · Harvard University

    Affiliation as printed

    Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States

19. Sanjay Aneja

    Yale New Haven Hospital · Yale University

    Affiliation as printed

    YNHH/ Yale Center for Outcomes Research and Evaluation (CORE), New Haven, Connecticut, United States

20. Hamid R. Mojibian

    Yale University

    Affiliation as printed

    Yale School of Medicine, NEW HAVEN, New York, United States

21. Samuel Zachary Goldhaber

    Brigham and Women's Hospital · Harvard University

    Affiliation as printed

    Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts, United States

22. Liqin Wang

    Brigham and Women's Hospital · Harvard University

    Affiliation as printed

    Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States

23. Li Zhou

    Brigham and Women's Hospital · Harvard University

    Affiliation as printed

    Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States

24. Harlan M. Krumholz

    Yale New Haven Hospital · Yale University

    Affiliation as printed

    YNHH/ Yale Center for Outcomes Research and Evaluation (CORE), New Haven, Connecticut, United States

25. Gregory Piazza

    Brigham and Women's Hospital · Harvard University

    Affiliation as printed

    Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts, United States

26. Behnood Bikdeli

    Brigham and Women's Hospital · Harvard University

    Affiliation as printed

    Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts, United States