Automating Clinical Chart Review: An Open-Source Natural Language Processing Pipeline Developed on Free-Text Radiology Reports From Patients With Glioblastoma

JCO Clinical Cancer Informatics, vol. 4, pp. 25–34

Abstract

PURPOSE: The aim of this study was to develop an open-source natural language processing (NLP) pipeline for text mining of medical information from clinical reports. We also aimed to provide insight into why certain variables or reports are more suitable for clinical text mining than others. MATERIALS AND METHODS: Various NLP models were developed to extract 15 radiologic characteristics from free-text radiology reports for patients with glioblastoma. Ten-fold cross-validation was used to optimize the hyperparameter settings and estimate model performance. We examined how model performance was associated with quantitative attributes of the radiologic characteristics and reports. RESULTS: = .52). All variables labeled with a near perfect inter-rater agreement were classified with excellent performance (area under the curve > 0.95). Excellent performance could be achieved for variables with only 50 to 100 observations in the minority group and class imbalances up to a 9:1 ratio. Report-level classification accuracy was not associated with the number of words or the vocabulary size in the distinct text documents. CONCLUSION: This study provides an open-source NLP pipeline that allows for text mining of narratively written clinical reports. Small sample sizes and class imbalance should not be considered as absolute contraindications for text mining in clinical research. However, future studies should report measures of inter-rater agreement whenever ground truth is based on a consensus label and use this measure to identify clinical variables eligible for text mining.

Authors 12

1. Joeky T. Senders Aachen Department of Neurosurgery, Leiden University Medical Center

   Brigham and Women's Hospital · Harvard University · Leiden University Medical Center

   Affiliation as printed

   Computational Neuroscience Outcomes Center, Department of Neurosurgery, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA

   Department of Neurosurgery, Leiden University Medical Center, Leiden, the Netherlands

2. Logan D. Cho

   Brigham and Women's Hospital · Harvard University · Brown University

   Affiliation as printed

   Computational Neuroscience Outcomes Center, Department of Neurosurgery, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA

   Department of Neuroscience, Brown University, Providence, RI

3. Paola Calvachi

   Brigham and Women's Hospital · Harvard University

   Affiliation as printed

   Computational Neuroscience Outcomes Center, Department of Neurosurgery, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA

4. John J. McNulty

   Brigham and Women's Hospital · Harvard University · Columbia University

   Affiliation as printed

   Computational Neuroscience Outcomes Center, Department of Neurosurgery, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA

   Vagelos College of Physicians and Surgeons, Columbia University, New York, NY

5. Joanna L. Ashby

   Brigham and Women's Hospital · Harvard University

   Affiliation as printed

   Computational Neuroscience Outcomes Center, Department of Neurosurgery, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA

6. Isabelle S. Schulte

   Brigham and Women's Hospital · Harvard University

   Affiliation as printed

   Computational Neuroscience Outcomes Center, Department of Neurosurgery, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA

7. Ahmad Kareem Almekkawi

   Brigham and Women's Hospital · Harvard University

   Affiliation as printed

   Computational Neuroscience Outcomes Center, Department of Neurosurgery, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA

8. Alireza Mehrtash

   Brigham and Women's Hospital · Harvard University

   Affiliation as printed

   Department of Radiology, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA

9. William Brian Gormley

   Brigham and Women's Hospital · Harvard University

   Affiliation as printed

   Computational Neuroscience Outcomes Center, Department of Neurosurgery, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA

10. Timothy Richard Smith

    Brigham and Women's Hospital · Harvard University

    Affiliation as printed

    Computational Neuroscience Outcomes Center, Department of Neurosurgery, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA

11. Marike L. D. Broekman Aachen Department of Neurosurgery, Leiden University Medical Center

    Leiden University Medical Center · Medisch Centrum Haaglanden · Stichting Haaglanden Medisch Centrum

    Affiliation as printed

    Department of Neurosurgery, Haaglanden Medical Center, The Hague, the Netherlands

    Department of Neurosurgery, Leiden University Medical Center, Leiden, the Netherlands

12. Omar M. Arnaout

    Brigham and Women's Hospital · Harvard University

    Affiliation as printed

    Computational Neuroscience Outcomes Center, Department of Neurosurgery, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA