Journal
JOURNAL OF PAIN AND SYMPTOM MANAGEMENT
Volume 63, Issue 6, Pages E713-E723Publisher
ELSEVIER SCIENCE INC
DOI: 10.1016/j.jpainsymman.2022.02.006
Keywords
Natural language processing; machine learning; goals of care; electronic health record; medical informatics
Funding
- National Palliative Care Research Center, the National Institutes of Health [K12 HL137940, R01 AG062441]
- Cambia Health Foundation
- National Center for Advancing Translational Sciences [UL1 TR002319]
Ask authors/readers for more resources
This study compares three natural language processing (NLP) modeling approaches for identifying documentation of goals-of-care discussions in electronic health records (EHR). The results show that NLP holds promise for identifying EHR-documented goals-of-care discussions, although the rarity of such content limits the performance. The study also identifies opportunities to optimize NLP modeling approaches.
Context. Documented goals-of-care discussions are an important quality metric for patients with serious illness. Natural language processing (NLP) is a promising approach for identifying goals-of-care discussions in the electronic health record (EHR). Objectives. To compare three NLP modeling approaches for identifying EHR documentation of goals-of-care discussions and generate hypotheses about differences in performance. Methods. We conducted a mixed-methods study to evaluate performance and misclassification for three NLP featurization approaches modeled with regularized logistic regression: bag-of-words (BOW), rule-based, and a hybrid approach. From a prospective cohort of 150 patients hospitalized with serious illness over 2018 to 2020, we collected 4391 inpatient EHR notes; 99 (2.3%) contained documented goals-of-care discussions. We used leave-one-out cross-validation to estimate performance by comparing pooled NLP predictions to human abstractors with receiver-operating-characteristic (ROC) and precision-recall (PR) analyses. We qualitatively examined a purposive sample of 70 NLP-misclassified notes using content analysis to identify linguistic features that allowed us to generate hypotheses underpinning misclassification. Results. All three modeling approaches discriminated between notes with and without goals-of-care discussions (AUC(ROC): BOW, 0.907; rule-based, 0.948; hybrid, 0.965). Precision and recall were only moderate (precision at 70% recall: BOW, 16.2%; rule-based, 50.4%; hybrid, 49.3%; AUC(PR): BOW, 0.505; rule-based, 0.579; hybrid, 0.599). Qualitative analysis revealed patterns underlying performance differences between BOW and rule-based approaches. Conclusion. NLP holds promise for identifying EHR-documented goals-of-care discussions. However, the rarity of goals-of-care content in EHR data limits performance. Our findings highlight opportunities to optimize NLP modeling approaches, and support further exploration of different NLP approaches to identify goals-of-care discussions. (C) 2022 American Academy of Hospice and Palliative Medicine. Published by Elsevier Inc. All rights reserved.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available