Identifying and classifying opioid-related overdoses: A validation study

Purpose The study aims to develop and validate algorithms to identify and classify opioid overdoses using claims and other coded data, and clinical text extracted from electronic health records using natural language processing (NLP). Methods Primary data were derived from Kaiser Permanente Northwest (2008-2014), an integrated health care system (similar to n > 475 000 unique individuals per year). Data included International Classification of Diseases, Ninth Revision (ICD-9) codes for nonfatal diagnoses, International Classification of Diseases, Tenth Revision (ICD-10) codes for fatal events, clinical notes, and prescription medication records. We assessed sensitivity, specificity, positive predictive value, and negative predictive value for algorithms relative to medical chart review and conducted assessments of algorithm portability in Kaiser Permanente Washington, Tennessee State Medicaid, and Optum. Results Code-based algorithm performance was excellent for opioid-related overdoses (sensitivity = 97.2%, specificity = 84.6%) and classification of heroin-involved overdoses (sensitivity = 91.8%, specificity = 99.0%). Performance was acceptable for code-based suicide/suicide attempt classifications (sensitivity = 70.7%, specificity = 90.5%); sensitivity improved with NLP (sensitivity = 78.7%, specificity = 91.0%). Performance was acceptable for the code-based substance abuse-involved classification (sensitivity = 75.3%, specificity = 79.5%); sensitivity improved with the NLP-enhanced algorithm (sensitivity = 80.5%, specificity = 76.3%). The opioid-related overdose algorithm performed well across portability assessment sites, with sensitivity greater than 96% and specificity greater than 84%. Cross-site sensitivity for heroin-involved overdose was greater than 87%, specificity greater than or equal to 99%. Conclusions Code-based algorithms developed to detect opioid-related overdoses and classify them according to heroin involvement perform well. Algorithms for classifying suicides/attempts and abuse-related opioid overdoses perform adequately for use for research, particularly given the complexity of classifying such overdoses. The NLP-enhanced algorithms for suicides/suicide attempts and abuse-related overdoses perform significantly better than code-based algorithms and are appropriate for use in settings that have data and capacity to use NLP.