Natural language processing in the electronic medical record: Assessing clinician adherence to tobacco treatment guidelines

Background: Comprehensively assessing care quality with electronic medical records (EMRs) is not currently possible because much data reside in clinicians' free-text notes. Methods: We evaluated the accuracy of MediClass, an automated, rule-based classifier of the EMR that incorporates natural language processing, in assessing whether clinicians: (1) asked if the patient smoked; (2) advised them to stop; (3) assessed their readiness to quit; (4) assisted them in quitting by providing information or medications; and (5) arranged for appropriate follow-up care (i.e., the 5A's of smoking-cessation care). Design: We analyzed 125 medical records of known smokers at each of four HMOs in 2003 and 2004. One trained abstractor at each HMO manually coded all 500 records according to whether or not each of the 5A's of smoking cessation care was addressed during routine outpatient visits. Measurements: For each patient's record, we compared the presence or absence of each of the 5A's as assessed by each human coder and by MediClass. We measured the chance-corrected agreement between the human raters and MediClass using the kappa statistic. Results: For ask and assist, agreement among human coders was indistinguishable from agreement between humans and MediClass (p > 0.05). For assess and advise, the human coders agreed more with each other than they did with MediClass (p > 0.01); however, MediClass performance was sufficient to assess quality in these areas. The frequency of arrange was too low to be analyzed. Conclusions: MediClass performance appears adequate to replace human coders of the 5A's of smoking-cessation care, allowing for automated assessment of clinician adherence to one of the most important, evidence-based guidelines in preventive health care.