Validity of using an electronic medical record for assessing quality of care in an outpatient setting

Objective: We sought to evaluate the validity of retrieving data from a commercial, outpatient electronic medical record (EMR) to assess the management of pharyngitis, an important measure of quality of healthcare in pediatrics and a new measure for the Health Plan Employer Data and Information Set (HEDIS). Methods: For children ages 3-18 years, we electronically identified clinical encounters with diagnoses of pharyngitis using 3 different strategies (an EMR-based strategy, an administrative data-based strategy, and a reference strategy which used medical record review). By each strategy, we calculated the proportion of episodes of pharyngitis during 1 year for which management of pharyngitis adhered to published guidelines. Results: Among 479 total episodes of pharyngitis, 434 (91%) were from the EMR-based strategy and 281 (59%) from the administrative data-based strategy. Review of the records (the reference strategy) found that 391 of 479 (82%) were confirmed episodes of pharyngitis. A diagnostic test for group A streptococcus (GAS) was performed at 310 of 434 (71%) of episodes identified by the EMR-based strategy and at 214 of 281 (76%) of episodes by the administrative data-based strategy (P = 0.045). By the reference strategy, a diagnostic test was done in 301 of 391 (77%); more than at episodes found by the EMR-based strategy (71%, P < 0.001). Conclusions: The EMR-based strategy resulted in a statistically different proportion of episodes having diagnostic testing for GAS compared with a reference strategy. Complete evaluations to validate strategies for extracting data from electronic databases are necessary before assuming that measures of quality of care will be the same regardless of the source of data.