Comparison of electronic versus manual abstraction for 2 standardized perinatal care measures

Objective Given that electronic clinical quality measures (eCQMs) are playing a central role in quality improvement applications nationwide, a stronger evidence base demonstrating their reliability is critically needed. To assess the reliability of electronic health record-extracted data elements and measure results for the Elective Delivery and Exclusive Breast Milk Feeding measures (vs manual abstraction) among a national sample of US acute care hospitals, as well as common sources of discrepancies and change over time. Materials and Methods eCQM and chart-abstracted data for the same patients were matched and compared at the data element and measure level for hospitals submitting both sources of data to The Joint Commission between 2017 and 2019. Sensitivity, specificity, and kappa statistics were used to assess reliability. Results Although eCQM denominator reliability had moderate to substantial agreement for both measures and both improved over time (Elective Delivery: kappa = 0.59 [95% confidence interval (CI), 0.58-0.61] in 2017 and 0.84 [95% CI, 083-0.85] in 2019; Exclusive Breast Milk Feeding: kappa = 0.58 [95% CI, 0.54-0.62] in 2017 and 0.70 [95% CI, 0.67-0.73] in 2019), the numerator status reliability was poor for Elective Delivery (kappa = 0.08 [95% CI, 0.03-0.12] in 2017 and 0.10 [95% CI, 0.05-0.15] in 2019) but near perfect for Exclusive Breast Milk Feeding (kappa = 0.85 [0.83, 0.87] in 2017 and 0.84 [0.83, 0.85] in 2019). The failure of the eCQM to accurately capture estimated gestational age, conditions possibly justifying elective delivery, active labor, and medical induction were the main reasons for the discrepancies. Conclusions Although eCQM denominator reliability for the Elective Delivery and Exclusive Breast Milk Feeding measures had moderate agreement when compared to medical record review, the numerator status reliability was poor for Elective Delivery, but near perfect for Exclusive Breast Milk Feeding. Improvements in eCQM data capture of some key data elements would greatly improve the reliability.

Automated summary

This study evaluated the reliability of electronic clinical quality measures (eCQMs) for Elective Delivery and Exclusive Breast Milk Feeding in US acute care hospitals. It found that while the denominator reliability had moderate to substantial agreement, the numerator status reliability varied between the two measures. The discrepancies in reliability were mainly attributed to inaccurate capture of key data elements by eCQMs.