Risk Stratification in Primary Care: Value-Based Contributions of Provider Adjudication

Background In primary care risk stratification, automated algorithms do not consider the same factors as providers. The process of adjudication, in which providers review and adjust algorithm-derived risk scores, may improve the prediction of adverse outcomes. Objective We assessed the patient factors that influenced provider adjudication behavior and evaluated the performance of an adjudicated risk model against a commercial algorithm. Design (1) Structured interviews with primary care providers (PCP) and multivariable regression analysis and (2) receiver operating characteristic curves (ROC) with sensitivity analyses. Participants Primary care patients aged 18 years and older with an adjudicated risk score. Approach and Main Measures (1) Themes from structured interviews and discrete variables associated with provider adjudication behavior; (2) comparison of concordance statistics and sensitivities between risk models. Key Results 47,940 patients were adjudicated by PCPs in 2018. Interviews revealed that, in adjudication, providers consider disease severity, presence of self-management skills, behavioral health, and whether a risk score is actionable. Provider up-scoring from the algorithmic risk score was significantly associated with patient male sex (OR 1.24, CI 1.15-1.34), age > 65 (OR 2.55, CI 2.24-2.91), Black race (1.26, CI 1.02-1.55), polypharmacy >10 medications (OR 4.87, CI 4.27-5.56), a positive depression screen (OR 1.57, CI 1.43-1.72), and hemoglobin A1c >9 (OR 1.89, CI 1.52-2.33). Overall, the adjudicated risk model performed better than the commercial algorithm for all outcomes: ED visits (c-statistic 0.689 vs. 0.684, p < 0.01), hospital admissions (c-statistic 0.663 vs. 0.649, p < 0.01), and death (c-statistic 0.753 vs. 0.721, p < 0.01). When limited to males or seniors, the adjudicated models displayed either improved or non-inferior performance compared to the commercial model. Conclusions Provider adjudication of risk stratification improves model performance because providers have a personal understanding of their patients and are able to apply their training to clinical decision-making.

Automated summary

In primary care risk stratification, provider adjudication of algorithms improves the prediction of adverse outcomes by considering patient factors that automated algorithms may overlook. The study found that providers take into account disease severity, self-management skills, behavioral health, and actionable risk scores during adjudication. Overall, the adjudicated risk model demonstrated better performance than a commercial algorithm, particularly in predicting ED visits, hospital admissions, and death.