Rates of False Flagging Due to Statistical Artifact in CMS Evaluations of Transplant Programs: Results of a Stochastic Simulation

The recent CMS conditions of participation are based on risk-adjusted models produced by the Scientific Registry for Transplant Recipients (SRTR). The accuracy of these models in identifying poor-performing centers is unknown. In this stochastic simulation study, 1-year mortality outcomes were simulated in virtual transplant centers, and used to flag centers according to the methods used byCMS, evaluating nine overlapping 2.5-year periods of simulated data. In a simulation where all centers had the same underlying risk, 10.2% were falsely flagged at least once during the 4.5 years of simulated evaluations. The probability of false-positive flagging was lowest in low-volume centers (2.5%) and highest in high-volume centers (16.2%). In another simulation where 5% of centers were assigned twofold risk (poor-performing centers''), only 32% of poorperforming centers were correctly flagged. In a final simulation where each center was assigned a unique mortality risk, 94% of flagged centers had greater-thanmedian risk, but only 32% of flagged centers were among the 5% with highest risk. Even after disregarding known covariate limitations to the risk adjustment models, statistical noise alone leads to spurious flagging of many adequately-performing transplant centers, yet the methods used by CMS fail to flag most centers with true elevated risk.