External Validation of the International Mission for Prognosis and Analysis of Clinical Trials in Traumatic Brain Injury: Prognostic Models for Traumatic Brain Injury on the Study of the Neuroprotective Activity of Progesterone in Severe Traumatic Brain Injuries Trial

Prediction models for patients with traumatic brain injury (TBI) are important for multiple reasons, including case-mix adjustment, trial design, and benchmarking for quality-of-care evaluation. Models should be generalizable and therefore require regular external validation. We aimed to validate the International Mission for Prognosis and Analysis of Clinical Trials in TBI (IMPACT) prognostic models for moderate and severe TBI in a recent randomized controlled trial. We studied 1124 patients enrolled in the multi-center randomized placebo-controlled Study of the Neuroprotective Activity of Progesterone in Severe Traumatic Brain Injuries (SyNAPSe) trial that evaluated the efficacy of progesterone in TBI. Treatment and placebo groups were combined for analysis. We evaluated the predictive performance of the three prognostic models (core, extended, and lab) from the IMPACT study with regard to discrimination (area under the receiver operating characteristic curve [AUC]) and calibration (comparison of observed to predicted risks). Substantial differences were found in case-mix and outcome distribution between IMPACT and SyNAPSe. In line with the more homogeneous case-mix of a clinical trial, the discriminative performance was reasonable. For the core model, an AUC of 0.677 and 0.684 was obtained for 6-month mortality and unfavorable outcome, respectively. Performance was slightly better for the extended model (0.693 and 0.705) and for the lab model (0.689 and 0.711, respectively). For calibration, we found overestimation ofmortality, especially at higher risk predictions, and underestimation of unfavorable outcome, especially at lower risk predictions. This pattern of miscalibration was consistent across all three models. In a contemporary trial setting, the IMPACT models have reasonable discrimination if enrollment restrictions apply. Observed changes in outcome distribution necessitate updating of previously developed prognostic models.