Journal
JOURNAL OF CLINICAL EPIDEMIOLOGY
Volume 68, Issue 1, Pages 25-34Publisher
ELSEVIER SCIENCE INC
DOI: 10.1016/j.jclinepi.2014.09.007
Keywords
Risk prediction model; Prognostic models; External validation; Discrimination; Area under the receiver operating characteristics curve; Derivation study
Funding
- NATIONAL HEART, LUNG, AND BLOOD INSTITUTE [R01HL126596, R01HL124233] Funding Source: NIH RePORTER
Ask authors/readers for more resources
Objectives: To evaluate how often newly developed risk prediction models undergo external validation and how well they perform in such validations. Study Design and Setting: We reviewed derivation studies of newly proposed risk models and their subsequent external validations. Study characteristics, outcome(s), and models' discriminatory performance [area under the curve, (AUC)] in derivation and validation studies were extracted. We estimated the probability of having a validation, change in discriminatory performance with more stringent external validation by overlapping or different authors compared to the derivation estimates. Results: We evaluated 127 new prediction models. Of those, for 32 models (25%), at least an external validation study was identified; in 22 models (17%), the validation had been done by entirely different authors. The probability of having an external validation by different authors within 5 years was 16%. AUC estimates significantly decreased during external validation vs. the derivation study [median AUC change: -0.05 (P < 0.001) overall; -0.04 (P = 0.009) for validation by overlapping authors; -0.05 (P < 0.001) for validation by different authors]. On external validation, AUC decreased by at least 0.03 in 19 models and never increased by at least 0.03 (P < 0.001). Conclusion: External independent validation of predictive models in different studies is uncommon. Predictive performance may worsen substantially on external validation. (C) 2015 Elsevier Inc. All rights reserved.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available