Do clinical prediction models improve concordance of treatment decisions in reproductive medicine?

Objective To assess whether the use of clinical prediction models improves concordance between gynaecologists with respect to treatment decisions in reproductive medicine. Design We constructed 16 vignettes of subfertile couples by varying fertility history, postcoital test, sperm motility, follicle-stimulating hormone level and Chlamydia antibody titre. Setting Thirty-five gynaecologists estimated three probabilities, i.e. the 1-year probability of spontaneous pregnancy, the pregnancy chance after intrauterine insemination (IUI) and the pregnancy chance after in vitro fertilisation (IVF). Subsequently they proposed therapeutic regimens for these 16 fictional couples, i.e. expectant management, IUI or IVF. Three months later, the participant gynaecologists again had to propose therapeutic regimes for the same 16 fictional cases but this time accompanied by pregnancy chances obtained from prediction models: predictions on spontaneous pregnancy, IUI and IVF. Population Thirty-five gynaecologists working in academic and nonacademic hospitals in the Netherlands. Methods Setting section. Main outcome measures The concordance between gynaecologists of probability estimates, expressed as interclass correlation coefficient (ICC) and the concordance between gynaecologists of treatment decisions, analysed by calculating Cohen's kappa (kappa). Results The gynaecologists differed widely in estimating pregnancy chances (ICC: 0.34). Furthermore, there was a huge variation in the proposed therapeutic regimens (kappa:0.21). The treatment decisions made by gynaecologists were consistent with the ranking of their probability estimates. When prediction models were used, the concordance (K) for treatment decisions increased from 0.21 to 0.38. The number of gynaecologists counselling for expectant management increased from 39 to 51%, whereas counselling for IVF dropped from 23 to 14%. Conclusion Gynaecologists differed widely in their estimation of prognosis in 16 fictional cases of subfertile couples. Their therapeutic regimens showed likewise huge variation. After confrontation with prediction models in the same 16 fictional cases, the proposed therapeutic regimens showed only slightly better concordance. Therefore a simple introduction of validated prediction models is insufficient to introduce concordant management between doctors.