Multi-Round versus Real-Time Delphi survey approach for achieving consensus in the COHESION core outcome set: a randomised trial

BackgroundDelphi surveys are commonly used to prioritise critical outcomes in core outcome set (COS) development. This trial aims to compare a three-round (Multi-Round) Delphi (MRD) with a Real-Time Delphi (RTD) in the prioritisation of outcomes for inclusion in a COS for neonatal encephalopathy treatments and explore whether 'feedback', 'iteration', and 'initial condition' effects may occur in the two survey methods.MethodsWe recruited 269 participants (parents/caregivers, healthcare providers and researchers/academics) of which 222 were randomised to either the MRD or the RTD. We investigated the outcomes prioritised in each survey and the 'feedback', 'iteration', and 'initial condition' effects to identify differences between the two survey methods.ResultsIn the RTD, n = 92 participants (83%) fully completed the survey. In the MRD, n = 60 participants (54%) completed all three rounds. Of the 92 outcomes presented, 26 (28%) were prioritised differently between the RTD and MRD. Significantly fewer participants amended their scores when shown stakeholder responses in the RTD compared to the MRD ('feedback effect'). The 'iteration effect' analysis found most experts appeared satisfied with their initial ratings in the RTD and did not amend their scores following stakeholder response feedback. Where they did amend their scores, ratings were amended substantially, suggesting greater convergence. Variance in scores reduced with subsequent rounds of the MRD ('iteration effect'). Whilst most participants did not change their initial scores in the RTD, of those that did, later recruits tended to align their final score more closely to the group mean final score than earlier recruits (an 'initial condition' effect).ConclusionThe feedback effect differed between the two Delphi methods but the magnitude of this difference was small and likely due to the large number of observations rather than because of a meaningfully large difference. It did not appear to be advantageous to require participants to engage in three rounds of a survey due to the low change in scores. Larger drop-out through successive rounds in the MRD, together with a lesser convergence of scores and longer time to completion, indicate considerable benefits of the RTD approach.

Automated summary

This study compared the effects of a Multi-Round Delphi (MRD) and a Real-Time Delphi (RTD) in the prioritisation of outcomes for a core outcome set (COS) in neonatal encephalopathy treatments. The results showed differences in the outcomes prioritised between the two methods, with participants in the RTD method making fewer changes to their scores compared to participants in the MRD method, indicating greater convergence in the RTD method. The RTD method also had lower drop-out rates and shorter completion time, suggesting advantages over the MRD method.