Statistical Methods for MRMC ROC Studies

Rationale and Objectives: In radiology, multireader, multicase (MRMC) receiver-operating characteristic studies are commonly used to evaluate the accuracy of diagnostic imaging systems. The special feature of an MRMC receiver-operating characteristic study requires that the same set of patients' images be examined by the same set of doctors. One main difficulty of analyzing MRMC data is a complicated correlation structure. Four commonly used methods are available for dealing with this complicated correlation structure. The authors conducted an extensive simulation study to assess the performance of these methods in finite sample sizes. They summarize the relative strengths and weaknesses of these methods and make recommendations on the use of these methods. Materials and Methods: A comprehensive simulation study was conducted to assess finite-sample performance of these methods with continuous data. The use of these methods for magnetic resonance imaging to predict extracapsular extension of prostate gland tumors is also illustrated. Results: The results indicate that when test outcomes are continuous, all four methods perform well for estimating the difference in areas under the curves for two diagnostic tests. On the basis of these results, it seems that any of these approaches is appropriate for analyzing an MRMC data set with continuous or pseudocontinuous data. Conclusions: The Dorfman-Berbaum-Metz method is the most practical analysis method to implement in a wide variety of scenarios. Also, in MRMC studies, radiologists should be encouraged to use the entire rating scale rather than tending toward a binary diseased or not diseased decision.