An Extension of the Bland-Altman Plot for Analyzing the Agreement of More than Two Raters

The Bland-Altman plot is the most common method to analyze and visualize agreement between raters or methods of quantitative outcomes in health research. While very useful for studies with two raters, a limitation of the classical Bland-Altman plot is that it is specifically used for studies with two raters. We propose an extension of the Bland-Altman plot suitable for more than two raters and derive the approximate limits of agreement with 95% confidence intervals. We validated the suggested limit of agreement by a simulation study. Moreover, we offer suggestions on how to present bias, heterogeneity among raters, as well as the uncertainty of the limits of agreement. The resulting plot could be utilized to investigate and present agreement in studies with more than two raters.

Automated summary

The Bland-Altman plot is commonly used to analyze agreement in health research, but is limited to studies with two raters. A proposed extension of the plot for more than two raters has been validated and can be utilized to investigate agreement in studies with multiple raters. Suggestions for presenting bias, heterogeneity among raters, and uncertainty of limits of agreement are also offered.