WEIGHTED KAPPA FOR MULTIPLE RATERS

Five procedures to calculate the probability of weighted kappa with Multiple raters under the null hypothesis of independence arc described and compared in terms of accuracy case of use, generality, and limitations. The five procedures arc (1) exact variance, (2) resampling contingency, (3) intraclass correlation, W) randomized block, and (5) resampling block. While each procedure possesses strengths and limitations, the resampling contingency procedure is shown to be the most versatile and accurate of the five procedures, provided the number of raters is not too large. The resampling contingency procedure permits any weighting scheme, accommodates both symmetrical and asymmetrical weights, is suitable for both weighted and unweighted kappa, and makes no assumptions about either the data distribution Or the probability distribution.