Gwet's AC1 is not a substitute for Cohen's kappa- A comparison of basic properties

Gwet's AC1 has been proposed as an alternative to Cohen's kappa in evaluating the agreement between two binary ratings. This approach is becoming increasingly popular, and researchers have been criticized for still using Cohen's kappa. However, a rigorous discussion of properties of Gwet's AC1 is still missing. In this paper several basic properties of Gwet's AC1 are investigated and compared with those of Cohen's kappa, in particular the dependence on the prevalence of positive ratings for a given agreement rate and the behaviour in case of no association or maximal disagreement. Both approaches compare the observed agreement rate with a comparative number. Cohen's kappa uses an expected agreement rate as comparator, whereas Gwet's AC1 uses an expected disagreement rate. Consequently, for a fixed agreement rate, Gwet's AC1 increases with increasing difference of the prevalence of positive ratings from 0.5. In contrast, Cohen's kappa decreases. Gwet's AC1 can take positive and negative values in the case of no association between the two raters, whereas Cohen's kappa is 0. Due to these fundamental differences, Gwet's AC1 should not be seen as a substitute for Cohen's kappa. In particular, the verbal classification of kappa values by Landis & Koch should not be applied to Gwet's AC1.

Automated summary

Gwet's AC1 is proposed as an alternative to Cohen's kappa for evaluating agreement between binary ratings. This paper investigates the properties of Gwet's AC1 and compares it to Cohen's kappa, particularly in terms of the impact of prevalence of positive ratings and behavior in cases of no association or maximal disagreement. The two approaches have fundamental differences in how they are calculated and interpreted, making Gwet's AC1 not a substitute for Cohen's kappa.