A mixture model approach to indexing rater agreement

Raters are an important potential source of measurement error when assigning targets to categories. Therefore, psychologists have devoted considerable attention to quantifying the extent to which ratings agree with each other. Two main approaches to analysing rater agreement data can be distinguished. While the first approach focuses on the development of summary statistics that index rater agreement, the second models the association pattern among the observers' ratings. With the modelling approach three groups of models can be distinguished: latent class models, simple quasi-symmetric agreement models, and mixture models. This paper discusses a class of mixture models that is defined by its characteristic of having a quasi-symmetric log-linear representation. This class of models has two interesting properties. First, the simple quasi-symmetric agreement models can be shown to be members of this class. Therefore, the results of a rater agreement analysis based on a simple quasi-symmetric agreement model may be interpreted in the mixture model framework. Second, since the mixture models readily provide a familiar measure of rater reliability, it is possible to obtain a model-based estimate of rater reliability from the simple quasi-symmetric agreement models. The suggested class of mixture models will be illustrated using data from a persuasive communication study in which three raters classified respondents on the basis of their elicited cognitive responses.