Beyond confusion matrix: learning from multiple annotators with awareness of instance features

Learning from multiple annotators aims to induce a high-quality classifier from training instances, where each of them is associated with a set of observed labels provided by multiple annotators under the impact of their varying abilities and own biases. When modeling the probability transition process from latent true labels to observed labels, most existing methods adopt class-level confusion matrices of annotators which assume that observed labels do not depend on the instance features and are just determined by the true labels. However, in practice the labeling process of annotators is impacted not only by the correlation between classes but also by the content of instances. Thus using only class-level confusion matrices to characterize the probability transition process may limit the performance that the classifier can achieve. In this work, we propose the noise transition matrix, that incorporates the impact of instance features on annotators' performance based on confusion matrices. Furthermore, we propose a simple and effective learning framework, which consists of a classifier module and a noise transition matrix module in a unified neural network architecture. Experimental results on synthetic and real datasets demonstrate the noise transition matrix is better than the confusion matrix for modeling multiple annotators and the superiority of our method in comparison with state-of-the-art methods.

Automated summary

Learning from multiple annotators is important for inducing a high-quality classifier. Existing methods often use class-level confusion matrices to model the probability transition process from true labels to observed labels, but ignore the impact of instance features on annotators' performance. In this work, we propose a noise transition matrix that incorporates this impact and a learning framework that outperforms state-of-the-art methods.