Journal
MACHINE LEARNING
Volume 112, Issue 3, Pages 1053-1075Publisher
SPRINGER
DOI: 10.1007/s10994-022-06211-x
Keywords
Answer aggregation; Learning from crowds; Crowdsourcing; Deep learning
Categories
Ask authors/readers for more resources
Learning from multiple annotators is important for inducing a high-quality classifier. Existing methods often use class-level confusion matrices to model the probability transition process from true labels to observed labels, but ignore the impact of instance features on annotators' performance. In this work, we propose a noise transition matrix that incorporates this impact and a learning framework that outperforms state-of-the-art methods.
Learning from multiple annotators aims to induce a high-quality classifier from training instances, where each of them is associated with a set of observed labels provided by multiple annotators under the impact of their varying abilities and own biases. When modeling the probability transition process from latent true labels to observed labels, most existing methods adopt class-level confusion matrices of annotators which assume that observed labels do not depend on the instance features and are just determined by the true labels. However, in practice the labeling process of annotators is impacted not only by the correlation between classes but also by the content of instances. Thus using only class-level confusion matrices to characterize the probability transition process may limit the performance that the classifier can achieve. In this work, we propose the noise transition matrix, that incorporates the impact of instance features on annotators' performance based on confusion matrices. Furthermore, we propose a simple and effective learning framework, which consists of a classifier module and a noise transition matrix module in a unified neural network architecture. Experimental results on synthetic and real datasets demonstrate the noise transition matrix is better than the confusion matrix for modeling multiple annotators and the superiority of our method in comparison with state-of-the-art methods.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available