Label augmented and weighted majority voting for crowdsourcing

Crowdsourcing provides an efficient way to obtain multiple noisy labels from different crowd workers for each unlabeled instance. Label integration methods are designed to infer the unknown true label of each instance from its multiple noisy label set. We argue that when the label quality is higher than random classification, the more the number of labels, the better the performance of label integration methods. However, in real-world crowd-sourcing scenarios, each instance cannot obtain enough labels for saving costs. To solve this problem, this paper proposes a novel label integration method called label augmented and weighted majority voting (LAWMV). At first, LAWMV uses the K-nearest neighbors (KNN) algorithm to find each instance's K-nearest neighbors (including itself) and merges their multiple noisy label sets to obtain its augmented multiple noisy label set. Then, the labels from different neighbors are weighted by the distances and the label similarities between each instance and its neighbors. Finally, the integrated label of each instance is inferred by weighted majority voting (MV). The experimental results on 34 simulated and two real-world crowdsourced datasets show that LAWMV significantly outperforms all the other state-of-the-art label integration methods. (C) 2022 Elsevier Inc. All rights reserved.

Automated summary

Crowdsourcing is an efficient way to obtain multiple noisy labels for each unlabeled instance. Label integration methods aim to infer the true label of each instance from its multiple noisy labels. This paper proposes a novel label integration method called LAWMA, which improves the performance by augmenting and weighting the labels.