Journal
DATA MINING AND KNOWLEDGE DISCOVERY
Volume 28, Issue 5-6, Pages 1189-1221Publisher
SPRINGER
DOI: 10.1007/s10618-014-0354-1
Keywords
Crowdsourcing; Wisdom of crowds; Labeler quality estimation; Approximating the crowd; Aggregating opinions
Ask authors/readers for more resources
The problem of approximating the crowd is that of estimating the crowd's majority opinion by querying only a subset of it. Algorithms that approximate the crowd can intelligently stretch a limited budget for a crowdsourcing task. We present an algorithm, CrowdSense, that works in an online fashion where items come one at a time. CrowdSense dynamically samples subsets of the crowd based on an exploration/exploitation criterion. The algorithm produces a weighted combination of the subset's votes that approximates the crowd's opinion. We then introduce two variations of CrowdSense that make various distributional approximations to handle distinct crowd characteristics. In particular, the first algorithm makes a statistical independence approximation of the labelers for large crowds, whereas the second algorithm finds a lower bound on how often the current subcrowd agrees with the crowd's majority vote. Our experiments on CrowdSense and several baselines demonstrate that we can reliably approximate the entire crowd's vote by collecting opinions from a representative subset of the crowd.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available