Estimation of the qualification and behavior of a contributor and aggregation of his answers in a crowdsourcing context

Crowdsourcing is the outsourcing of tasks to a crowd of contributors on a dedicated platform. The crowd on these platforms is very diversified and includes various profiles of contributors which generates data of uneven quality. However, majority voting, which is the aggregating method commonly used in platforms, gives equal weight to each contribution. To overcome this problem, we propose a method, MONITOR, which estimates the contributor's profile and aggregates the collected data by taking into account their possible imperfections thanks to the theory of belief functions. To do so, MONITOR starts by estimating the profile of the contributor through his qualification for the task and his behavior. Crowdsourcing campaigns have been carried out to collect the necessary data to test MONITOR on real data in order to compare it to existing approaches. The results of the experiments show that thanks to the use of the MONITOR method, we obtain a better rate of correct answer after aggregation of the contributions compared to the majority voting. Our contributions in this article are for the first time the proposal of a model that takes into account both the qualification of the contributor and his behavior in the estimation of his profile. For the second one, the weakening and the aggregation of the answers according to the estimated profiles.

Automated summary

Crowdsourcing is the outsourcing of tasks to a diverse crowd of contributors, resulting in uneven quality of data. To address this issue, a method called MONITOR is proposed, which estimates the contributors' profiles and aggregates the collected data using belief functions to account for possible imperfections. MONITOR starts by estimating the contributor's profile based on their qualification and behavior. Experiments using real data collected from crowdsourcing campaigns show that MONITOR outperforms majority voting in terms of achieving a higher rate of correct answers after aggregation. This article contributes by introducing a model that considers both the qualification and behavior of contributors in profile estimation, as well as the weakening and aggregation of answers based on estimated profiles.