Understanding crowdsourcing projects: A systematic review of tendencies, workflow, and quality management

Crowdsourcing is a powerful method that leverages human intelligence to solve problems in the medical, linguistics, machine learning, and in a considerable number of other fields of study. This systematic review aims to understand how crowdsourcing projects are designed and executed in the state-of-the-art, considering the following dimensions: Task execution, quality management, and platform usage. Our results summarized trends of the important aspects of a crowdsourcing project, such as crowd and task types, crowdsourcing platforms, and activities used to manage the quality; we also addressed functions and limitations in traditional crowdsourcing platforms, the definition of a crowdsourcing workflow, and the lack of standardization when developing a crowdsourcing project. As future works, we wish to design a conceptual model that represents the important elements of a crowdsourcing project and their relationships, also, we wish to do further researchers focusing on how the quality mechanisms listed in this work should be used in crowdsourcing projects to get reliable results.