Journal
METHODOLOGY-EUROPEAN JOURNAL OF RESEARCH METHODS FOR THE BEHAVIORAL AND SOCIAL SCIENCES
Volume 18, Issue 2, Pages 126-143Publisher
PSYCHOPEN
DOI: 10.5964/meth.8331
Keywords
online data collection; screening solutions; vetting procedures; data quality
Ask authors/readers for more resources
Amazon's Mechanical Turk (MTurk) has become a crucial source for participant recruitment in social-sciences, but concerns about data quality have been raised. In response, CloudResearch developed an intensive pre-screening procedure to vet MTurk participants and exclude those providing low-quality data. Results showed that the reliability and validity of scales improved with the implementation of this prescreening procedure, especially in more recent versions. This suggests that the procedure is a valuable tool for ensuring high-quality data collection on MTurk.
In recent years, Amazon???s Mechanical Turk (MTurk) has become a pivotal source for participant recruitment in many social-science fields. In the last several years, however, concerns about data quality have arisen. In response, CloudResearch developed an intensive pre-screening procedure to vet the full participant pool available on MTurk and exclude those providing low-quality data. To assess its efficacy, we compared three MTurk samples that completed identical measures: Sample 1 was collected prior to the pre-screening???s implementation. Sample 2 was collected shortly following its implementation, and Sample 3 was collected nearly a full-year after its implementation. Results indicated that the reliability and validity of scales improved with the implementation of this prescreening procedure, and that this was especially apparent with more recent versions. Thus, this prescreening procedure appears to be a valuable tool to help ensure the collection of high-quality data on MTurk.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available