Journal
ADVANCES IN METHODS AND PRACTICES IN PSYCHOLOGICAL SCIENCE
Volume 1, Issue 3, Pages 337-356Publisher
SAGE PUBLICATIONS INC
DOI: 10.1177/2515245917747646
Keywords
crowdsourcing science; data analysis; scientific transparency; open data; open materials
Categories
Funding
- Institute of Education Sciences, U.S. Department of Education [R305B090002]
- European Research Council [283876]
- Jan Wallander and Tom Hedelius Foundation [P2015-0001:1]
- Swedish Foundation for Humanities and Social Sciences [NHS14-1719:1]
- Leverhulme Trust [ECF-2011-576, RPG2013-326]
- Natural Sciences and Engineering Research Council of Canada [CGSD2-426287-2012]
- Swedish Foundation for Humanities and Social Sciences [NHS14-1719:1] Funding Source: Swedish Foundation for Humanities and Social Sciences
- European Research Council (ERC) [283876] Funding Source: European Research Council (ERC)
Ask authors/readers for more resources
Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 (Mdn = 1.31) in odds-ratio units. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. Overall, the 29 different analyses used 21 unique combinations of covariates. Neither analysts' prior beliefs about the effect of interest nor their level of expertise readily explained the variation in the outcomes of the analyses. Peer ratings of the quality of the analyses also did not account for the variability. These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy in which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective, analytic choices influence research results.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available