Journal
NATURE
Volume 582, Issue 7810, Pages 84-+Publisher
NATURE PORTFOLIO
DOI: 10.1038/s41586-020-2314-9
Keywords
-
Categories
Funding
- Austrian Science Fund [P29362-G27]
- Israel Science Foundation [ISF 2004/15]
- Swedish Foundation for Humanities and Social Sciences [NHS14-1719:1]
- National Institutes of Health (NIH) [R24MH117179]
- Austrian Science Fund (FWF) [SFB F63, P 32686]
- Research Foundation Flanders (FWO)
- European Union [665501]
- University of Basel Research Fund
- Research Foundation Flanders [12T2517N]
- Marie Skodowska-Curie Actions under COFUND [665501]
- Obra Social La Caixa
- Vienna Science and Technology Fund [WWTF VRG13-007]
- National Natural Science Foundation of China [71801110, 71971199, 71602175, 71942004]
- MOE (Ministry of Education in China) Project of Humanities and Social Sciences [18YJC630268]
- China Postdoctoral Science Foundation [2018M633270]
- Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy 'Science of Intelligence' (EXC 2002/1) [390523135]
- Brain Canada
- Health Canada
- NIH [NIH-NIBIB P41 EB019936, NIH-NIMH R01 MH083320, NIH RF1 MH120021, R01 DA041353]
- National Institute Of Mental Health of the NIH [R01MH096906]
- Canada First Research Excellence Fund
- European Union's Horizon 2020 Research and Innovation Programme [785907]
- Max Planck Society
- Dutch foundation LSH-TKI [LSHM16053-SGF]
- NSF [1631325]
- T32 Predoctoral Fellowship from the NIH
- Deutsche Forschungsgemeinschaft [CRC1193]
- Australian National Imaging Facility
- National Collaborative Research Infrastructure Strategy (NCRIS) capability
- VIDI from the Netherlands Organisation for Scientific Research [452-17-013]
- Swedish Research Council
- NIH IRP [ZICMH002888, ZICMH002960]
- Tianqiao and Chrissy Center for Social and Decision Neuroscience Center Leadership Chair
- Belgian Excellence of Science program (EOS) from the FNRS-Belgium [30991544]
- EraNET Neuron [R4195]
- Ministry of Education of Humanities and Social Science [16YJC630103]
- Wellcome Trust [100309/Z/12/Z]
- Swedish Foundation for Humanities and Social Sciences [NHS14-1719:1] Funding Source: Swedish Foundation for Humanities and Social Sciences
Ask authors/readers for more resources
Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses(1). The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset(2-5). Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed. The results obtained by seventy different teams analysing the same functional magnetic resonance imaging dataset show substantial variation, highlighting the influence of analytical choices and the importance of sharing workflows publicly and performing multiple analyses.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available