One data set, many analysts: Implications for practicing scientists

Researchers routinely face choices throughout the data analysis process. It is often opaque to readers how these choices are made, how they affect the findings, and whether or not data analysis results are unduly influenced by subjective decisions. This concern is spurring numerous investigations into the variability of data analysis results. The findings demonstrate that different teams analyzing the same data may reach different conclusions. This is the many-analysts problem. Previous research on the many-analysts problem focused on demonstrating its existence, without identifying specific practices for solving it. We address this gap by identifying three pitfalls that have contributed to the variability observed in many-analysts publications and providing suggestions on how to avoid them.

Automated summary

Researchers often make choices in the data analysis process, but it is unclear to readers how these choices are made and how they affect the findings. Concerns about subjective decisions influencing data analysis results have led to investigations into the variability of these results. Previous research focused on the existence of the "many-analysts problem" without suggesting specific practices to solve it. This study identifies three pitfalls contributing to the observed variability and provides suggestions to avoid them.