Journal
JOURNAL OF MEMORY AND LANGUAGE
Volume 103, Issue -, Pages 151-175Publisher
ACADEMIC PRESS INC ELSEVIER SCIENCE
DOI: 10.1016/j.jml.2018.07.004
Keywords
Type M error; Replicability; Surprisal; Expectation; Locality; Bayesian data analysis; Parameter estimation
Categories
Funding
- Volkswagen Foundation [89 953]
- Deutsche Forschungsgemeinschaft, Collaborative Research Center [(SFB) 1287]
- U.S. Office of Naval Research [N00014-15-1-2541]
Ask authors/readers for more resources
It is well-known in statistics (e.g., Gelman & Carlin, 2014) that treating a result as publishable just because the p-value is less than 0.05 leads to overoptimistic expectations of replicability. These effects get published, leading to an overconfident belief in replicability. We demonstrate the adverse consequences of this statistical significance filter by conducting seven direct replication attempts (268 participants in total) of a recent paper (Levy & Keller, 2013). We show that the published claims are so noisy that even non-significant results are fully compatible with them. We also demonstrate the contrast between such small-sample studies and a larger-sample study; the latter generally yields a less noisy estimate but also a smaller effect magnitude, which looks less compelling but is more realistic. We reiterate several suggestions from the methodology literature for improving current practices.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available