Lessons from a Decade of Replications at the Quarterly Journal of Political Science

To allow researchers to investigate not only whether a paper's methods are theoretically sound but also whether they have been properly implemented and are robust to alternative specifications, it is necessary that published papers be accompanied by their underlying data and code. This article describes experiences and lessons learned at the Quarterly Journal of Political Science since it began requiring authors to provide this type of replication code in 2005. It finds that of the 24 empirical papers subjected to in-house replication review since September 2012, only four packages did not require any modifications. Most troubling, 14 packages (58%) had results in the paper that differed from those generated by the author's own code. Based on these experiences, this article presents a set of guidelines for authors and journals for improving the reliability and usability of replication packages.