The luck of the referee draw: the effect of exchanging reviews

In journal peer review, editorial decisions on submitted manuscripts are informed by referees' expert recommendations; however, the choice of referees may affect these decisions. Using data from Angewandte Chemie International Edition (AC-IE), this study tested what would have happened if referee reports had been received in a different order. In AC-IE's peer-review process, a manuscript is generally published only if two referees rate the results of the study as important and also recommend publication in the journal (what we have called the 'clear-cut' rule). For 23% of those manuscripts for which a third referee report arrived after the editorial decision was made (37 of 162), this rule would have led to a different decision if the third report had replaced either of the others.