From black box to white box at open access journals: predictive validity of manuscript reviewing and editorial decisions at Atmospheric Chemistry and Physics

More than 4,500 open access (OA) journals have now become established in science But doubts exist about the quality of the manuscript selection process for publication in these journals In this study we investigate the quality of the selection process of an OA journal, taking as an example the journal Atmospheric Chemistry and Physics (ACP) ACP is working with a new system of public peer review We examine the predictive validity of the ACP peer review system, namely, whether the process selects the best of the manuscripts submitted We have data for 1,111 manuscripts that went through the complete ACP selection process in the years 2001 to 2006 The predictive validity was investigated on the basis of citation counts for the later published manuscripts The results of the citation analysis confirm the predictive validity of the reviewers' ratings and the editorial decisions at ACP Both co-vary with citation counts for the published manuscripts