The reliability of peer reviews of papers on information systems

This paper analyses the reliability of the double-blind peer review systems used for submissions to the 2001 and 2002 UK Academy for Information Systems (UKAIS) conferences. The level of reliability found in the first conference was marginally lower than would be expected from a model based on chance. in the second conference the reliability level was significantly better, but still low. The paper explores some of the implications of this for the reviewing system, and suggests a model for assessing the impact of low levels of reliability.