The reliability of peer review in anthrozoology

We evaluated the reliability of peer review ratings of 218 proposals that were submitted for presentation at the 2004 meeting of the International Association of Human-Animal Interaction Organizations (IAHAIO). Each proposal was evaluated on four dimensions by different sets of three raters who were blind to the identity of the authors. Single measure intraclass correlations between the evaluations of raters indicated that the reliabilities of judgments were poor for abstracts submitted as research presentations and as descriptions of applied human-animal programs, and fair for abstracts submitted as critical reviews. Reliability of the selection system was considerably improved by averaging the scores of the three raters and using the combined scores to make accept/reject decisions. We suggest modifications to the review process that may enhance the reliability of the selection process for proposals submitted for meetings and for journal submissions in anthrozoology.