Journal
SCIENTOMETRICS
Volume 69, Issue 3, Pages 591-606Publisher
SPRINGER
DOI: 10.1007/s11192-006-0171-4
Keywords
-
Funding
- Economic and Social Research Council [ES/F041292/1] Funding Source: researchfish
Ask authors/readers for more resources
Peer reviews are highly valued in academic life, but are notoriously unreliable. A major problem is the substantial measurement error due to the idiosyncratic responses when large numbers of different assessors each evaluate only a single or a few submissions (e.g., journal articles, grants, etc.). To address this problem. the main funding body of academic research in Australia conducted a trial reader system in which each of a small number of senior academics read all proposals within their subdiscipline. The traditional peer review process for 1996 (2,989 proposals, 6,233 assessors) resulted in unacceptably low reliabilities comparable with those found in other research (0.475 for research project, 0.572 for researcher). For proposals from psychology and education in 1997, the new reader system resulted in substantially higher reliabilities: 0.643 and 0.881, respectively. In comparison to the traditional peer review approach, the new reader system is substantially more reliable, timely, and cost efficient - and applicable to many peer review situations.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available