Rater agreement and rater severity: A many-faceted Rasch analysis of performance assessments in the Test Deutsch als Fremdsprache (TestDaF)

Performance assessments are subject to rater biases which may operate to strongly reduce the assessments' precision and validity. One particularly influential rater bias refers to the severity, or leniency, of ratings. The present research employs a many-faceted Rasch model (Linacre, 1989; Linacre & Wright, 2002) which allows to measure each rater's severity and to put these severity measures into a common frame of reference along with measures of examinee proficiency and task or criteria difficulty. Moreover, this model yields ability estimates corrected for differences in rater severity. In an application of this approach to the Test of German as a Foreign Language (Test Deutsch als Fremdsprache, TestDaF), ratings of the writing performance of 1359 examinees given by 2 raters each out of a total of 29 raters on 3 criteria are closely examined. The group of raters is shown to be highly heterogeneous, necessitating a correction of assessments for rater severity bias. Finally, various implications of the many-faceted Rasch measurement approach to the evaluation of performance assessments are discussed.