Journal
EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT
Volume 78, Issue 3, Pages 430-459Publisher
SAGE PUBLICATIONS INC
DOI: 10.1177/0013164416689696
Keywords
rater effects; social network analysis; exponential random graph models; Rasch model
Ask authors/readers for more resources
It is common practice for assessment programs to organize qualifying sessions during which the raters (often known as markers or judges) demonstrate their consistency before operational rating commences. Because of the high-stakes nature of many rating activities, the research community tends to continuously explore new methods to analyze rating data. We used simulated and empirical data from two high-stakes language assessments, to propose a new approach, based on social network analysis and exponential graph models, to evaluate the readiness of a group of raters for operational rating. The results of this innovative approach are compared with the results of a Rasch analysis, which is a well-established approach for the analysis of such data. We also demonstrate how the new approach can be practically used to investigate important research questions such as whether rater severity is stable across rating tasks. The merits of the new approach, and the consequences for practice are discussed.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available