The Effect of Small Group Discussion on Cutoff Scores During Standard Setting

Standard setting methods, like the Bookmark procedure, are used to assist education experts in formulating performance standards. Small group discussion is meant to help these experts in setting more reliable and valid cutoff scores. This study is an analysis of 15 small group discussions during two standards setting trajectories and their effect on the cutoff scores on four performance levels for comprehensive reading and mathematics. Discussion decreased the variability of the cutoff scores among the expert panelists, but the direction of the adaptations varied among groups. Furthermore, also the duration and the content of the audio-taped discussions differed among groups. There was no relationship between the increase in agreement among the panelists and the duration of their discussions or their use of arguments concerning learning content. It was concluded that an increased consensus among panelists alone does not provide enough information on the reliability and validity of cutoff scores. Additional measures aimed at the content of group discussions have appeared to be necessary, since the use of content arguments in these discussions is not guaranteed.