People who judge people

Experts who judge people usually provide opinions. It can be challenging to evaluate the professional performance of those experts, because for many domains there is no applicable external standard against which to verify the opinions. We review traditional methods for assessment and propose the purely empirical CWS approach as an alternative. Expert judgment entails discriminating among the various stimuli within the domain as well as being consistent when judging similar stimuli. We combine observed measures of these two components to form a ratio that we call the CWS index of expertise. We demonstrate the value of the index in an analysis of prioritization judgments made by occupational therapy students before and after they received specific training. The students' CWS scores improved considerably after training. The promise of the index as a selection tool is supported by the positive correlation of pre-training scores with both post-training scores and with course grades. Copyright (c) 2006 John Wiley & Sons, Ltd.