Examiner characteristics and interrater reliability in a communication OSCE

Objective: To identify inter-individual examiner factors associated with interrater reliability in a summative communication OSCE in the 4th study year. Methods: The OSCE consists of 4 stations assessed with a 4-item 5-point global rating instrument. A bivariate secondary analysis of interrater reliability in relation to 4 examiner factors (gender, profession, OSCE experience, examiner training) was conducted. Intraclass correlation coefficients (ICC) were calculated and compared between examiner dyads of different similarity. Results: 169 pairwise ratings from 19 different examiners in 16 dyads were analysed. Interrater reliability is significantly higher in examiner dyads of same vs. different gender (ICC = 0.76 (95% CI = 0.65-0.83) vs. ICC = 0.41 (95% CI = 0.21-0.57)), in dyads of two clinicians vs. non-clinical/mixed professions (ICC = 0.72 (95% CI = 0.56-0.83) vs. ICC = 0.57 (95% CI = 0.41-0.69)), and in dyads with high vs. low/mixed OSCE experience (ICC = 0.73 (95% CI 0.50-0.87) vs. ICC = 0.56 (95% CI = 0.41-0.69)). Participation in recent examiner training had no influence on ICCs. Conclusion: Better concordance of ratings between clinically active examiners might be a hint for context specificity of good communication. Higher interrater reliability between examiners with same gender may indicate gender-specific communication concepts. Practice implications: Medical faculties introducing summative assessment of communication competence should focus the influence of examiner characteristics on interrater reliability. (C) 2017 Elsevier B.V. All rights reserved.