Assessment of a 360-degree instrument to evaluate residents' competency in interpersonal and communication skills

Objective. To test the reliability of the 360-degree evaluation instrument for assessing residents' competency in interpersonal and communication skills. Method. Ten-item questionnaires were distributed to residents and evaluators at Monmouth Medical Center in Long Branch, New Jersey, in March/April, 2002. The scoring scale was 1-5; the highest score was 50. Data were maintained strictly confidential; each resident was assigned a code. Completed data sheets were collated by category and entered into a spreadsheet. The total and mean scores by each category of evaluator were calculated for each resident and a rank order list created. Shrout-Fleiss (model 2) intraclass correlation coefficients measured reliability of ratings within each group of evaluators. Reliability/reproducibility among evaluators' scores were tested by the Pearson correlation coefficient (p < .05). Results. Intraclass correlation coefficients showed a narrow range, from .85-.54. The highest ranked resident overall ranked high and the lowest was low with most evaluators. The rank order among fellow residents was markedly different from other evaluator categories. Pearson correlation coefficients showed significant correlation between faculty and ancillary staff, (p = .002). Patients as evaluators did show intraclass correlation, but did not correlate significantly with other categories. Scores from colleagues correlated negatively with all other categories of evaluators. Conclusions. The 360-degree instrument appears to be reliable to evaluate residents' competency in interpersonal and communication skills. Information from the assessment may provide feedback to residents. Areas of improvement identified by the scores would suggest areas for improvement and further ongoing assessment.