Setting standards for the assessment of operative competence

Background. We have previously shown that the operative competence of trainees can be reliably assessed using structured checklists and video recording. These assessments are useful for training (formative assessment). However, a standard setting exercise is required before they can be used for examinations (summative assessment). Methods. Blinded videos of a saphenofemoral disconnection by an experienced (competent) trainee and an inexperienced trainee (not competent) were scored, using a structured checklist, by 14 consultant vascular surgeons and 14 vascular trainees. The observers were also asked to decide whether the surgeon was competent, borderline or not competent. Thirteen vascular operating room (OR) nurses performed the same exercise. The 'contrasting groups' method was used to compare the cut point between the scores. Results. There was complete separation between the surgeons' scores for the experienced trainee (median 16, range 13-18) and the inexperienced trainee (median 6.5, range 2-12), Chi-square p = 0.0001. This separation was confirmed by the judgements for competent (14-18), borderline (15-7), and not competent (8-2), p = 0.0001. Trainees awarded lower scores than the consultants to both videos, although this difference was not significant. The nurses performed almost as well as the surgeons. Conclusions. Surgeons can discriminate between the video recordings of a competent and non-competent trainee. Such recordings could form part of a trainee's portfolio and contribute to subsequent examinations (stummative assessment). It seems that nurses can also be used to assess the operative competence of surgical trainees.