Reliability of a procedural checklist as a high-stakes measurement of advanced technical skill

BACKGROUND: The purpose of this study was to determine the reliability of a previously validated laparoscopic Nissen fundoplication procedural checklist as a measurement of advanced technical skill. METHODS: Five surgeons, using a 65-step procedural checklist, independently evaluated 2 video recordings of expert surgeon operative performances of a laparoscopic Nissen fundoplication. Results were analyzed for percent agreement and Fleiss kappa correlation for each operation independently and combined as a whole. RESULTS: Sixty-four of the 65 steps had 80% or higher percent agreement for both operative case A and B independently and when considered overall. The Fleiss kappa coefficients for operative case A (kappa = .95) and operative case B (kappa = .96) and overall (operative case A + B) (kappa = .95). CONCLUSION: The percentage agreement and kappa coefficients for all 3 analyses indicate a high degree of reliability (>.80) that would support the use of this instrument for high-stakes assessment of a laparoscopic Nissen fundoplication. (C) 2010 Elsevier Inc. All rights reserved.