Discrimination, reliability, sensitivity, and specificity of metric-based assessment of an unstable pertrochanteric 31A2 intramedullary nailing procedure performed by experienced and novice surgeons

Introduction: Identifying objective performance metrics for surgical training in orthopedic surgery is im-perative for effective training and patient safety. The objective of this study was to determine if an inter-nationally agreed, metric-based objective assessment of video recordings of an unstable pertrochanteric 31A2 intramedullary nailing procedure distinguished between the performance of experienced and novice orthopedic surgeons.Materials and methods: Previously agreed procedure metrics (i.e., 15 phases of the procedure, 75 steps, errors, and 28 sentinel errors) for a closed reduction and standard cephalomedullary nail fixation with single cephalic element of an unstable pertrochanteric 31A2 fracture. Experienced surgeons trained to as-sess the performance metrics with an interrater reliability (IRR) > 0.8 assessed 14 videos from 10 novice surgeons (orthopaedic residents/trainees) and 20 videos from 14 experienced surgeons (orthopaedic sur-geons) blinded to group and procedure order.Results: The mean IRR of procedure assessments was 0.97. No statistically significant differences were observed between the two groups for Procedure Steps, Errors, Sentinel Errors, and Total Errors. A small number of Experienced surgeons made a similar number of Total Errors as the weakest performing Novices. When the scores of each group were divided at the median Total Error score, large differences were observed between the Experienced surgeons who made the fewest errors and the Novices making the most errors (p < 0.001). Experienced surgeons who made the most errors made significantly more than their Experienced peers (p < 0.003) and the best performing Novices (p < 0.001). Error metrics assessed with Area Under the Curve demonstrated good to excellent Sensitivity and Specificity (0.807- 0.907).Discussion: Binary performance metrics previously agreed by an international Delphi meeting discrimi-nated between the objectively assessed video-recorded performance of Experienced and Novice orthope-dic surgeons when group scores were sub-divided at the median for Total Errors. Error metrics discrim-inated best and also demonstrated good to excellent Sensitivity and Specificity. Some very experienced surgeons performed similar to the Novice group surgeons that made most errors.Conclusions: The procedure metrics used in this study reliably distinguish Novice and Experienced or-thopaedic surgeons' performance and will underpin quality-assured novice training.(c) 2022 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)

Automated summary

This study used internationally agreed procedure metrics to objectively assess and distinguish the performance of experienced and novice orthopedic surgeons in an intramedullary nailing procedure for unstable pertrochanteric 31A2 fractures. While there were no significant differences in Procedure Steps, Errors, Sentinel Errors, and Total Errors between the two groups, a clear distinction was observed when group scores were divided at the median Total Error score. Experienced surgeons with the fewest errors performed significantly better than their peers and the best performing novices, demonstrating good to excellent sensitivity and specificity in error metrics.