Predicting Performance at Graduation From Early ACGME Milestone Ratings: Longitudinal Learning Analytics in Professionalism and Communication in Vascular Surgery

OBJECTIVE: Program directors in surgical disciplines need more tools from the ACGME to help them use Mile-stone ratings to improve trainees' performance. This is especially true in competencies that are notoriously diffi-cult to measure, such as professionalism (PROF) and interpersonal and communication skills (ICS). It is now widely understood that skills in these two areas have direct impact on patient care outcomes. This study investigated the potential for generating early predictors of final Milestone ratings within the PROF and ICS com-petency categories.DESIGN: This retrospective cohort study utilized Mile-stone ratings from all ACGME-accredited vascular surgery training programs, covering residents and fellows who completed training in June 2019. The outcome measure studied was the rate of achieving the recommended grad-uation target of Milestone Level 4 (possible range: 1-5), while the predictors were the Milestone ratings attained at earlier stages of training. Predictive probability values (PPVs) were calculated for each of the 3 PROF and two ICS sub-competencies to estimate the probability of train-ees not reaching the recommended graduation target based on their previous Milestone ratings. SETTING: All ACGME-accredited vascular surgery train-ing programs within the United States.PARTICIPANTS: : All trainees completing a 2 year vascu-lar surgery fellowship (VSF) in June 2019 (n = 119) or a 5 year integrated vascular surgery residency (IVSR) in June 2019 (n = 52) were included in the analyses.RESULTS: The overall rate of failing to achieve the rec-ommended graduation target across all PROF and ICS sub-competencies ranged from 7.7% to 21.8% of all train-ees. For trainees with a Milestone rating at < 2.5 with 1 year remaining in their training program, the predictive probability of not achieving the recommended gradua-tion target ranged from 37.0% to 71.5% across sub -com-petencies, with the highest risks observed under PROF for Administrative Tasks (71.5%) and under ICS for Communication with the Healthcare Team (56.7%).CONCLUSIONS: As many as 1 in 4 vascular surgery trainees did not achieve the ACGME vascular surgery Milestones targets for graduation in at least one of the PROF and ICS sub-competencies. Biannual ACGME Mile-stone assessment ratings of PROF and ICS during early training can be used to predict achievement of compe-tency targets at time of graduation. Early clues to problems in PROF and ICS enable programs to address potential deficits early in training to ensure competency in these essential non-technical skills prior to entering unsupervised practice.( J Surg Ed 80:235-246. (c) 2022 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)

Automated summary

This study investigated the potential for generating early predictors of final Milestone ratings in professionalism and interpersonal and communication skills. It found that as many as 1 in 4 vascular surgery trainees did not achieve the ACGME Milestone targets in these competencies. Therefore, early assessment ratings can be used to predict achievement of competency targets at the time of graduation, allowing programs to address potential deficits early in training.