Reliability and validity of procedure-based assessments in otolaryngology training

Objectives/HypothesisTo investigate the reliability and construct validity of procedure-based assessment (PBA) in assessing performance and progress in otolaryngology training. Study DesignRetrospective database analysis using a national electronic database. MethodsWe analyzed PBAs of otolaryngology trainees in North London from core trainees (CTs) to specialty trainees (STs). The tool contains six multi-item domains: consent, planning, preparation, exposure/closure, technique, and postoperative care, rated as satisfactory or development required, in addition to an overall performance rating (pS) of 1 to 4. Individual domain score, overall calculated score (cS), and number of development-required items were calculated for each PBA. Receiver operating characteristic analysis helped determine sensitivity and specificity. ResultsThere were 3,152 otolaryngology PBAs from 46 otolaryngology trainees analyzed. PBA reliability was high (Cronbach's 0.899), and sensitivity approached 99%. cS correlated positively with pS and level in training (r(s): +0.681 and +0.324, respectively). ST had higher cS and pS than CT (93%0.6 and 3.20.03 vs. 71%+/- 3.1 and 2.3 +/- 0.08, respectively; P<.001). cS and pS increased from CT1 to ST8 showing construct validity (r(s): +0.348 and +0.354, respectively; P<.001). The technical skill domain had the highest utilization (98% of PBAs) and was the best predictor of cS and pS (r(s): +0.96 and +0.66, respectively). ConclusionsPBA is reliable and valid for assessing otolaryngology trainees' performance and progress at all levels. It is highly sensitive in identifying competent trainees. The tool is used in a formative and feedback capacity. The technical domain is the best predictor and should be given close attention. Level of EvidenceNA Laryngoscope, 125:1328-1335, 2015