Observational Teamwork Assessment for Surgery: Content Validation and Tool Refinement

BACKGROUND: Effective teamwork is crucial for safe surgery. Failures in nontechnical and teamwork skills are frequently implicated in adverse events. The Observational Teamwork Assessment for Surgery (OTAS) tool assesses teamwork of the entire team in the operating room. Empirical testing of OTAS has yet to explore the content validity of the tool. STUDY DESIGN: This was a cross-sectional observational study. Data were collected in 30 procedures by 2 trained researchers. Five teamwork behaviors were scored (ie, communication, leadership, cooperation, coordination, and monitoring) and behavior exemplar completion was recorded (phase 1). Expert operating room personnel (5 surgeons, 5 anesthesiologists, and 5 scrub nurses) assessed the content validity of the OTAS exemplar behaviors. Finally, a panel of operating room patient-safety experts refined the exemplars (phase 2). RESULTS: In total, the observability (presence/absence) of 130 exemplars was assessed by 2 blinded observers in 30 general surgical cases. Observer agreement was high (Cohen's kappa >= 0.41) for 83.85% (109 of 130) of exemplar behaviors; 60.77% (79 of 130) of exemplar behaviors were observed frequently with high observer agreement. The majority of the exemplars were rated by expert operating room practitioners and an expert panel as substantial contributors to teamwork and patient safety. Based on expert consensus, 21 behavior exemplars were removed from OTAS and an additional 23 were modified. CONCLUSIONS: The exemplars of OTAS demonstrated very good content validity. Taken together with recent evidence on the construct validity of the tool, these findings demonstrate that OTAS is psychometrically robust for capturing teamwork in the operating room. (J Am Coll Surg 2011;212: 234-243. (C) 2010 by the American College of Surgeons)