Nurse anesthetists' evaluations of anesthesiologists' operating room performance are sensitive to anesthesiologists' years of postgraduate practice

Study objective: The first aim of this study was to test whether a 7 item evaluation scale developed by our department's certified registered nurse anesthetist (CRNAs) was psychometrically reliable. The second aim was to test whether anesthesiologist' performance changed with their years of postgraduate experience. Design, setting, measurements: Sixty-two University of Iowa CRNAs evaluated 81 anesthesiologists during one weekend. Anesthesiologists' scores were adjusted for CRNA rater leniency. Anesthesiologists' scores were tested for sensitivity to CRNA-anesthesiologist case-specific variables. Scores also were tested against anesthesiologists' years of postgraduate experience. The latter association was tested for sensitivity to case-specific variables, anesthesiologists' clinical supervision scores provided by residents, and anesthesiologist clinical assignment variables. Main results: The 7 items demonstrated a single-factor structure, allowing calculation of mean score over the 7 items. Individual anesthesiologist scores were reliable when scores were provided by at least 10 different CRNAs. Anesthesiologists' scores (mean 3.34 [SD 0.411) were not affected by the interval since last CRNA-anesthesiologist interaction, number of interactions, or case-specific variables. There was a negative association between leniency-adjusted anesthesiologist scores and years of anesthesiologist postgraduate practice (coefficient -0.20 per decade, t = -19.39, P < 0.0001). The association remained robust when accounting for case-specific variables, resident clinical supervision scores, and overall clinical assignment variables. Conclusions: Anesthesiologist' operating room performance can be evaluated reliably by non-physician anesthesia providers (CRNAs). The evaluation process can be done reliably and validly using an assessment scale consisting of only a few (< 10) items and with evaluations by only a few individuals (>= 10 CRNA raters). There is no indication evaluations provided by CRNAs were significantly influenced by the interval between interaction and evaluation, number of interactions, or other case-specific variables. From CRNAs' perspectives, on average, as anesthesiologists gain experience, anesthesiologists' behaviors in the operating room change, providing CRNAs with less direct assistance in patient care.