The Development and Evaluation of a Novel Instrument Assessing Residents' Discharge Summaries

Purpose To develop and determine the reliability of a novel measurement instrument assessing the quality of residents' discharge summaries. Method In 2014, the authors created a discharge summary evaluation instrument based on consensus recommendations from national regulatory bodies and input from primary care providers at their institution. After a brief pilot, they used the instrument to evaluate discharge summaries written by first-year internal medicine residents (n = 24) at a single U.S. teaching hospital during the 2013-2014 academic year. They conducted a generalizability study to determine the reliability of the instrument and a series of decision studies to determine the number of discharge summaries and raters needed to achieve a reliable evaluation score. Results The generalizability study demonstrated that 37% of the variance reflected residents' ability to generate an adequate discharge summary (true score variance). The decision studies estimated that the mean score from six discharge summary reviews completed by a unique rater for each review would yield a reliability coefficient of 0.75. Because of high interrater reliability, multiple raters per discharge summary would not significantly enhance the reliability of the mean rating. Conclusions This evaluation instrument reliably measured residents' performance writing discharge summaries. A single rating of six discharge summaries can achieve a reliable mean evaluation score. Using this instrument is feasible even for programs with a limited number of inpatient encounters and a small pool of faculty preceptors.