Inter-rater reliability of assessments regarding the quality of drug treatment, and drug-related hospital admissions

Aims To investigate inter-rater agreement on the quality of drug treatment, and the relationship between the drug treatment and hospital admission. Methods Three specialist physicians and two resident physicians determined, independently and in consensus, the quality of drug treatment from an overall medical perspective, and its association with admission, in 30 randomly selected patients (50% female, median age 72 years) admitted to Sahlgrenska University Hospital, Sweden, in April 2018. The inter-rater agreement was evaluated with Gwet's agreement coefficient (AC(1)). Results In all, 200 (95%) out of 210 drugs at admission and 238 (97%) out of 245 drugs at discharge were assessed as reasonable drug treatment by all assessors. Conversely, none of the drugs at admission, and two at discharge, were assessed as unreasonable drug treatment by all assessors (AC(1): 0.88 and 0.94 [all], 0.86 and 0.95 [specialists], 0.92 and 0.92 [residents], respectively). The assessments regarding the association between the drug treatment and the hospital admission (not related or main/contributory reason) were consistent between the assessors for 16 out of 30 patients (AC(1): 0.67 [all], 0.74 [specialists], 0.54 [residents]). In none of the three cases where the hospital admission was considered possibly attributable to a prescribing error did the assessors make consistent assessments. Conclusions As the inter-rater agreement ranged between weak and almost perfect, the reliability of assessments of drug treatment quality, as well as adverse consequences, appears to be a methodological concern. To yield acceptably reliable results regarding both drug treatment aspects at issue, specialist physicians should be involved.

Automated summary

Overall, there was a high level of agreement among assessors regarding the quality of drug treatment at admission and discharge, but a lower agreement on the association between drug treatment and hospital admission. In cases where hospital admission was possibly attributable to a prescribing error, assessors had poor agreement. Therefore, involving specialist physicians is recommended for more reliable assessments on both drug treatment quality and its impact on hospital admission.