Interrater reliability of the Patient Education Materials Assessment Tool (PEMAT)

Objective: To assess the interrater reliability (IRR) and usability of the Patient Education Materials Assessment Tool (PEMAT) and the relationship between PEMAT scores and readability levels. Methods: One hundred ten materials (80 print, 30 audiovisual) were evaluated, each by two raters, using the PEMAT. IRR was calculated using Gwet's AC1 and summarized across items in each PEMAT domain (understandability and actionability) and by material type. A survey was conducted to solicit raters' experience using the PEMAT. Readability of each material was assessed using the SMOG Index. Results: The median IRR was 0.92 for understandability and 0.93 for actionability across all relevant items, indicating good IRR. Eight PEMAT items had Gwet's AC1 values less than 0.81. PEMAT and SMOG Index scores were inversely correlated, with a Spearman's rho of -0.20 (p = 0.081) for understandability and -0.15 (p = 0.194) for actionability. While 92% of raters agreed the PEMAT was easy to use, survey results suggested specific items for clarification. Conclusion: While the PEMAT demonstrates moderate to excellent IRR overall, amendments to items with lower IRR may increase the usefulness of the tool. (c) 2017 Elsevier B.V. All rights reserved.