Assessment of service provider competency for child and adolescent psychological treatments and psychosocial services in global mental health: evaluation of feasibility and reliability of the WeACT tool in Gaza, Palestine

Background. There is a scarcity of evaluated tools to assess whether non-specialist providers achieve minimum levels of competency to effectively and safely deliver psychological interventions in low- and middle-income countries. The objective of this study was to evaluate the reliability and utility of the newly developed Working with children - Assessment of Competencies Tool (WeACT) to assess service providers' competencies in Gaza, Palestine. Methods. The study evaluated; (1) psychometric properties of the WeACT based on observed role-plays by trainers/supervisors (N = 8); (2) sensitivity to change among service provider competencies (N = 25) using pre-and-post training WeACT scores on standardized role-plays; (3) in-service competencies among experienced service providers (N = 64) using standardized role-plays. Results. We demonstrated moderate interrater reliability [intraclass correlation coefficient, single measures, ICC = 0.68 (95% CI 0.48-0.86)] after practice, with high internal consistency (alpha = 0.94). WeACT assessments provided clinically relevant information on achieved levels of competencies (55% of the competencies were scored as adequate pre-training; 71% post-training; 62% in-service). Pre-post training assessment saw significant improvement in competencies (W = -3.64; p < 0.001). Conclusion. This study demonstrated positive results on the reliability and utility of the WeACT, with sufficient inter-rater agreement, excellent internal consistency, sensitivity to assess change, and providing insight needs for remedial training. The WeACT holds promise as a tool for monitoring quality of care when implementing evidence-based care at scale.

Automated summary

This study evaluated the reliability and utility of the WeACT tool in assessing service providers' competencies in Gaza, demonstrating moderate interrater reliability, high internal consistency, sensitivity to assess change, and providing insight into remedial training needs. WeACT holds promise as a tool for monitoring the quality of care when implementing evidence-based care at scale.