The effectiveness of automated writing evaluation in EFL/ESL writing: a three-level meta-analysis

The present study performs a three-level meta-analysis to investigate the overall effectiveness of automated writing evaluation (AWE) on EFL/ESL student writing performance. 24 primary studies representing 85 between-group effect sizes and 34 studies representing 178 within-group effect sizes found from 1993 to 2021 were separately meta-analyzed. The results indicated a medium overall between-group effect size (g = 0.59) and a large overall within-group effect size (g = 0.98) of AWE on student writing performance. Analyses of moderators show that: (1)- AWE is more effective in improving vocabulary usage but less effective in improving grammar in students' writing; (2)- Grammarly shows potential in being a highly effective tool, though Pigai did not demonstrate such effectiveness; (3)- Medium to long duration of AWE usage leads to a higher effect, but short duration leads to a lower effect in writing outcome compared to non-AWE treatment; (4)- Studying with peers in AWE condition potentially produces a large effect; (5)- AWE is beneficial to students at the undergraduate level, students in the EFL context, and students with intermediate English proficiency. Directions for future research are also discussed in the present study. Overall, AWE is a beneficial application and is recommended for integration in the writing classroom.

Automated summary

This study conducted a meta-analysis to examine the overall effectiveness of automated writing evaluation (AWE) on EFL/ESL student writing performance. The findings revealed a medium overall between-group effect size and a large overall within-group effect size of AWE on student writing performance. The analysis of moderators showed that AWE is more effective in improving vocabulary usage than grammar. Additionally, Grammarly showed potential as a highly effective tool, while Pigai did not demonstrate such effectiveness. The duration of AWE usage, studying with peers, and student characteristics also influenced the effectiveness of AWE. Overall, AWE is recommended for integration in the writing classroom.