Effectiveness of automated writing evaluation systems in school settings: A systematic review of studies from 2000 to 2020

Background Automated Writing Evaluation (AWE) systems to aid writing learning and instruction in primary and secondary education are growing increasingly popular. However, their effectiveness is hardly known. We conducted a systematic review focusing on the effects of these systems providing writing feedback to students in school settings. Objectives Our goal was to identify and characterize AWE systems tested in the last 20 years for Grades 1-12 and examine their impact on text quality and other writing-related outcomes. Methods The review followed PRISMA guidelines. We identified eight studies reporting the effects on writing of six AWE systems on 1659 students 11-17 years of age. Results and conclusions Our review supported the usefulness of AWE systems for writing learning and instruction. Except for one, all studies showed a positive effect of automated feedback in at least one writing-related measure. The integration of AWE systems into more extensive instructional programs, the amount of writing practice provided to students, the type of the control groups, and the role of teachers are factors influencing their impact on students' writing outcomes. Relevance Our review generally supported the value of AWE systems in the teaching/learning process of writing. A closer look into the conditions in which AWE systems are put to practice suggested that they are particularly effective when embedded into comprehensive instructional programs providing ample writing opportunities. Findings from this review expand knowledge on AWE systems as valuable tools to enhance writing in school settings.

Automated summary

A systematic review on the effects of Automated Writing Evaluation (AWE) systems in school settings found that these systems generally have a positive impact on writing learning and instruction. They are particularly effective when integrated into comprehensive instructional programs providing ample writing opportunities.