Impact of automated writing evaluation on teacher feedback, student revision, and writing improvement

Recent years have witnessed an increasing interest in the use of automated writing evaluation (AWE) in second language writing classrooms. This increase is partially due to the belief that AWE can assist teachers by allowing them to devote more feedback to higher-level (HL) writing skills, such as content and organization, while the technology addresses lower-level (LL) skills, such as grammar. As is speculated, student revisions will then be positively impacted. However, little evidence has supported these claims, calling into question the impact of AWE on teaching and learning. The current study explored these claims by comparing two second language writing classes that were assigned to either an AWE + teacher feedback condition or a teacher-only-feedback condition. Findings suggest that using AWE as a complement to teacher feedback did not have a significant impact on the amount of HL teacher feedback, but the teacher who did not use AWE tended to provide a greater amount of LL feedback than AWE alone. Furthermore, students seemed to revise the teacher's LL feedback more frequently than LL feedback from the computer. Interestingly, students retained their improvement in accuracy in the long-term when they had access to AWE, but students who did not have access appeared to have lower retention. We explain the relevance of our findings in relation to an argument-based validation framework to align our work with state-of-the-art research in the field and contribute to a broader discussion about how AWE can be best provided to support second language writing development.

Automated summary

In recent years, there has been growing interest in the use of automated writing evaluation (AWE) in second language writing classrooms. However, little evidence has supported the claims of its effectiveness. This study compared two groups of second language writing classes, one with AWE and teacher feedback, and the other with only teacher feedback. The results showed that AWE did not significantly impact the amount of feedback provided by teachers, but teachers without AWE tended to provide more lower-level feedback. Interestingly, students seemed to revise the teacher's feedback more frequently than computer-generated feedback and retained their improvement in accuracy in the long-term when they had access to AWE. This study contributes to the discussion on how AWE can best support second language writing development.