Journal
APPLIED LINGUISTICS
Volume 42, Issue 5, Pages 924-944Publisher
OXFORD UNIV PRESS
DOI: 10.1093/applin/amaa067
Keywords
-
Categories
Funding
- ETS under a Committee of Examiners and the Test of English as a Foreign Language research grant
Ask authors/readers for more resources
This study explores the challenges of large-scale oral communication assessments and suggests that using a spoken dialog system can be a feasible approach. The results show that the computer partner is more reliable in assessing interactional competence, but human raters generally prefer the human partner due to its perceived authenticity and naturalness.
A challenge of large-scale oral communication assessments is to feasibly assess a broad construct that includes interactional competence. One possible approach in addressing this challenge is to use a spoken dialog system (SDS), with the computer acting as a peer to elicit a ratable speech sample. With this aim, an SDS was built and four trained human raters assessed the discourse elicited from 40 test takers that completed a paired oral task with both a human and a computer partner. The test takers were evaluated based on the analytic operational oral communication rating scales which included interactional competence, fluency, pronunciation, and grammar/vocabulary. Repeated-measures ANOVA indicated that fluency, pronunciation, and grammar and vocabulary were scored similarly across the two conditions, while interactional competence was scored substantially higher in the human partner condition. A g-study indicated that the computer partner was more reliable in assessing interactional competence, and rater questionnaire and interview data suggested the computer provided a more standardized assessment. Conversely, raters generally favored the human partner, in part because of its perceived authenticity and naturalness.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available