Can Short Answers to Open Response Questions Be Auto-Graded Without a Grading Rubric?

Auto-grading short-answers seems to be sufficiently resolved. However, most auto-graders require comprehensive scoring rubrics, which were not always available. This paper used modern machine learning techniques to build auto-graders without expressly defining the rubrics. The result shows that the best auto-grading model is able to achieve a good inter-rater agreement (kappa = 0.625) with expert grading. The agreement can be further improved (kappa = 0.726) if the auto-grading model gave up scoring some of the answers.