Journal
JOURNAL OF SCIENCE EDUCATION AND TECHNOLOGY
Volume 25, Issue 3, Pages 358-374Publisher
SPRINGER
DOI: 10.1007/s10956-015-9598-9
Keywords
Computer scoring; Open-ended assessment; Misspelled words; Machine learning; Misclassification; Computers; Assessment
Funding
- National Science Foundation TUES grant [1322872]
- Division Of Undergraduate Education
- Direct For Education and Human Resources [1322872, 1347700] Funding Source: National Science Foundation
- Division Of Undergraduate Education
- Direct For Education and Human Resources [1347733, 1322851, 1323011, 1347578, 1322962, 1323162, 1347740, 1347626, 1347729, 1323022] Funding Source: National Science Foundation
Ask authors/readers for more resources
Automated computerized scoring systems (ACSSs) are being increasingly used to analyze text in many educational settings. Nevertheless, the impact of misspelled words (MSW) on scoring accuracy remains to be investigated in many domains, particularly jargon-rich disciplines such as the life sciences. Empirical studies confirm that MSW are a pervasive feature of human-generated text and that despite improvements, spell-check and auto-replace programs continue to be characterized by significant errors. Our study explored four research questions relating to MSW and text-based computer assessments: (1) Do English language learners (ELLs) produce equivalent magnitudes and types of spelling errors as non-ELLs? (2) To what degree do MSW impact concept-specific computer scoring rules? (3) What impact do MSW have on computer scoring accuracy? and (4) Are MSW more likely to impact false-positive or false-negative feedback to students? We found that although ELLs produced twice as many MSW as non-ELLs, MSW were relatively uncommon in our corpora. The MSW in the corpora were found to be important features of the computer scoring models. Although MSW did not significantly or meaningfully impact computer scoring efficacy across nine different computer scoring models, MSW had a greater impact on the scoring algorithms for na < ve ideas than key concepts. Linguistic and concept redundancy in student responses explains the weak connection between MSW and scoring accuracy. Lastly, we found that MSW tend to have a greater impact on false-positive feedback. We discuss the implications of these findings for the development of next-generation science assessments.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available