Automated and interactive game-based assessment of critical thinking

Critical thinking is one of the important higher-order skills very much treasured in education, but hard to be measured using paper-pencil tests. In line with recent recommendation to measure high-order thinking skills with interactive tasks (vs. static one set of questions), in this study we developed an interactive and automated game-based assessment of critical thinking, using the Toulmin Model. In two real-life simulation stories, through interactive tasks in progressing scenes in the stories, students chose and rated evidence and conflicting reasons as supportive or non-supportive arguments in making the eventual decision. Critical thinking scores were awarded on choosing the appropriate evidence and reasons. The psychometric quality of the game was evaluated with 185 Chinese senior secondary students. Results showed that (i) reliabilities as measured by Cronbach's alpha of the whole scale and subdomains were reasonable; (ii) parallel form reliability was high; (iii) its correlation and convergent validity with the popular Halpern Critical Thinking Assessment were comparable to those in other studies; (iv) it was generally not related to academic performance; and (v) the game was interesting and engaging. We also noted that students hesitated to query others and they were weak in applying critical thinking to problem-solving, which were in congruent with previous research showing students rarely used critical thinking to solve complex, real-world problems. In sum, we demonstrated successfully the use of interactive simulation tasks in measuring critical thinking. With the advancement of technology, our study suggested the possibility of assessing hard-to-measure important complex higher-order competence with dynamic games.

Automated summary

The study successfully demonstrated the use of interactive simulation tasks in measuring critical thinking. Results showed reasonable reliability and validity, high student engagement with the game, but also highlighted students' hesitation in questioning others and applying critical thinking to problem-solving.