Cognitive Foundations for Science Assessment Design: Knowing What Students Know About Evolution

To improve assessments of academic achievement, test developers have been urged to use an assessment triangle that starts with research-based models of cognition and learning [NRC (2001) Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press]. This approach has been successful in designing high-quality reading and math assessments, but less progress has been made for assessments in content-rich sciences such as biology. To rectify this situation, we applied the assessment triangle to design and evaluate new items for an instrument (ACORNS, Assessing Contextual Reasoning about Natural Selection) that had been proposed to assess students' use of natural selection to explain evolutionary change. Design and scoring of items was explicitly guided by a cognitive model that reflected four psychological principles: with development of expertise, (1) core concepts facilitate long-term recall, (2) causally-central features become weighted more strongly in explaining phenomena, (3) normative ideas co-exist but increasingly outcompete naive ideas in reasoning, and (4) knowledge becomes more abstract and less specific to the learning situation. We conducted an evaluation study with 320 students to examine whether scores from our new ACORNS items could detect gradations of expertise, provide insight into thinking about evolutionary change, and predict teachers' assessments of student achievement. Findings were consistent with our cognitive model, and ACORNS was revealing about undergraduates' thinking about evolutionary change. Results indicated that (1) causally-central concepts of evolution by natural selection typically co-existed and competed with the presence of naive ideas in all students' explanations, with naive ideas being especially prevalent in low-performers' explanations; (2) causally-central concepts were elicited most frequently when students were asked to explain evolution of animals and familiar plants, with influence of superficial features being strongest for low-performers; and (3) ACORNS scores accurately predicted students' later achievement in a college-level evolution course. Together, findings illustrate usefulness of cognitive models in designing instruments intended to capture students' developing expertise. (c) 2012 Wiley Periodicals, Inc. J Res Sci Teach 49: 744-777, 2012