Psychometric properties of a simple measure of conceptual knowledge: The concept retrieval technique

The objective of the present study was to establish reliability and validity evidence of a simple procedure to measuring conceptual knowledge, the Concept Retrieval Technique (CRT). The CRT procedure requires test takers to freely retrieve concepts from memory they feel are relevant to a given topic, with raters awarding one mark for each correctly retrieved concept when matched against a target word list. Four studies were conducted. In Study 1 (N = 73), inter-rater agreement for the marking procedure of the CRT was determined by means of Cohen's kappa. The kappa was kappa = .85. Study 2 (N = 164) explored how consistent inter-rater agreement was across different subjects, age groups, and raters. Mean kappa was kappa = .92 suggesting that the CRT can reliably be marked. Study 3 was concerned with establishing the convergent and discriminant validity of the CRT (N = 55). The correlation between the CRT scores and scores on essay-type items was r = .69 and between assessments that measured other ability domains r = .04 to .21 (n.s.). In Study 4, an experiment was conducted to determine the construct validity of the CRT (N = 45). Participants either acquired, or did not acquire, new knowledge of a particular topic and this manipulation was consistently reflected in their CRT scores. The results of the studies suggest that the CRT is a reliable and valid procedure to measure conceptual knowledge. Implications for education are discussed.

Automated summary

The objective of this study was to establish the reliability and validity of the Concept Retrieval Technique (CRT) for measuring conceptual knowledge. Through four studies, it was shown that the CRT is a reliable and valid procedure for measuring conceptual knowledge.