Randomization of CATA attributes: Should attribute lists be allocated to assessors or to samples?

For the check-all-that-apply (CATA) question format, it is good practice to vary the attribute list order between evaluations to account for possible (and likely) position bias in the data. If attribute lists are to be randomized, the question is how to allocate these attribute lists orders. Some authors recommend a to samples allocation order, randomizing the attribute list order for each sample presented. Other authors recommend a to assessors allocation, randomizing the attribute list order for each assessor, such that the list order is stable across samples for each assessor (if replication is used, assessors are given a new attribute list order per replication). In this study, consumers (n = 93) performed CATA evaluations on 6 breads twice. The evaluation was done once using the to assessors CATA list order allocation scheme, and once using the to samples CATA list order allocation scheme, with the order of allocation schemes balanced by experimental design. Results suggest higher operational power when using the to assessor CATA attribute list order. This conclusion is supported by theoretical considerations. (C) 2015 Elsevier Ltd. All rights reserved.