The impact of response options ordering on respondents' answers to rating questions: results of two experiments

This paper aims to explore the response-order effects for rating questions presented in item-by-item and grid formats. It was hypothesized that the primacy effect occurs for both formats of questions, and that this effect is dependent on age, education, and type of device used for responding to questions. Two randomized experiments were conducted in 28 pre-course surveys of massive open online course students (N = 22,910). Our findings suggest that the order of response options affects respondents' perception of the option lists and their responding patterns. The primacy effect is found for the item-by-item question, while there is no evidence for the presence of such an effect for the grid question format. Primacy effect for the item-by-item layout is lower for respondents with higher education degree while there are no interaction effects between ordering and age, gender, and type of device. For a grid question, mixed results were observed.