Efficient Selection Between Hierarchical Cognitive Models: Cross-Validation With Variational Bayes

Model comparison is the cornerstone of theoretical progress in psychological research. Common practice overwhelmingly relies on tools that evaluate competing models by balancing in-sample descriptive adequacy against model flexibility, with modern approaches advocating the use of marginal likelihood for hierarchical cognitive models. Cross-validation is another popular approach but its implementation remains out of reach for cognitive models evaluated in a Bayesian hierarchical framework, with the major hurdle being its prohibitive computational cost. To address this issue, we develop novel algorithms that make variational Bayes (VB) inference for hierarchical models feasible and computationally efficient for complex cognitive models of substantive theoretical interest. It is well known that VB produces good estimates of the first moments of the parameters, which gives good predictive densities estimates. We thus develop a novel VB algorithm with Bayesian prediction as a tool to perform model comparison by cross-validation, which we refer to as CVVB. In particular, CVVB can be used as a model screening device that quickly identifies bad models. We demonstrate the utility of CVVB by revisiting a classic question in decision making research: what latent components of processing drive the ubiquitous speed-accuracy tradeoff? We demonstrate that CVVB strongly agrees with model comparison via marginal likelihood, yet achieves the outcome in much less time. Our approach brings cross-validation within reach of theoretically important psychological models, making it feasible to compare much larger families of hierarchically specified cognitive models than has previously been possible. To enhance the applicability of the algorithm, we provide Matlab code together with a user manual so users can easily implement VB and/or CVVB for the models considered in this article and their variants. Translational Abstract Progress in psychological science can be made by choosing between competing theories: Does sleep deprivation cause attentional lapses? Does alcohol impair the speed of information processing or reduce cautiousness, or both? When these theories are quantitative models that can be estimated from observed data, the problem is known as statistical model selection. Cross-validation is a desirable method for model selection as it directly addresses the question that scientists are often interested in: how well will this model predict new data? However, cross-validation is not used very widely in psychological science due to its computational cost. Modern psychological theories often include random effects, which are necessary to account for individual differences, but these can mean that Bayesian evaluation of the model can take days of computer time. This makes repeated evaluation for cross-validation impractical. Another practical limitation arises when comparing a very large number of competing hypotheses, and this can amplify issues of researcher bias: researchers are forced to focus on a limited subset of models, which they identify by idiosyncratic means. We propose cross-validation with variational Bayes (CVVB) to tackle these issues. CVVB allows researchers to address larger sets of model-based hypotheses, even using modern multilevel theories. As a demonstration, we use CVVB to investigate an important question about strategies for trading speed versus caution in simple decision-making.

Automated summary

Model comparison is crucial in psychological research, but existing methods have limitations in terms of computational cost and the number of models that can be compared. This paper proposes a novel algorithm, CVVB, which combines variational Bayes inference and Bayesian prediction for cross-validation, to address these issues. The results show that CVVB agrees strongly with model comparison using marginal likelihood but requires less time, enabling researchers to compare larger families of hierarchically specified cognitive models.