Avoiding Bias From Sum Scores in Growth Estimates: An Examination of IRT-Based Approaches to Scoring Longitudinal Survey Responses

A huge portion of what we know about how humans develop, learn, behave, and interact is based on survey data. Researchers use longitudinal growth modeling to understand the development of students on psychological and social-emotional learning constructs across elementary and middle school. In these designs, students are typically administered a consistent set of self-report survey items across multiple school years, and growth is measured either based on sum scores or scale scores produced based on item response theory (IRT) methods. Although there is great deal of guidance on scaling and linking IRT-based large-scale educational assessment to facilitate the estimation of examinee growth, little of this expertise is brought to bear in the scaling of psychological and social-emotional constructs. Through a series of simulation and empirical studies, we produce scores in a single-cohort repeated measure design using sum scores as well as multiple IRT approaches and compare the recovery of growth estimates from longitudinal growth models using each set of scores. Results indicate that using scores from multidimensional IRT approaches that account for latent variable covariances over time in growth models leads to better recovery of growth parameters relative to models using sum scores and other IRT approaches. Translational Abstract A huge portion of what we know about how humans develop, learn, behave, and interact is based on survey data. In particular, researchers use growth modeling to understand the development of students on psychological and social-emotional learning constructs across elementary and middle school, including how to support that development. In these designs, students are typically administered a consistent set of survey items across multiple school years, and growth is estimated either based on scores that simply total the item responses or scale scores produced using statistical models. Little is known about how these different approaches to scoring longitudinal survey data impact our understanding of how students develop psychologically and social-emotionally. We examine that question by simulating student longitudinal survey data with known growth properties, and by conducing similar analyses with real-world growth mindset data. We find that the scoring approach is very consequential for understanding student development.

Automated summary

A large amount of our knowledge about human development, learning, behavior, and interaction is derived from survey data. Researchers utilize longitudinal growth modeling to study the development of students in terms of psychological and social-emotional learning constructs during elementary and middle school. Through simulations and empirical studies, it has been found that using multidimensional item response theory (IRT) approaches that take into account latent variable covariances over time leads to better estimation of growth parameters in longitudinal growth models.