The Reliability Factor: Modeling Individual Reliability with Multiple Items from a Single Assessment

Reliability is a crucial concept in psychometrics. Although it is typically estimated as a single fixed quantity, previous work suggests that reliability can vary across persons, groups, and covariates. We propose a novel method for estimating and modeling case-specific reliability without repeated measurements or parallel tests. The proposed method employs a Reliability Factor that models the error variance of each case across multiple indicators, thereby producing case-specific reliability estimates. Additionally, we use Gaussian process modeling to estimate a nonlinear, non-monotonic function between the latent factor itself and the reliability of the measure, providing an analogue to test information functions in item response theory. The reliability factor model is a new tool for examining latent regions with poor conditional reliability, and correlates thereof, in a classical test theory framework.

Automated summary

This study proposes a novel method for estimating and modeling case-specific reliability without repeated measurements or parallel tests. By using a reliability factor model and Gaussian process modeling, the study provides insights into the variability of reliability and its relationship with latent factors.