Comparing Random Effects Models, Ordinary Least Squares, or Fixed Effects With Cluster Robust Standard Errors for Cross-Classified Data

Cross-classified random effects modeling (CCREM) is a common approach for analyzing cross-classified data in psychology, education research, and other fields. However, when the focus of a study is on the regression coefficients at Level 1 rather than on the random effects, ordinary least squares regression with cluster robust variance estimators (OLS-CRVE) or fixed effects regression with CRVE (FE-CRVE) could be appropriate approaches. These alternative methods are potentially advantageous because they rely on weaker assumptions than those required by CCREM. We conducted a Monte Carlo Simulation study to compare the performance of CCREM, OLS-CRVE, and FE-CRVE in models, including conditions where homoscedasticity assumptions and exogeneity assumptions held and conditions where they were violated, as well as conditions with unmodeled random slopes. We found that CCREM out-performed the alternative approaches when its assumptions are all met. However, when homoscedasticity assumptions are violated, OLS-CRVE and FE-CRVE provided similar or better performance than CCREM. When the exogeneity assumption is violated, only FE-CRVE provided adequate performance. Further, OLS-CRVE and FE-CRVE provided more accurate inferences than CCREM in the presence of unmodeled random slopes. Thus, we recommend two-way FE-CRVE as a good alternative to CCREM, particularly if the homoscedasticity or exogeneity assumptions of the CCREM might be in doubt.Translational AbstractCross-classified data involve observations that are nested within more than one type of higher-level unit, which are not themselves hierarchically nested. For example, if students are included within neighborhoods and schools that are not perfectly nested within each other, data on those students has a cross-classified structure. In psychology, education, and other research areas, a common approach for analyzing cross-classified data is cross-classified random effects modeling (CCREM). However, CCREM is based on strict assumptions. If its assumptions are violated, the estimates from CCREM may be systematically biased and uncertainty assessments from CCREM may be misleading. When the focus of a study is on an individual-level predictor, ordinary least squares regression with cluster robust variance estimators (OLS-CRVE) or fixed effects regression with CRVE (FE-CRVE) offer practical alternatives to CCREM. These alternatives are beneficial in not relying on all the assumptions required by CCREM. In this study, we compared the performance of CCREM, OLS-CRVE, and FE-CRVE in conditions where key assumptions held true and conditions where those assumptions were violated. When individual-level errors have nonconstant variance, OLS-CRVE and FE-CRVE provided similar or better performance than CCREM. When there is correlation between model predictors and random effects, only FE-CRVE showed adequate performance. Further, when the random slopes were incorrectly omitted from the model, OLS-CRVE and FE-CRVE provided a more accurate statistical inference than CCREM. Thus, if there are potential concerns regarding the assumptions of the CCREM, we recommend two-way FE-CRVE as an alternative to CCREM.

Automated summary

Cross-classified random effects modeling (CCREM) is a common approach for analyzing cross-classified data. However, ordinary least squares regression with cluster robust variance estimators (OLS-CRVE) or fixed effects regression with CRVE (FE-CRVE) may be more appropriate approaches when the focus is on Level 1 regression coefficients. We compared the performance of CCREM, OLS-CRVE, and FE-CRVE in different model conditions. CCREM out-performed the alternative approaches when its assumptions are met, but OLS-CRVE and FE-CRVE provided similar or better performance when assumptions were violated.