Performance of the Kenward-Roger method when the covariance structure is selected using AIC and BIC

This study investigated Type I error rates for tests of fixed effects in mixed linear models using Wald F-statistics with the Kenward-Roger adjustment. Data were generated using 15 covariance structures. Correct covariance structures as well as those selected using the Akaike's Information Criterion (AIC) and Schwarz's Bayesian Information Criterion (BIC) criteria were examined. Performance of the AIC and BIC criteria in selecting the true covariance structure was also studied. Type I error rates for the correct models were often adequate depending on the sample size and complexity of covariance structure. Type I error rates for the best AIC and BIC models were always higher than target values, but those obtained using BIC were closer to the target value than those obtained using AIC. For unbalanced data, Type I error rates for the between-subjects effect were closer to target values for positive pairing while those for the within-subject effect were closer for negative pairing. Success of AIC and BIC in selecting the correct covariance structure was low.