Model Specification for Nonlinearity and Heterogeneity of Regression in Randomized Pretest Posttest Studies: Practical Solutions for Missing Data

The randomized pretest posttest design is common in psychology, as is the corresponding missing data concern. Although missing data handling has seen advances over the past several decades, effective and practical solutions for handling missing data in randomized pretest posttest designs are lacking, particularly when assumptions of commonly used statistical models are violated. Although analysis of covariance can capture the average treatment effect with complete data, even when assumptions are tenuous, this becomes more difficult with missing data. This investigation fills this gap in the literature by comparing a variety of analysis models for estimating the average treatment effect under violations of linearity and homogeneity of regression slopes, when data are missing by several plausible, but understudied, missing at random patterns for randomized pretest posttest studies. Two missing data handling techniques, listwise deletion and multiple imputation, were considered. Listwise deletion provided maximum likelihood estimates (unbiased and appropriately precise) of the average treatment effect as long as the analysis model was appropriately specified to handle the violated assumption and the pretest mean was estimated using all cases. Although multiple imputation was effective as long as the imputation model was correct, the results highlight to the importance of model specification in the context of missing data. Importantly, the specific pattern of missing at random data had implications for results, emphasizing the need to consider the particular pattern of missingness beyond the general appropriateness of the missing at random assumption. Translational Abstract Researchers often use randomized pretest posttest designs to test treatment effectiveness. The dependent variable is measured at pretest and participants are randomly placed into either the treatment or control group. Following treatment, the dependent variable is measured at posttest. However, it is not uncommon for participants to drop out before posttest, leading to missing data. Appropriate solutions for dealing with this in these types of designs are lacking, especially when the assumptions of the models used to analyze such data are violated. Analysis of covariance, the model typically used to analyze such data, may not work well when there is missing data and its assumptions are violated. This investigation compared a variety of models for testing the treatment effect under violations of linearity (pretest and posttest have a linear association) and homogeneity of regression slopes (pretest and posttest are related in the same way for the treatment and control groups), when participants dropped out based on their pretest score, group membership, or a combination of both. Additionally, dropping missing cases (listwise deletion) was compared with multiple imputation, which fills in missing scores using a variety of algorithms. Listwise deletion worked well as long as (a) the analysis model was one that could handle the violated assumption, and (b) the mean of pretest was measured using all scores, not only those participants who stayed in the study. The effectiveness of multiple imputation relied on using a correct imputation model. Finally, the specific nature of how participants dropped out influenced results, emphasizing the need to consider more specific dropout mechanisms for these types of studies.

Automated summary

This investigation compared various analysis models for estimating the average treatment effect in randomized pretest posttest designs when assumptions of commonly used statistical models are violated. The study found that listwise deletion and multiple imputation are effective techniques for handling missing data. The specific pattern of missing at random data had implications for results, emphasizing the need to consider the particular pattern of missingness beyond the general appropriateness of the missing at random assumption.