Effect sizes in ANCOVA and difference-in-differences designs

It is common practice in both randomized and quasi-experiments to adjust for baseline characteristics when estimating the average effect of an intervention. The inclusion of a pre-test, for example, can reduce both the standard error of this estimate and-in non-randomized designs-its bias. At the same time, it is also standard to report the effect of an intervention in standardized effect size units, thereby making it comparable to other interventions and studies. Curiously, the estimation of this effect size, including covariate adjustment, has received little attention. In this article, we provide a framework for defining effect sizes in designs with a pre-test (e.g., difference-in-differences and analysis of covariance) and propose estimators of those effect sizes. The estimators and approximations to their sampling distributions are evaluated using a simulation study and then demonstrated using an example from published data.

Automated summary

It is common practice to adjust for baseline characteristics in estimating intervention effects in both randomized and quasi-experiments. Including a pre-test can reduce standard error and bias in non-randomized designs. However, the estimation of effect sizes, including covariate adjustment, has been given little attention. This article provides a framework and estimators for defining effect sizes in designs with a pre-test and evaluates them using simulations and real data.