Research design influence on study outcomes in crime and justice: a partial replication with public area surveillance

Does the quality of research design have an influence on study outcomes in crime and justice? This was the subject of an important study by Weisburd et al. (2001). They found a moderate and significant inverse relationship between research design and study outcomes: weaker designs, as indicated by internal validity, produced stronger effect sizes. Using a database of evaluations (n=136) from systematic reviews that investigated the effects of public area surveillance on crime, this paper carried out a partial replication of Weisburd et al.'s study. We view it as a partial replication because it included only area-or place-based studies (i.e., there were no individual-level studies) and these studies used designs at the lower end of the evaluation hierarchy (i.e., not one of the studies used a randomized experimental design). In the present study, we report findings that are highly concordant with the earlier study. The overall correlation between research design and study outcomes is moderate but negative and significant (Tau-b= -.175, p=.029). This suggests that stronger research designs are less likely to report desirable effects or, conversely, weaker research designs may be biased upward. We explore possible explanations for this finding. Implications for policy and research are discussed.