Experimental Design in Marketplaces

Classical Randomized Controlled Trials (RCTs), or A/B tests, are designed to draw causal inferences about a population of units, for example, individuals, plots of land or visits to a website. A key assumption under-lying a standard RCT is the absence of interactions between units, or the stable unit treatment value assumption (Ann. Statist. 6 (1978) 34-58). Mod-ern experimentation, however, is often conducted in settings characterized by complex interactions between units. Such interactions can invalidate the standard estimators and make classical experimental designs ineffective. Al-though the presence of interference forces us to make untestable assumptions on the nature of the interactions even under randomization, sophisticated experimental designs can ameliorate the dependence on such assumptions. In this manuscript, we review the recent and rapidly growing literature on novel experimental designs for these settings. One key feature common to many of these designs is the presence of multiple layers of randomization within the same experiment. We discuss a novel experimental design, called Multiple Randomization Designs or MRDs, that provides a general framework for such experiments. Through these complex designs, we can study questions about causal effects in the presence of interference that cannot be answered by classical RCTs.

Automated summary

Classical Randomized Controlled Trials (RCTs) are designed to draw causal inferences, but modern experiments often involve complex interactions between units, making classical designs ineffective. In this manuscript, we review novel experimental designs, such as Multiple Randomization Designs (MRDs), that allow for the study of causal effects in the presence of interference.