RCTs to Scale: Comprehensive Evidence From Two Nudge Units

Nudge interventions have quickly expanded from academic studies to larger implementation in so-called Nudge Units in governments. This provides an opportunity to compare interventions in research studies, versus at scale. We assemble a unique data set of 126 RCTs covering 23 million individuals, including all trials run by two of the largest Nudge Units in the United States. We compare these trials to a sample of nudge trials in academic journals from two recent meta-analyses. In the Academic Journals papers, the average impact of a nudge is very large-an 8.7 percentage point take-up effect, which is a 33.4% increase over the average control. In the Nudge Units sample, the average impact is still sizable and highly statistically significant, but smaller at 1.4 percentage points, an 8.0% increase. We document three dimensions which can account for the difference between these two estimates: (i) statistical power of the trials; (ii) characteristics of the interventions, such as topic area and behavioral channel; and (iii) selective publication. A meta-analysis model incorporating these dimensions indicates that selective publication in the Academic Journals sample, exacerbated by low statistical power, explains about 70 percent of the difference in effect sizes between the two samples. Different nudge characteristics account for most of the residual difference.

Automated summary

Nudge interventions have been widely implemented in both academic studies and government units. However, there are significant differences in the impact of nudges between these two settings. This study compares data from 126 randomized controlled trials (RCTs) conducted by Nudge Units and academic journals, and identifies three factors contributing to the differences: statistical power, characteristics of the interventions, and selective publication. The findings suggest that selective publication and low statistical power are the major contributors to the disparities, while variation in nudge characteristics explains the remaining differences.