Methods for testing publication bias in ecological and evolutionary meta-analyses

Publication bias threatens the validity of quantitative evidence from meta-analyses as it results in some findings being overrepresented in meta-analytic datasets because they are published more frequently or sooner (e.g. 'positive' results). Unfortunately, methods to test for the presence of publication bias, or assess its impact on meta-analytic results, are unsuitable for datasets with high heterogeneity and non-independence, as is common in ecology and evolutionary biology. We first review both classic and emerging publication bias tests (e.g. funnel plots, Egger's regression, cumulative meta-analysis, fail-safe N, trim-and-fill tests, p-curve and selection models), showing that some tests cannot handle heterogeneity, and, more importantly, none of the methods can deal with non-independence. For each method, we estimate current usage in ecology and evolutionary biology, based on a representative sample of 102 meta-analyses published in the last 10 years. Then, we propose a new method using multilevel meta-regression, which can model both heterogeneity and non-independence, by extending existing regression-based methods (i.e. Egger's regression). We describe how our multilevel meta-regression can test not only publication bias, but also time-lag bias, and how it can be supplemented by residual funnel plots. Overall, we provide ecologists and evolutionary biologists with practical recommendations on which methods are appropriate to employ given independent and non-independent effect sizes. No method is ideal, and more simulation studies are required to understand how Type 1 and Type 2 error rates are impacted by complex data structures. Still, the limitations of these methods do not justify ignoring publication bias in ecological and evolutionary meta-analyses.

Automated summary

Publication bias poses a significant threat to the validity of quantitative evidence in meta-analyses, with some findings being overrepresented due to publication frequency or timing. Current methods to detect publication bias are inadequate for datasets with high heterogeneity and non-independence, common in ecology and evolutionary biology. A new multilevel meta-regression method is proposed to address these challenges and provide practical recommendations for researchers in the field.