A Note on Cherry-Picking in Meta-Analyses

We study selection bias in meta-analyses by assuming the presence of researchers (meta-analysts) who intentionally or unintentionally cherry-pick a subset of studies by defining arbitrary inclusion and/or exclusion criteria that will lead to their desired results. When the number of studies is sufficiently large, we theoretically show that a meta-analysts might falsely obtain (non)significant overall treatment effects, regardless of the actual effectiveness of a treatment. We analyze all theoretical findings based on extensive simulation experiments and practical clinical examples. Numerical evaluations demonstrate that the standard method for meta-analyses has the potential to be cherry-picked.

Automated summary

We investigate selection bias in meta-analyses by assuming the existence of researchers (meta-analysts) who selectively choose a subset of studies based on arbitrary inclusion and/or exclusion criteria to achieve desired results. Regardless of the actual effectiveness of a treatment, our theoretical analysis shows that meta-analysts can falsely obtain (non)significant overall treatment effects when the number of studies is sufficiently large. We validate our theoretical findings through extensive simulation experiments and practical clinical examples, demonstrating the potential for cherry-picking in standard methods for meta-analyses.