The effects of clinical and statistical heterogeneity on the predictive values of results from meta-analyses

Variance between studies in a meta-analysis will exist. This heterogeneity may be of clinical, methodological or statistical origin. The last of these is quantified by the I-2-statistic. We investigated, using simulated studies, the accuracy of I-2 in the assessment of heterogeneity and the effects of heterogeneity on the predictive value of meta-analyses. The relevance of quantifying I-2 was determined according to the likely presence of heterogeneity between studies (low, high, or unknown) and the calculated I-2 (low or high). The findings were illustrated by published meta-analyses of selective digestive decontamination and weaning protocols. As expected, I-2 increases and the likelihood of drawing correct inferences from a meta-analysis decreases with increasing heterogeneity. With low levels of heterogeneity, I-2 does not appear to be predictive of the accuracy of the meta-analysis result. With high levels of heterogeneity, even meta-analyses with low I-2-values have low predictive values. Most commonly, the level of heterogeneity in a meta-analysis will be unknown. In these scenarios, I-2 determination may help to identify estimates with low predictive values (high I-2). In this situation, the results of a meta-analysis will be unreliable. With low I-2-values and unknown levels of heterogeneity, predictive values of pooled estimates may range extensively, and findings should be interpreted with caution. In conclusion, quantifying statistical heterogeneity through I-2-statistics is only helpful when the amount of clinical heterogeneity is unknown and I-2 is high. Objective methods to quantify the levels of clinical and methodological heterogeneity are urgently needed to allow reliable determination of the accuracy of meta-analyses.