Can we individualize the number needed to treat? An empirical study of summary effect measures in meta-analyses

Background Meta-analyses summarize the magnitude of treatment effect using a number of measures of association, including the odds ratio (OR), risk ratio (RR), risk difference (RD) and/or number needed to treat (NNT). In applying the results of a meta-analysis to individual patients, some textbooks of evidence-based medicine advocate individualizing NNT, based on the RR and the patient's expected event rate (PEER). This approach assumes constant RR but no empirical study to date has examined the validity of this assumption. Methods We randomly selected a subset of meta-analyses from a recent issue of the Cochrane Library (1998, Issue 3). When a meta-analysis pooled more than three randomized controlled trials (RCT) to produce a summary measure for an outcome, we compared the OR, RR and RD of each RCT with the corresponding pooled OR, RR and RD from the meta-analysis of all the other RCT. Using the conventional P-value of 0.05, we calculated the percentage of comparisons in which there were no statistically significant differences in the estimates of OR, RR or RD, and refer to this percentage as the 'concordance rate'. Results For each effect measure, we made 1843 comparisons, extracted from 55 meta-analyses. The random effects model OR had the highest concordance rate, closely followed by the fixed effects model OR and random effects model RR. The minimum concordance rate for these indices was 82%, even when the baseline risk differed substantially. The concordance rates for RD, either fixed effects or random effects model, were substantially lower (54-65%). Conclusions The fixed effects OR, random effects OR and random effects RR appear to be reasonably constant across different baseline risks. Given the interpretational and arithmetic ease of RR, clinicians may wish to rely on the random effects model RR and use the PEER to individualize NNT when they apply the results of a meta-analysis in their practice.