How objective are systematic reviews? Differences between reviews on complementary medicine

Systematic reviews are considered the most reliable tool to summarize existing evidence. To determine whether reviews that address the same questions can produce different answers we examined systematic reviews of herbal medicine, homeopathy, and acupuncture taken from a previously established database. Information on literature searching, inclusion criteria, selection process, quality assessment, data extraction, methods to summarize primary studies, number of included studies, results and conclusions was compared qualitatively. Seventeen topics (eight on acupuncture, six on herbal medicines, three on homeopathy) had been addressed by 2-5 systematic reviews each. The number of primary studies in the reviews varied greatly within most topics. The most obvious reason for discrepancies between the samples was different inclusion criteria (in thirteen topics). Methods of literature searching may have contributed with some topics but the equivalence of the searches was difficult to assess. Differences were frequently observed in other methodological aspects, in results and in conclusions. This analysis shows that, at least in the three areas examined, systematic reviews often differ considerably. Readers should be aware that apparently minor decisions in the review process can have major impact.