Assessing observational studies of medical treatments

Background: Previous studies have assessed the validity of the observational study design by comparing results of studies using this design to results from randomized controlled trials. The present study examined design features of observational studies that could have influenced these comparisons. Methods: To find at least 4 observational studies that evaluated the same treatment, we reviewed meta-analyses comparing observational studies and randomized controlled trials for the assessment of medical treatments. Details critical for interpretation of these studies were abstracted and analyzed qualitatively. Results: Individual articles reviewed included 61 observational studies that assessed 10 treatment comparisons evaluated in two studies comparing randomized controlled trials and observational studies. The majority of studies did not report the following information: details of primary and ancillary treatments, outcome definitions, length of follow-up, inclusion/exclusion criteria, patient characteristics relevant to prognosis or treatment response, or assessment of possible confounding. When information was reported, variations in treatment specifics, outcome definition or confounding were identified as possible causes of differences between observational studies and randomized controlled trials, and of heterogeneity in observational studies. Conclusion: Reporting of observational studies of medical treatments was often inadequate to compare study designs or allow other meaningful interpretation of results. All observational studies should report details of treatment, outcome assessment, patient characteristics, and confounding assessment.