Quality of reporting of experimental studies in medical education: a systematic review

OBJECTIVE Determine the prevalence of essential elements of reporting in experimental studies in medical education. DESIGN Systematic review. DATA SOURCES Articles published in 2003 and 2004 in Academic Medicine, Advances in Health Sciences Education, American journal of Surgery, journal of General Internal Medicine, Medical Education, and Teaching and Learning in Medicine. REVIEW METHODS Articles describing education experiments, including evaluation studies with experimental designs, were identified (n = 185) by reviewing titles and abstracts. A random sample (n = 110) was selected for full review. The full text of each article was evaluated for the presence of guideline-based features of quality reporting: a critical literature review, conceptual framework, statement of study intent (e.g. aim, research question, or hypothesis), statement of study design, definition of main intervention and comparison intervention or control group, and consideration of human subject rights. RESULTS Of the 105 articles suitable for review, 47 (45%) contained a critical literature review and 58 (55%) presented a conceptual framework. A statement of study intent was present in 80 articles (76%), among which the independent and dependent variables were operationally defined in 38 (47%) and 26 articles (32%), respectively. A total of 17 articles (16%) contained an explicit study design statement. Among the 48 studies with a comparison group, 35 (73%) clearly defined the comparison intervention or control group. Institutional review board approval or participant consent was reported in 44 articles (42%). CONCLUSIONS The quality of reporting of experimental studies in medical education was generally poor. Criteria are proposed as a starting point for establishing reporting standards for medical education research.