Evaluating the Equivalence of, or Difference Between, Psychological Treatments: An Exploration of Recent Intervention Studies

In behavioral science research there is often the need to determine if an outcome variable differs, or is equivalent, across groups. Significance tests are the most prevalently applied data analysis method for this type of question. The purpose of this study was to examine how statistical tests for equivalence and difference have been applied to compare clinical interventions. Peer-reviewed journal articles that made treatment comparisons were examined. For each study, the primary hypothesis, statistical test usage, and the stated conclusion were recorded. Of the 270 studies investigated, 54.4% inappropriately made equivalence-based conclusions from difference-based test statistics (e.g., t test, ANOVA). Significance tests are often applied as a matter of course regardless of the research question. We have found that difference tests are similarly favored and have been applied to examine difference and inappropriately applied to examine equivalence. We discuss our findings and provide resources for researchers who want to statistically evaluate between-groups equivalence.