Beyond statistical significance: nuanced interpretations of statistically nonsignificant results were rare in Cochrane reviews- a metaepidemiological study

Objectives: To assess the language used by systematic review authors to emphasize that statistically nonsignificant results show mean-ingful differences. To determine whether the magnitude of these treatment effects was distinct from nonsignificant results that authors in-terpreted as not different. Study Design and Setting: We screened Cochrane reviews published between 2017 and 2022 for statistically nonsignificant effect es-timates that authors presented as meaningful differences. We classified interpretations qualitatively and assessed them quantitatively by calculating the areas under the curve of the portions of confidence intervals exceeding the null or a minimal important difference, indicating one intervention's greater effect. Results: In 2,337 reviews, we detected 139 cases where authors emphasized meaningful differences in nonsignificant results. Authors commonly used qualifying words to express uncertainty (66.9%). Sometimes (26.6%), they made absolute claims about one intervention's greater benefit or harm without acknowledging statistical uncertainty. The areas under the curve analyses indicated that some authors may overstate the importance of nonsignificant differences, whereas others may overlook meaningful differences in nonsignificant effect estimates. Conclusion: Nuanced interpretations of statistically nonsignificant results were rare in Cochrane reviews. Our study highlights the need for a more nuanced approach by systematic review authors when interpreting statistically nonsignificant effect estimates. & COPY; 2023 The Author(s). Published by Elsevier Inc. This is an open access article under the CC BY license (http://creativecommons.org/licenses/ by/4.0/).

Automated summary

The study examines the language used by systematic review authors to emphasize the meaningful differences shown by statistically nonsignificant results. It also compares the magnitude of these treatment effects with nonsignificant results that authors interpreted as not different. The findings suggest that some authors may overstate the importance of nonsignificant differences, while others may overlook meaningful differences.