Randomized controlled trials published on patients with hernias have a high percentage of unreproducible statistics

Introduction Surgeons rely on randomized controlled trials (RCT) to compare the effectiveness of treatments. RCTs require careful planning and substantial effort to complete. Because of the careful study design, statistics performed are often easy to reproduce such as Chi-squared or t-test. Issues such as statistical discordance, or reporting statistical results that cannot be reproduced, should be uncommon. Methods RCTs pertaining to hernias were identified in PubMed using the search terms hernia and randomized controlled trial. Studies were selected using a random number generator. Studies were included if the primary outcome could be reproduced using the data and statistical test reported in the manuscript. Discordance between the obtained p-value from our analysis and the published p-value was assessed. Primary outcome was the number of studies that reported p-values that crossed the level of statistical significance (p-value = 0.05) but on reproduction analysis did not. Results Of the 100 included RCTs, five reported p-values that crossed the p = 0.05 threshold that our team was unable to reproduce using the statistical test reported in the manuscript. An additional three studies reported p-values that crossed the p = 0.05 threshold that our team was unable to reproduce using the appropriate statistical test (i.e., Fisher's exact test when all expected cell counts < 5). All eight studies published p-values < 0.05, whereas, our re-analysis demonstrated p >= 0.05. Conclusion Eight percent of the RCTs analyzed in this study reported p-values < 0.05 that on reproduction analysis was >= 0.05. The next steps should be to determine reasons for discordance and how to prevent this from happening.

Automated summary

In this study, 8% of the analyzed RCTs reported p-values < 0.05 that were >= 0.05 on reproduction analysis. The next steps should be to determine reasons for discordance and how to prevent this from happening.