Statistical reproducibility for pairwise t-tests in pharmaceutical research

This paper investigates statistical reproducibility of the t-test. We formulate reproducibility as a predictive inference problem and apply the nonparametric predictive inference method. Within our research framework, statistical reproducibility provides inference on the probability that the same test outcome would be reached, if the test were repeated under identical conditions. We present an nonparametric predictive inference algorithm to calculate the reproducibility of the t-test and then use simulations to explore the reproducibility both under the null and alternative hypotheses. We then apply nonparametric predictive inference reproducibility to a real-life scenario of a preclinical experiment, which involves multiple pairwise comparisons of test groups, where different groups are given a different concentration of a drug. The aim of the experiment is to decide the concentration of the drug which is most effective. In both simulations and the application scenario, we study the relationship between reproducibility and two test statistics, the Cohen's d and the p-value. We also compare the reproducibility of the t-test with the reproducibility of the Wilcoxon Mann-Whitney test. Finally, we examine reproducibility for the final decision of choosing a particular dose in the multiple pairwise comparisons scenario. This paper presents advances on the topic of test reproducibility with relevance for tests used in pharmaceutical research.

Automated summary

This paper investigates the statistical reproducibility of the t-test using nonparametric predictive inference method, both in simulations and a real-life preclinical experiment scenario. It studies the relationship between test statistics and reproducibility, compares the reproducibility of t-test and Mann-Whitney test, and examines reproducibility in the final decision of choosing a specific dose in multiple pairwise comparisons.