Prior Sensitivity of Null Hypothesis Bayesian Testing

Researchers increasingly use Bayes factor for hypotheses evaluation. There are two main applications: null hypothesis Bayesian testing (NHBT) and informative hypothesis Bayesian testing (IHBT). As will be shown in this article, NHBT is sensitive to the specification of the scale parameter of the prior distribution, while IHBT is not. As will also be shown in this article, for NHBT using four different Bayes factors, use of the recommended default values for the scaling parameters results in unpredictable operating characteristics, that is, the Bayes factor will usually be biased against or in favor of the null hypothesis. As will furthermore be shown in this article, this problem can be addressed by choosing the scaling parameter such that the Bayes factor is 19 in favor of the null hypothesis over the alternative hypothesis if the observed effect size is equal to zero, because this renders a Bayes factor with clearly specified operating characteristics. However, this does not solve all problems regarding NHBT. The discussion of this article contains elaborations with respect to: the multiverse of Bayes factors; the choice of 19; Bayes factor calibration outside the context of the univariate normal linear model; and, reporting the results of NHBT. Translational Abstract Researchers increasingly use Bayes factor for hypotheses evaluation. However, Bayes factors do not actually evaluate hypotheses, they evaluate the prior distributions corresponding to these hypotheses. Loosely formulated, prior distributions represent for each hypotheses how each possible value of the parameters appearing in the hypothesis should be weighted. Whereas researchers using the Bayes factor while analyzing their data find it relatively easy to specify hypotheses, it is often not completely clear how the prior distributions should be specified. As will be shown in this article, in the context of the normal linear model, one solution to this problem is obtained if the Bayes factor of the null-hypothesis versus the alternative hypothesis is calibrated such that it is 19 if the observed effect size equals zero. If this calibration is used, the prior distributions corresponding to each hypothesis can uniquely be determined and do no longer have to be specified by the researchers using Bayes factor. With this article come R functions that can be used to apply the approach prosed in combination with the R package bain (https://informative-hypotheses.sites.uu.nl/software/bain/).

Automated summary

Researchers increasingly use Bayes factor for hypotheses evaluation, with NHBT being sensitive to the specification of the scale parameter of the prior distribution, while IHBT is not. Using recommended default values for scaling parameters in NHBT leads to unpredictable operation characteristics, but selecting the scaling parameter to bias the Bayes factor towards the null hypothesis by 19 if the observed effect size is zero can address this issue in some cases. However, this does not solve all problems associated with NHBT.