Type S error rates for classical and Bayesian single and multiple comparison procedures

Fn classical statistics, the significance of comparisons (e.g., theta(1) - theta(2)) is calibrated using the Type 1 error rate, relying on the assumption that the true difference is zero, which makes no sense in many applications. We set up a more relevant framework in which a true comparison can be positive or negative, and, based on the data, you can state theta(1) > theta(2) with confidence, theta(2) > theta(1) with confidence, or no claim with confidence. We focus on the Type S (for sign) error, which occurs when you claim theta(1) > theta(2) with confidence when theta(2) > theta(1) (or vice-versa). We compute the Type S error rates for classical and Bayesian confidence statements and find that classical Type S error rates can be extremely high (up to 50%). Bayesian confidence statements are conservative, in the sense that claims based on 95% posterior intervals have Type S error rates between 0 and 2.5%. For multiple comparison situations, the conclusions are similar.