Choice of a null distribution in resampling-based multiple testing

This paper investigates different choices of test statistic null distribution for resampling-based multiple testing in the context of single parameter hypotheses. We show that the test statistic null distribution for strongly controlling type I error may be obtained by projecting the true test statistic distribution onto the space of mean zero distributions. For common choices of test statistics, this distribution is asymptotically multivariate normal with the covariance of the vector influence curve for the parameter estimator. Applying the ordinary non-parametric or model-based bootstrap to mean zero centered test statistics produces an estimated test statistic null distribution which provides asymptotic strong control. In contrast, the usual practice of obtaining an estimated test statistic null distribution via an estimated data null distribution (e.g. null restricted bootstrap) only provides an asymptotically correct test statistic null distribution if the covariance of the vector influence curve is the same under the chosen data null distribution as under the true data distribution. This condition is the formal analogue of the subset pivotality condition (Westfall and Young, Resampling-based Multiple Testing: Examples and Methods for p-value adjustment, Wiley, New York, 1993). We demonstrate the use of our proposed ordinary bootstrap null distribution with a single-step multiple testing method which is equivalent to constructing an error-specific confidence region for the true parameter and checking if it contains the hypothesized value. We also study the two sample problem and show that the permutation method produces an asymptotically correct null distribution if (i) the sample sizes are equal or (ii) the populations have the same covariance structure. (C) 2004 Elsevier B.V. All rights reserved.