Designing Monte Carlo implementations of permutation or bootstrap hypothesis tests

This article considers hypothesis testing using resampling or Monte Carlo methods, such as a bootstrap or a permutation procedure, and explores designs for resampling that minimize the expected number of resamples after meeting two constraints. First, we bound the size of the test at the nominal level. Second, we bound the resampling risk, which we define as the expected value of the probability of reaching an accept/reject decision different from complete enumeration. This second bound holds over a postulated set of distributions for the p value, where each distribution is associated with a probability model of the data. In relation to these constraints, we examine the fixed resample size design and two sequential resampling designs, a simple curtailed sampling design, and a new, more complicated design with smaller expected resampling size.