The saw-toothed behavior of power versus sample size and software solutions: Single binomial proportion using exact methods

With the speed of modem computers it is now feasible to replace normal approximations with exact methods. For continuous random variables it is intuitively clear that for a given significance level and alternative hypothesis the power function increases monotonically as the sample size increases. One would expect the same to be true for discrete random variables, but such is not the case. This article demonstrates in the simple case of a test of a single binomial proportion compared to a hypothesized standard that the power function has a saw-toothed shape and hence is not monotonically increasing. We demonstrate this using the software package nQuery Advisor 4.0, the version that has the capability of doing power calculations using exact binomial methods. We compare that software with competitors that also do exact binomial calculations. The reason for this counter-intuitive result is that a significance level of 0.05, for example, cannot be obtained exactly for most sample sizes. So by a 0.05 significance level for a given sample size n we mean the largest level below 0.05. That level determines the critical value for the test statistic and the critical value determines the power at any given alternative. If n is such that the exact significance level is appreciably lower than 0.05 (say 0.01) and the power is 80% at a particular alternative, the power could easily drop below 80% for n + 1 and the same alternative hypothesis, since the exact significance level does not remain at 0.01 but may stay below but get closer to 0.05. This is particularly the case for small n. We find this result to be important in determining sample size for our clinical trials. We often have endpoints that are proportions and we use fixed sample size trials to compare our device's performance to a control or an objective performance criterion. We illustrate this result in the case of a single binomial proportion, but this property really depends only on the discrete nature of the sampling distribution for the test statistic. Hence it also occurs for permutation tests and in the comparison of the difference or ratio of two binomials.