Significance testing in ecological null models

In the past decade, the use of null models has become widespread in the testing of ecological theory. Along with increasing usage, null models have also become more complex particularly with regard to tests of significance. Despite the complexity, there are essentially only two distinct ways in which tests of significance are conducted. Direct tests derive a p value directly from the null distribution of a test statistic, as the proportion of the distribution more extreme than the observed value of the test statistic. Indirect tests compare an observed value of a parameter to a null distribution by conducting an additional analysis such as a chi-square test, Kolmogorov-Smirnov test, or regression, although in many cases, this additional step is not necessary. Many kinds of indirect tests require that the null distribution is normal whereas direct tests carry no assumptions about the form of the null distribution. Therefore, when assumptions are violated, indirect tests may have higher type I and II error rates than their counterpart direct tests. A review of 108 null model papers revealed that direct tests were used in 56.5% of studies and indirect tests used in 45.5%. A few studies used both types of test. In general, the randomization algorithms used in most null models should produce normal null distributions, but this could not be confirmed because most studies did not present any description of the null distribution. Researchers should be aware of the differences between direct and indirect tests so as to better use, communicate, and evaluate null models. In many cases, direct tests should be favored for their simplicity and parsimony.