Benchmarking the Mantel test and derived methods for testing association between distance matrices

Testing the association between objects is central in ecology, evolution, and quantitative sciences in general. Two types of variables can describe the relationships between objects: point variables (measured on individual objects), and distance variables (measured between pairs of objects). The Mantel test and derived methods have been extensively used for distance variables. Yet, these methods have been criticized due to low statistical power and inflated type I error when spatial autocorrelation is present. Here, we assessed the statistical power between different types of tested variables and the type I error rate over a wider range of autocorrelation intensities than previously assessed, both on univariate and multivariate data. We also illustrated the performance of distance matrix statistics through computational simulations of genetic diversity. We show that the Mantel test and derived methods are not affected by inflated type I error when spatial autocorrelation affects only one variable when investigating correlations, or when either the response or the explanatory variable(s) is affected by spatial autocorrelation while investigating causal relationships. As previously noted, with autocorrelation affecting more variables, inflated type I error could be reduced by modifying the significance threshold. Additionally, the Mantel test has no problem of statistical power when the hypothesis is formulated in terms of distance variables. We highlight that transformation of variable types should be avoided because of the potential information loss and modification of the tested hypothesis. We propose a set of guidelines to help choose the appropriate method according to the type of variables and defined hypothesis.

Automated summary

Testing associations between objects is crucial in ecology, evolution, and quantitative sciences. This study evaluates the statistical power and error rate of different types of variables in the presence of spatial autocorrelation, and provides guidelines for choosing appropriate methods.