A new analytical approach to landscape genetic modelling: least-cost transect analysis and linear mixed models

Landscape genetics aims to assess the effect of the landscape on intraspecific genetic structure. To quantify interdeme landscape structure, landscape genetics primarily uses landscape resistance surfaces (RSs) and least-cost paths or straight-line transects. However, both approaches have drawbacks. Parameterization of RSs is a subjective process, and least-cost paths represent a single migration route. A transect-based approach might oversimplify migration patterns by assuming rectilinear migration. To overcome these limitations, we combined these two methods in a new landscape genetic approach: least-cost transect analysis (LCTA). Habitat-matrix RSs were used to create least-cost paths, which were subsequently buffered to form transects in which the abundance of several landscape elements was quantified. To maintain objectivity, this analysis was repeated so that each landscape element was in turn regarded as migration habitat. The relationship between explanatory variables and genetic distances was then assessed following a mixed modelling approach to account for the nonindependence of values in distance matrices. Subsequently, the best fitting model was selected using the statistic. We applied LCTA and the mixed modelling approach to an empirical genetic dataset on the endangered damselfly, Coenagrion mercuriale. We compared the results to those obtained from traditional least-cost, effective and resistance distance analysis. We showed that LCTA is an objective approach that identifies both the most probable migration habitat and landscape elements that either inhibit or facilitate gene flow. Although we believe the statistical approach to be an improvement for the analysis of distance matrices in landscape genetics, more stringent testing is needed.