Effect of mutation on genetic differentiation among nonequilibrium populations

The usefulness of G(ST) and similar measures of genetic differentiation has been questioned repeatedly because of their dependence on the amount of heterozygosity within populations, creating problems when comparing degrees of divergence at loci with different mutation rates. Although the effect of mutation on G(ST) is expected to be small in the early phases of divergence, it is unclear for how long after separation from a common ancestral population that G(ST) is largely unaffected by mutation and by the resulting effect on heterozygosity. We address this question through analysis of the recursion equations for gene identity under the infinite allele model of mutation, and derive conditions describing when the effect of mutation on G(ST) can be ignored under mutation-migration-drift equilibrium conditions and during the preceding transition phase. An important result is that during the transition phase G(ST) is not only affected by mutation, but also by the heterozygosity in the base population from which the subpopulations diverged. The effect of mutation on G(ST) is significant from the very start of the divergence process when initial heterozygosity is low, whereas G(ST) is only weakly affected by mutation in the early phases of differentiation when initial heterozygosity is high. Thus, differentiation following a severe bottleneck is strongly dependent on mutation. The standardized measure of differentiation, G'(ST), suggested by Hedrick (2005), may be helpful when comparing amounts of divergence at loci with different mutation rates under steady-state conditions, provided that migration is very low. In many other situations the use of G'(ST) might be misleading, however, and its application should be exercised with caution.