The distribution of the coalescence time and the number of pairwise nucleotide differences in a model of population divergence or speciation with an initial period of gene flow

This paper is concerned with a model of isolation with an initial period of migration, where a panmictic ancestral population split into n descendant populations which exchanged migrants symmetrically at a constant rate for a period of time and subsequently became completely isolated. In the limit as the population split occurred an infinitely long time ago, the model becomes an isolation after migration model, describing completely isolated descendant populations which arose from a subdivided ancestral population. The probability density function of the coalescence time of a pair of genes and the probability distribution of the number of pairwise nucleotide differences are derived for both models. Whilst these are theoretical results of interest in their own right, they also give an exact analytical expression for the likelihood, for data consisting of the numbers of nucleotide differences between pairs of DNA sequences where each pair is at a different, independent locus. The behaviour of the distribution of the number of pairwise nucleotide differences under these models is illustrated and compared to the corresponding distributions under the isolation with migration and complete isolation models. It is shown that the distribution of the number of nucleotide differences between a pair of DNA sequences from different descendant populations in the model of isolation with an initial period of migration can be quite different from that under the isolation with migration model, even if the average migration rate over time (and hence the total number of migrants) is the same in both scenarios. It is also illustrated how the results can be extended to other demographic scenarios that can be described by a combination of isolated panmictic populations and symmetric island models. (C) 2012 Elsevier Inc. All rights reserved.