OptM: estimating the optimal number of migration edges on population trees using Treemix

The software Treemix has become extensively used to estimate the number of migration events, or edges (m), on population trees from genome-wide allele frequency data. However, the appropriate number of edges to include remains unclear. Here, I show that an optimal value of m can be inferred from the second-order rate of change in likelihood (Delta m) across incremental values of m. Repurposed from its original use to estimate the number of population clusters in the software Structure (Delta K), I show using simulated populations that Delta m performs equally as well as current recommendations for Treemix. A demonstration of an empirical dataset from domestic dogs indicates that this method may be preferable in large, complex population histories and can prioritize migration events for subsequent investigation. The method has been implemented in a freely available R package called OptM and as a web application (https://rfitak.shinyapps.io/OptM/) to interface directly with the output files of Treemix.

Automated summary

The study shows that inferring the optimal number of migration events in population trees can be achieved by examining the second-order rate of change in likelihood (Δm), which performs equally well as current recommendations for Treemix. This method may be preferable in large, complex population histories.