Testing hybridization hypotheses based on incongruent gene trees

Hybridization is an important evolutionary mechanism in plants and has been increasingly documented in animals. Difficulty in reconstruction of reticulate evolution, however, has been a longstanding problem in phylogenetics. Consequently, hybrid speciation may play a major role in causing topological incongruence between gene trees. The incongruence, in turn, offers an opportunity to detect hybrid speciation. Here we characterized certain distinctions between hybridization and other biological processes, including lineage sorting, paralogy, and lateral gene transfer, that are responsible for topological incongruence between gene trees. Consider two incongruent gene trees with three taxa, A, B, and C, where B is a sister group of A on gene tree 1 but a sister group of C on gene tree 2. With a theoretical model based on the molecular clock, we demonstrate that time of divergence of each gene between taxa A and C is nearly equal in the case of hybridization (B is a hybrid) or lateral gene transfer, but differs significantly in the case of lineage sorting or paralogy. After developing a bootstrap test to test these alternative hypotheses, we extended the model and test to account for incongruent gene trees with numerous taxa. Computer simulation studies supported the validity of the theoretical model and bootstrap test when each gene evolved at a constant rate. The computer simulation also suggested that the model remained valid as long as the rate heterogeneity was occurring proportionally in the same taxa for both genes. Although the model could not test hypotheses of hybridization versus lateral gene transfer as the cause of incongruence, these two processes may be distinguished by comparing phylogenies of multiple unlinked genes.