Speciation without gene-flow in hybridizing deer

Under the ecological speciation model, divergent selection acts on ecological differences between populations, gradually creating barriers to gene flow and ultimately leading to reproductive isolation. Hybridisation is part of this continuum and can both promote and inhibit the speciation process. Here, we used white-tailed (Odocoileus virginianus) and mule deer (O. hemionus) to investigate patterns of speciation in hybridizing sister species. We quantified genome-wide historical introgression and performed genome scans to look for signatures of four different selection scenarios. Despite ample modern evidence of hybridisation, we found negligible patterns of ancestral introgression and no signatures of divergence with gene flow, rather localized patterns of allopatric and balancing selection were detected across the genome. Genes under balancing selection were related to immunity, MHC and sensory perception of smell, the latter of which is consistent with deer biology. The deficiency of historical gene-flow suggests that white-tailed and mule deer were spatially separated during the glaciation cycles of the Pleistocene and genome wide differentiation accrued via genetic drift. Dobzhansky-Muller incompatibilities and selection against hybrids are hypothesised to be acting, and diversity correlations to recombination rates suggests these sister species are far along the speciation continuum.

Automated summary

Under the ecological speciation model, divergent selection creates barriers to gene flow and leads to reproductive isolation. Hybridisation can both promote and inhibit speciation. Using white-tailed and mule deer, we found negligible ancestral introgression and no divergence with gene flow, but localized allopatric and balancing selection. Balancing selection affected genes related to immunity, MHC, and smell perception, consistent with deer biology. The lack of historical gene flow suggests spatial separation during glaciation cycles and differentiation via genetic drift. Dobzhansky-Muller incompatibilities and selection against hybrids are hypothesized to be acting, and diversity correlations suggest advanced speciation.