The genomic and ecological context of hybridization affects the probability that symmetrical incompatibilities drive hybrid speciation

Despite examples of homoploid hybrid species, theoretical work describing when, where, and how we expect homoploid hybrid speciation to occur remains relatively rare. Here, I explore the probability of homoploid hybrid speciation due to symmetrical incompatibilities under different selective and genetic scenarios. Through simulation, I test how genetic architecture and selection acting on traits that do not themselves generate incompatibilities interact to affect the probability that hybrids evolve symmetrical incompatibilities with their parent species. Unsurprisingly, selection against admixture at adaptive loci that are linked to loci that generate incompatibilities tends to reduce the probability of evolving symmetrical incompatibilities. By contrast, selection that favors admixed genotypes at adaptive loci can promote the evolution of symmetrical incompatibilities. The magnitude of these outcomes is affected by the strength of selection, aspects of genetic architecture such as linkage relationships and the linear arrangement of loci along a chromosome, and the amount of hybridization following the formation of a hybrid zone. These results highlight how understanding the nature of selection, aspects of the genetics of traits affecting fitness, and the strength of reproductive isolation between hybridizing taxa can all be used to inform when we expect to observe homoploid hybrid speciation due to symmetrical incompatibilities.