Transgressive segregation in mating traits drives hybrid speciation

Hybridization can rapidly generate novel genetic variation, which can promote ecological speciation by creating novel adaptive phenotypes. However, it remains unclear how hybridization, creating novel mating phenotypes (e.g., mating season, genitalia shapes, sexual displays, mate preferences), affects speciation especially when the phenotypes do not confer adaptive advantages. Here, based on individual-based evolutionary simulations, we propose that transgressive segregation of mating traits can drive incipient hybrid speciation. Simulations demonstrated that incipient hybrid speciation occurred most frequently when the hybrid population received moderate continued immigration from parental lineages causing recurrent episodes of hybridization. Recurrent hybridization constantly generated genetic variation, which promoted the rapid stochastic evolution of mating phenotypes in a hybrid population. The stochastic evolution continued until a novel mating phenotype came to dominate the hybrid population, which reproductively isolates the hybrid population from parental lineages. However, too frequent hybridization rather hindered the evolution of reproductive isolation by inflating the variation of mating phenotypes to produce phenotypes allowing mating with parental lineages. Simulations also revealed conditions for the long-term persistence of hybrid species after their incipient emergence. Our results suggest that recurrent transgressive segregation of mating phenotypes can offer a plausible explanation for hybrid speciation and radiations that involved little adaptive ecological divergence.

Automated summary

Hybridization can promote ecological speciation through generating novel genetic variation and creating new mating phenotypes. This study suggests that transgressive segregation of mating traits can drive incipient hybrid speciation, with moderate continued immigration facilitating recurrent hybridization events. Recurrent hybridization leads to the rapid stochastic evolution of mating phenotypes until a novel phenotype dominates the hybrid population, resulting in reproductive isolation from parental lineages. However, excessive hybridization hinders the evolution of reproductive isolation by producing mating phenotypes allowing interbreeding with parental lineages. The study also identifies conditions for the long-term persistence of hybrid species.