Mating system and speciation I: Accumulation of genetic incompatibilities in allopatry

Self-fertilisation is widespread among hermaphroditic species across the tree of life. Selfing has many consequences on the genetic diversity and the evolutionary dynamics of populations, which may in turn affect macroevolutionary processes such as speciation. On the one hand, because selfing increases genetic drift and reduces migration rate among populations, it may be expected to promote speciation. On the other hand, because selfing reduces the efficacy of selection, it may be expected to hamper ecological speciation. To better understand under which conditions and in which direction selfing affects the build-up of reproductive isolation, an explicit population genetics model is required. Here, we focus on the interplay between genetic drift, selection and genetic linkage by studying speciation without gene flow. We test how fast populations with different rates of selfing accumulate mutations leading to genetic incompatibilities. When speciation requires populations to pass through a fitness valley caused by underdominant and compensatory mutations, selfing reduces the depth and/or breadth of the valley, and thus overall facilitates the fixation of incompatibilities. When speciation does not require populations to pass through a fitness valley, as for Bateson-Dobzhanzky-Muller incompatibilities (BDMi), the lower effective population size and higher genetic linkage in selfing populations both facilitate the fixation of incompatibilities. Interestingly, and contrary to intuitive expectations, local adaptation does not always accelerate the fixation of incompatibilities in outcrossing relative to selfing populations. Our work helps to clarify how incompatibilities accumulate in selfing vs. outcrossing lineages, and has repercussions on the pace of speciation as well as on the genetic architecture of reproductive isolation. Author summary Hermaphroditic organisms may use their male gametes to fertilise their own female gametes, but species vary greatly in how much they self-fertilise. Self-fertilisation (selfing) induces many genetic modifications in populations, which may ultimately affect the rates at which lineages diversify. Here we aim to build predictions on how selfing affects the rate at which hybrid incompatibility arises between geographically isolated populations. We developed theoretical models in which populations with variable selfing rates may fixate mutations leading to hybrid incompatibility. First, we explored scenarios wherein hybrid incompatibility is caused by mutations whose fixations necessitates a population to experience temporally deleterious effects (i.e., a fitness valley), and found that selfing reduces the breadth and depth of the fitness valley and thereby overall facilitates the accumulation of such mutations. Second, we explored scenarios wherein hybrid incompatibility is caused by interactions between derived alleles of different genes (i.e., BDMi). We found that selfing reduces the manifestation of BDMi within population, and thereby facilitates their fixation. This effect prevails even in the face of local adaptation. Thus, our study clarifies how fast hybrid incompatibilities should accumulate in selfing populations, and helps to understand how new species are expected to arise in selfing lineages.

Automated summary

Self-fertilisation is common in hermaphroditic species and has significant impacts on genetic diversity and evolutionary dynamics. It can promote speciation by increasing genetic drift and reducing migration, but it can also hinder ecological speciation by reducing selection efficacy. This study uses a population genetics model to investigate the effects of selfing on the accumulation of reproductive isolation. The results show that selfing reduces the depth and breadth of fitness valleys, facilitating the fixation of incompatibilities. In addition, selfing accelerates the fixation of incompatibilities caused by interactions between derived alleles of different genes, even in the presence of local adaptation. These findings contribute to our understanding of the accumulation of incompatibilities in selfing and outcrossing lineages, and shed light on the process of speciation in selfing lineages.