The effects of dominance, regular inbreeding and sampling design on QST, an estimator of population differentiation for quantitative traits

To test whether quantitative traits are under directional or homogenizing selection, it is common practice to compare population differentiation estimates at molecular Markers (F-ST) and quantitative traits (Q(ST)). If the trait is neutral and its determinism is additive, then theory predicts that Q(ST) = F-ST, while Q(ST) > F-ST is predicted under directional selection for different local opima, and Q(ST) < F-ST is predicted under homogenizing selection. However, nonadditive effects can alter these predictions. Here, we investigate the influence of dominance on the relation between Q(ST) and F-ST for neutral traits. Using analytical results and computer simulations, we show that dominance generally deflates Q(ST)-relative to F-ST. Under inbreeding, the effect of dominance vanishes, and we show that for selfing species, a better estimate of Q(ST) is obtained front selfed families than from half-sib families. We also compare several sampling designs and find that it is always best to sample many populations (>20) with few families (five) rather than few populations with many families. Provided that estimates of Q(ST) are derived from individuals originating from many populations, we Conclude that the pattern Q(ST) > F-ST, and hence the inference of directional selection to]different local optima, is robust to the effect of nonadditive gene actions.