An Improved Fst Estimator

The fixation index F-st plays a central role in ecological and evolutionary genetic studies. The estimators of Wright ((F) over cap (st1)), Weir and Cockerham ((F) over cap (st2)), and Hudson et al. ((F) over cap st3) are widely used to measure genetic differences among different populations, but all have limitations. We propose a minimum variance estimator (F) over cap (stm) using (F) over cap (st1) and (F) over cap (st2). We tested (F) over cap (stm) in simulations and applied it to 120 unrelated East African individuals from Ethiopia and 11 subpopulations in HapMap 3 with 464,642 SNPs. Our simulation study showed that (F) over cap (stm) has smaller bias than (F) over cap (st2) for small sample sizes and smaller bias than (F) over cap (st1) for large sample sizes. Also, (F) over cap (stm) has smaller variance than (F) over cap (st2) for small F-st values and smaller variance than (F) over cap (st1) for large F-st values. We demonstrated that approximately 30 subpopulations and 30 individuals per subpopulation are required in order to accurately estimate F-st.