Species Tree Inference from Gene Splits by Unrooted STAR Methods

The NJ(st) method was proposed by Liu and Yu to infer a species tree topology from unrooted topological gene trees. While its statistical consistency under the multispecies coalescent model was established only for a four-taxon tree, simulations demonstrated its good performance on gene trees inferred from sequences for many taxa. Here, we prove the statistical consistency of the method for an arbitrarily large species tree. Our approach connects NJ(st) to a generalization of the STAR method of Liu, Pearl, and Edwards, and a previous theoretical analysis of it. We further show NJ(st) utilizes only the distribution of splits in the gene trees, and not their individual topologies. Finally, we discuss how multiple samples per taxon per gene should be handled for statistical consistency.