Outgroup sampling in phylogenetics: Severity of test and successive outgroup expansion

Outgroup sampling is a fundamental step in the design of phylogenetic analyses, independent of optimality criterion, taxonomic group, or source of evidence. Studies have demonstrated the efficient analysis of many thousands of terminals, all of which could be included in any empirical investigation, yet outgroup samples typically include only a small number of terminals. Most discussion of outgroup sampling centers on employing correct or appropriate outgroup terminals to increase accuracy or reliability by preventing errors such as long branch attraction and incorrect ingroup rooting. As an alternative, I develop a theory of outgroup sampling grounded in the logic of scientific discovery, whereby the objective is to test nested hypotheses of ingroup topology and character-state transformation as severely as possible by incorporating outgroup terminals in unconstrained, simultaneous analysis, using background knowledge to select the terminals that have the greatest chance of refuting those hypotheses. This framework provides a logical basis for selecting outgroup taxa but does not provide grounds for limiting the outgroup sample, given that, ceteris paribus, testability and explanatory power increase with the inclusion of additional terminals. Therefore, I propose the ancillary procedure of successively expanding the outgroup sample until ingroup hypotheses become stable (insensitive) to increased sampling, with each expansion guided by the scientific objectives of outgroup sampling. This is a heuristic procedure that does not prevent more outgroup terminals from being sampled or guarantee that ingroup hypotheses will remain insensitive to further outgroup expansion, and it has no bearing on the objective support of a given hypothesis. Nevertheless, it provides an objective, empirical basis for limiting outgroup sampling in a given research cycle. I illustrate this procedure by examining the effect of successive outgroup expansion on the relationships among the poison frog genera Adelphobates, Dendrobates, and Oophaga.