Appropriate likelihood ratio tests and marginal distributions for evolutionary tree models with constraints on parameters

We show how to make appropriate likelihood ratio tests for evolutionary tree models when parameters such as edge (intermodes or branches) lengths have nonnegativity constraints. In such cases, under the null model of an edge length being zero, the marginal distribution of this parameter is proven to be a half-normal, that is, 50% zero values and 50% the positive half of a normal distribution. Other constrained parameters, such as the proportion of invariant sites, give similar results. To make likelihood ratio tests between nested models, e.g., H-0: homogeneous site rates, and H-1: site rates follow a gamma distribution with variance 1/k, then asymptotically as sequence length increases, the distribution under H-0, becomes a mixture of chi(2) distributions, in this case 50% chi(0)(2), and 50% chi(1)(2) (where the subscript denotes degrees of freedom, i.e., not the usually assumed 100% chi(1)(2); which leads to a conservative test). Such mixtures are sometimes called <(chi)over bar>(2) distributions. Simulations show that even with sequences as short as 125 sites, some parameters, including the proportion of invariant sites, fit asymptotic distributions closely.