Order Selection in Finite Mixture Models With a Nonsmooth Penalty

Order selection is a fundamental and challenging problem in the application of finite mixture models. We develop a new penalized likelihood approach that we call MSCAD. MSCAD deviates from information-based methods, such as Akaike information criterion and the Bayes information criterion, by introducing two penalty functions that depend on the mixing proportions and the component parameters. It is consistent in estimating both the order of the mixture model and the mixing distribution. Simulations show that MSCAD performs much better than some existing methods. Two real-data examples are examined to illustrate its performance.