Gibbs Priors for Bayesian Nonparametric Variable Selection with Weak Learners

We consider the problem of high-dimensional Bayesian nonparametric variable selection using an aggregation of so-called weak learners. The most popular variant of this is the Bayesian additive regression trees (BART) model, which is the natural Bayesian analog to boosting decision trees. In this article, we use Gibbs distributions on random partitions to induce sparsity in ensembles of weak learners. Looking at BART as a special case, we show that the class of Gibbs priors includes two recently proposed models- the Dirichlet additive regression trees (DART) model and the spike-and-forest model-as extremal cases, and we show that certain Gibbs priors are capable of achieving the benefits of both the DART and spike and-forest models while avoiding some of their key drawbacks. We then show the promising performance of Gibbs priors for other classes of weak learners, such as tensor products of spline basis functions. A Polya Urn scheme is developed for efficient computations. Supplementary materials for this article are available online.

Automated summary

This article investigates the problem of high-dimensional Bayesian nonparametric variable selection using an aggregation of weak learners. The authors propose a solution by inducing sparsity in ensembles of weak learners through the use of Gibbs distributions and show the advantages of this approach.