Consistent Model Selection for Marginal Generalized Additive Model for Correlated Data

We consider the generalized additive model when responses from the same cluster are correlated. Incorporating correlation in the estimation of nonparametric components for the generalized additive model is important because it improves estimation efficiency and increases statistical power for model selection. In our setting, there is no specified likelihood function for the generalized additive model, because the outcomes could be nonnormal and discrete, which makes estimation and model selection very challenging problems. We propose consistent estimation and model selection that incorporate the correlation structure. We establish an asymptotic property with L-2-norm consistency for the nonparametric components, which achieves the optimal rate of convergence. In addition, the proposed model selection strategy is able to select the correct generalized additive model consistently. That is, with probability approaching to 1, the estimators for the zero function components converge to 0 almost surely. We illustrate our method using numerical studies with both continuous and binary responses, along with a real data application of binary periodontal data. Supplemental materials including technical details are available online.