4.4 Article

Variable Selection for Distributed Sparse Regression Under Memory Constraints

出版社

SPRINGER HEIDELBERG
DOI: 10.1007/s40304-022-00291-w

关键词

Variable selection; Distributed sparse regression; Memory constraints; Distributed penalized likelihood algorithm

向作者/读者索取更多资源

This paper studies variable selection in distributed sparse regression with large sample size and limited memory constraint. By improving the traditional divide-and-conquer method, the proposed method can better control the false discovery rate and reduce the computational burden. Theoretical properties and computational algorithms are established, and the method is evaluated through simulations and a real example.
This paper studies variable selection using the penalized likelihood method for distributed sparse regression with large sample size n under a limited memory constraint. This is a much needed research problem to be solved in the big data era. A naive divide-and-conquer method solving this problem is to split the whole data into N parts and run each part on one of N machines, aggregate the results from all machines via averaging, and finally obtain the selected variables. However, it tends to select more noise variables, and the false discovery rate may not be well controlled. We improve it by a special designed weighted average in aggregation. Although the alternating direction method of multiplier can be used to deal with massive data in the literature, our proposed method reduces the computational burden a lot and performs better by mean square error in most cases. Theoretically, we establish asymptotic properties of the resulting estimators for the likelihood models with a diverging number of parameters. Under some regularity conditions, we establish oracle properties in the sense that our distributed estimator shares the same asymptotic efficiency as the estimator based on the full sample. Computationally, a distributed penalized likelihood algorithm is proposed to refine the results in the context of general likelihoods. Furthermore, the proposed method is evaluated by simulations and a real example.

作者

我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。

评论

主要评分

4.4
评分不足

次要评分

新颖性
-
重要性
-
科学严谨性
-
评价这篇论文

推荐

暂无数据
暂无数据