期刊
CMC-COMPUTERS MATERIALS & CONTINUA
卷 62, 期 1, 页码 217-231出版社
TECH SCIENCE PRESS
DOI: 10.32604/cmc.2020.06431
关键词
News text topic clustering; spark platform; countvectorizer algorithm; TF-IDF algorithm; latent dirichlet allocation model
资金
- Science Research Projects of Hunan Provincial Education Department [18A174, 18C0262]
- National Natural Science Foundation of China [61772561]
- Key Research & Development Plan of Hunan Province [2018NK2012, 2019SK2022]
- Degree & Postgraduate Education Reform Project of Hunan Province [209]
- Postgraduate Education and Teaching Reform Project of Central South Forestry University [2019JG013]
Due to the slow processing speed of text topic clustering in stand-alone architecture under the background of big data, this paper takes news text as the research object and proposes LDA text topic clustering algorithm based on Spark big data platform. Since the TF-IDF (term frequency-inverse document frequency) algorithm under Spark is irreversible to word mapping, the mapped words indexes cannot be traced back to the original words. In this paper, an optimized method is proposed that TF-IDF under Spark to ensure the text words can be restored. Firstly, the text feature is extracted by the TF-IDF algorithm combined CountVectorizer proposed in this paper, and then the features are inputted to the LDA (Latent Dirichlet Allocation) topic model for training. Finally, the text topic clustering is obtained. Experimental results show that for large data samples, the processing speed of LDA topic model clustering has been improved based Spark. At the same time, compared with the LDA topic model based on word frequency input, the model proposed in this paper has a reduction of perplexity.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据