期刊
IEEE TRANSACTIONS ON CLOUD COMPUTING
卷 10, 期 1, 页码 292-303出版社
IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
DOI: 10.1109/TCC.2019.2947678
关键词
Cloud computing; cost-effectiveness; clustering algorithms; big data; data mining
类别
资金
- China Scholarship Council, National Key Research and Development Plan of China [2016YFB0502604, 2016YFC0803000]
- Key Science and Technology Project of Beijing [Z171100005117002]
This research proposes a novel approach to achieve cost-effective big data clustering in the cloud by training a regression model with sampling data, allowing k-means and EM algorithms to stop automatically when desired accuracy is obtained. Experiment results show high cost-effectiveness with k-means needing only 47.71-71.14% of the computation cost for 99% accuracy and EM needing 16.69-32.04%, potentially saving up to $94,687.49 for the government in each use case.
Clustering big data often requires tremendous computational resources where cloud computing is undoubtedly one of the promising solutions. However, the computation cost in the cloud can be unexpectedly high if it cannot be managed properly. The long tail phenomenon has been observed widely in the big data clustering area, which indicates that the majority of time is often consumed in the middle to late stages in the clustering process. In this research, we try to cut the unnecessary long tail in the clustering process to achieve a sufficiently satisfactory accuracy at the lowest possible computation cost. A novel approach is proposed to achieve cost-effective big data clustering in the cloud. By training the regression model with the sampling data, we can make widely used k-means and EM (Expectation-Maximization) algorithms stop automatically at an early point when the desired accuracy is obtained. Experiments are conducted on four popular data sets and the results demonstrate that both k-means and EM algorithms can achieve high cost-effectiveness in the cloud with our proposed approach. For example, in the case studies with the much more efficient k-means algorithm, we find that achieving a 99 percent accuracy needs only 47.71-71.14 percent of the computation cost required for achieving a 100 percent accuracy while the less efficient EM algorithm needs 16.69-32.04 percent of the computation cost. To put that into perspective, in the United States land use classification example, our approach can save up to $94,687.49 for the government in each use.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据