Cutting the Unnecessary Long Tail: Cost-Effective Big Data Clustering in the Cloud

Clustering big data often requires tremendous computational resources where cloud computing is undoubtedly one of the promising solutions. However, the computation cost in the cloud can be unexpectedly high if it cannot be managed properly. The long tail phenomenon has been observed widely in the big data clustering area, which indicates that the majority of time is often consumed in the middle to late stages in the clustering process. In this research, we try to cut the unnecessary long tail in the clustering process to achieve a sufficiently satisfactory accuracy at the lowest possible computation cost. A novel approach is proposed to achieve cost-effective big data clustering in the cloud. By training the regression model with the sampling data, we can make widely used k-means and EM (Expectation-Maximization) algorithms stop automatically at an early point when the desired accuracy is obtained. Experiments are conducted on four popular data sets and the results demonstrate that both k-means and EM algorithms can achieve high cost-effectiveness in the cloud with our proposed approach. For example, in the case studies with the much more efficient k-means algorithm, we find that achieving a 99 percent accuracy needs only 47.71-71.14 percent of the computation cost required for achieving a 100 percent accuracy while the less efficient EM algorithm needs 16.69-32.04 percent of the computation cost. To put that into perspective, in the United States land use classification example, our approach can save up to $94,687.49 for the government in each use.

Automated summary

This research proposes a novel approach to achieve cost-effective big data clustering in the cloud by training a regression model with sampling data, allowing k-means and EM algorithms to stop automatically when desired accuracy is obtained. Experiment results show high cost-effectiveness with k-means needing only 47.71-71.14% of the computation cost for 99% accuracy and EM needing 16.69-32.04%, potentially saving up to $94,687.49 for the government in each use case.