期刊
IEEE TRANSACTIONS ON BIG DATA
卷 8, 期 2, 页码 458-469出版社
IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
DOI: 10.1109/TBDATA.2019.2948338
关键词
Performance modeling; big data analytics; adaptive sampling; delaunay triangulation
资金
- Academy of Finland [310321]
- Crowdsourced Battery Optimization AI for a Connected World (CBAI) [319017]
- Academy of Finland (AKA) [310321, 310321] Funding Source: Academy of Finland (AKA)
This paper presents a performance prediction framework called d-Simplexed to address the challenge of understanding the relationship between different parameter configurations in Spark. By constructing a mesh using computational geometry and using adaptive sampling techniques, execution time can be predicted with high accuracy.
Big Data processing systems (e.g., Spark) have a number of resource configuration parameters, such as memory size, CPU allocation, and the number of running nodes. Regular users and even expert administrators struggle to understand the mutual relation between different parameter configurations and the overall performance of the system. In this paper, we address this challenge by proposing a performance prediction framework, called d-Simplexed, to build performance models with varied configurable parameters on Spark. We take inspiration from the field of Computational Geometry to construct a d-dimensional mesh using Delaunay Triangulation over a selected set of features. From this mesh, we predict execution time for various feature configurations. To minimize the time and resources in building a bootstrap model with a large number of configuration values, we propose an adaptive sampling technique to allow us to collect as few training points as required. Our evaluation on a cluster of computers using WordCount, PageRank, Kmeans, and Join workloads in HiBench benchmarking suites shows that we can achieve less than 5 percent error rate for estimation accuracy by sampling less than 1 percent of data.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据