Journal
IEEE TRANSACTIONS ON BIG DATA
Volume 8, Issue 2, Pages 458-469Publisher
IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
DOI: 10.1109/TBDATA.2019.2948338
Keywords
Performance modeling; big data analytics; adaptive sampling; delaunay triangulation
Funding
- Academy of Finland [310321]
- Crowdsourced Battery Optimization AI for a Connected World (CBAI) [319017]
- Academy of Finland (AKA) [310321, 310321] Funding Source: Academy of Finland (AKA)
Ask authors/readers for more resources
This paper presents a performance prediction framework called d-Simplexed to address the challenge of understanding the relationship between different parameter configurations in Spark. By constructing a mesh using computational geometry and using adaptive sampling techniques, execution time can be predicted with high accuracy.
Big Data processing systems (e.g., Spark) have a number of resource configuration parameters, such as memory size, CPU allocation, and the number of running nodes. Regular users and even expert administrators struggle to understand the mutual relation between different parameter configurations and the overall performance of the system. In this paper, we address this challenge by proposing a performance prediction framework, called d-Simplexed, to build performance models with varied configurable parameters on Spark. We take inspiration from the field of Computational Geometry to construct a d-dimensional mesh using Delaunay Triangulation over a selected set of features. From this mesh, we predict execution time for various feature configurations. To minimize the time and resources in building a bootstrap model with a large number of configuration values, we propose an adaptive sampling technique to allow us to collect as few training points as required. Our evaluation on a cluster of computers using WordCount, PageRank, Kmeans, and Join workloads in HiBench benchmarking suites shows that we can achieve less than 5 percent error rate for estimation accuracy by sampling less than 1 percent of data.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available