Journal
IEEE ACCESS
Volume 9, Issue -, Pages 2793-2804Publisher
IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
DOI: 10.1109/ACCESS.2020.3040719
Keywords
Data skew; spark; scheduling optimization; cloud computing
Categories
Funding
- National Key Research and Development Program of China [2018YFB1402500]
- National Natural Science Foundation of China [61872077, 61832004]
- National Hi-Tech Project [315055101]
- project of advanced research of the leading professional teachers in Higher Vocational Colleges in Jiangsu Province [2019GRFX078]
- Collaborative Innovation Center of Wireless Communications Technology
Ask authors/readers for more resources
This paper investigates Spark task scheduling with data skew and deadline constraints, and proposes an optimized algorithm which outperforms existing algorithms in big data processing performance based on experimental results.
Data skew has an essential impact on the performance of big data processing. Spark task scheduling with data skew and deadline constraints is considered to minimize the total rental cost in this paper. A modified scheduling architecture is developed in terms of the unique characteristics of the considered problem. A mathematical model is constructed, and a Spark task scheduling algorithm is proposed considering both the data skew and deadline constraints. The algorithm consists of three components: stage sequencing, task scheduling, and scheduling adjustment. Strategies for each of the components are presented. The parameters and components of the proposed algorithm are calibrated over many random instances. The calibrated algorithm is compared to two existing algorithms for similar problems over classical scientific workflow applications. Experimental results show that the proposed algorithm outperforms the compared algorithms statistically.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available