4.6 Article

Scheduling Spark Tasks With Data Skew and Deadline Constraints

Journal

IEEE ACCESS
Volume 9, Issue -, Pages 2793-2804

Publisher

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
DOI: 10.1109/ACCESS.2020.3040719

Keywords

Data skew; spark; scheduling optimization; cloud computing

Funding

  1. National Key Research and Development Program of China [2018YFB1402500]
  2. National Natural Science Foundation of China [61872077, 61832004]
  3. National Hi-Tech Project [315055101]
  4. project of advanced research of the leading professional teachers in Higher Vocational Colleges in Jiangsu Province [2019GRFX078]
  5. Collaborative Innovation Center of Wireless Communications Technology

Ask authors/readers for more resources

This paper investigates Spark task scheduling with data skew and deadline constraints, and proposes an optimized algorithm which outperforms existing algorithms in big data processing performance based on experimental results.
Data skew has an essential impact on the performance of big data processing. Spark task scheduling with data skew and deadline constraints is considered to minimize the total rental cost in this paper. A modified scheduling architecture is developed in terms of the unique characteristics of the considered problem. A mathematical model is constructed, and a Spark task scheduling algorithm is proposed considering both the data skew and deadline constraints. The algorithm consists of three components: stage sequencing, task scheduling, and scheduling adjustment. Strategies for each of the components are presented. The parameters and components of the proposed algorithm are calibrated over many random instances. The calibrated algorithm is compared to two existing algorithms for similar problems over classical scientific workflow applications. Experimental results show that the proposed algorithm outperforms the compared algorithms statistically.

Authors

I am an author on this paper
Click your name to claim this paper and add it to your profile.

Reviews

Primary Rating

4.6
Not enough ratings

Secondary Ratings

Novelty
-
Significance
-
Scientific rigor
-
Rate this paper

Recommended

No Data Available
No Data Available