Journal
IEEE ACCESS
Volume 11, Issue -, Pages 79176-79189Publisher
IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
DOI: 10.1109/ACCESS.2023.3298049
Keywords
Distributed computing; load balance task-oriented; task scheduling
Ask authors/readers for more resources
Most research on commonly used distributed computing frameworks ignores the difficulty of learning these frameworks. A new distributed computing system is proposed to address the balance between versatility and user learning difficulty. The system is designed based on the idea of divide and conquer in distributed computing.
Most current research on commonly used distributed computing frameworks neglects the difficulty of learning these frameworks. These commonly used distributed computing frameworks are designed for professionals, and users must understand them well to use them for calculation tasks. In order to solve the challenging balance problem between versatility and user learning difficulty, a new distributed computing system is proposed. The design of this system is based on the idea of divide and conquer in distributed computing. The paper uses Netty, Zookeeper, Hadoop Distributed File System (HDFS) and other tools to implement the system functions. Firstly, the paper analyzes the research status of distributed computing frameworks at home and abroad; Then, according to the idea of divide and conquer, the paper uses Netty to implement the basic architecture of the system, uses Zookeeper and Netty to implement relevant distributed mechanisms, uses HDFS as the distributed storage, designs and implements a Task-Oriented General-Purpose (TOGP) distributed computing system, TOGP; Finally, the paper intensely studies the task scheduling problem of the system, proposes a scheduling algorithm, Node Processing Capacity and Distributed Lock Task Scheduling (CLTS) algorithm. The system scalability test proves that the TOGP has good horizontal scalability. In the comparative experiment, the processing capacity of the TOGP using the round robin is about 16% higher than that of Hadoop, which proves that the TOGP is available and efficient. A series of experiments prove that the CLTS scheduling algorithm can effectively adapt to the heterogeneous cluster environment and efficiently schedule tasks.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available