Optimized load balancing in high-performance computing for big data analytics

New generation application problems in big data and high-performance computing (HPC) areas claim very diverse operational properties. The convergence requires the dynamic behavior of system components. Load balancing is a critical issue in response to the highly unpredictable, dynamic, and data-oriented behavior of the system. Possible practical constraints such as communication and load transfer delays play an essential role in designing a dynamic load balancer. On the other hand, according to most of the new platforms' distributed nature, the load balancer should be able to perform in a fully distributed manner. In this research, we consider practical issues, including different processing power, storage capability, communication, load transfer delays, and propose two distributed and optimized load balancing methods in HPC for Big Data processing. We model the constraints and present an argument named compensating factor for the optimized load balancer. We try to minimize the task execution time by reducing the nodes' idle time. We evaluate the proposed methods in different scenarios by using Monte Carlo. Evaluations results show that proposed methods decrease idle time significantly while being scalable to network size and applicable in heterogeneous networks with dynamic resources and configuration.

Automated summary

New generation application problems in big data and high-performance computing (HPC) areas pose diverse operational properties that require dynamic system component behavior, emphasizing the critical issue of load balancing. In this research, two distributed and optimized load balancing methods are proposed for HPC in Big Data processing, aiming to minimize task execution time by reducing idle time. Evaluations show that the proposed methods significantly decrease idle time, scale well to network size, and are applicable in heterogeneous networks with dynamic resources and configuration.