Efficient in-situ workflow planning for geographically distributed heterogeneous environments

In-situ workflows are a particular class of scientific workflows where different components (such as simulation, visualization, machine learning, and data analysis) run concurrently. In an in-situ workflow, data continuously flows between components during the whole workflow lifetime in a pipeline fashion. The overall throughput of an in-situ workflow is limited by the slowest moving part of the workflow: either a time-consuming computation component or an expensive data transfer operation between different components. Given geographically distributed heterogeneous computing environments and an in-situ workflow comprising various component applications, it is challenging to produce optimized deployment plans to decide where to launch each component application and how much computing resources to allocate for each component. In this work, we first formulate and define the in-situ workflow planning problem and then discuss the uniqueness of the problem. We design a heuristic-based solver based on our new Scheduled-Neighbors-Lookup(SNL) algorithm, which produces effective deployment plans with much less time, compared with the mathematicaloptimization-based solver using CPLEX. Our experiments with both synthetic and real-world workflows show that the SNL algorithm can find optimized solutions whose qualities are comparable to that of the CPLEX optimization method with significantly reduced problem-solving time (e.g., more than 20,000 times faster on average for in-situ workflows with 14 components). Compared with existing methods such as HEFT, R-Storm, and T3 schedulers, the SNL algorithm can generate resource plans with higher throughput, with increased time and resource efficiency.

Automated summary

Optimizing deployment plans for in-situ workflows in geographically distributed heterogeneous computing environments is challenging. This study presents a heuristic-based solver using the SNL algorithm, which produces effective deployment plans with significantly reduced problem-solving time compared to the CPLEX optimization method.