Improved Hadoop-based cloud for complex model simulation optimization: Calibration of SWAT as an example

A simulation optimization framework requires a substantial number of model simulations, which are computationally intensive and may be impractical when the model simulations are extremely time-consuming. This paper presents an improved Hadoop-based cloud framework to alleviate the computational burden of optimization. The framework parallelizes conventional sequential-model-based optimization techniques by concurrently orchestrating multiple model computations within Hadoop MapReduce. It guarantees the reliability of simulation optimization tasks by handling node failures without affecting the ongoing simulation. A case study, using Bayesian optimization to calibrate a SWAT model, achieved a speedup of nearly 55-58 when using 100 cores, demonstrating the efficiency of parallelizing the Bayesian optimization algorithm on the Hadoop-based cloud. Experiments in which computing nodes were dynamically increased or decreased demonstrated that the framework can automatically rebalance the workload across the remaining nodes. The framework is readily adaptable to other complex model applications that perform sequential-model-based optimizations or large-scale simulations.

Automated summary

The paper introduces an improved Hadoop-based cloud framework that parallelizes conventional sequential-model-based optimization techniques to alleviate computational burden and enhance efficiency. The framework can automatically rebalance workload and is adaptable to other complex model applications.