Spark-based cooperative coevolution for large scale global optimization

The cooperative coevolution framework was introduced to address the shortcomings of metaheuristic algorithms in solving continuous large-scale global optimization problems. By dividing the problem into subcomponents which can be optimized separately, the framework can improve on both the solution's quality as well as the computational speed by exposing a degree of parallelism. Distributed computing platforms, such as Apache Spark, have long been used to improve the speed of different algorithms in solving computational problems. This work proposes a distributed implementation of the cooperative coevolution framework for solving large-scale global optimization problems on the Apache Spark distributed computing platform. By using a formerly outlined distributed variant of the cooperative coevolution framework, features of the Spark platform are utilized to enhance the computational speed of the algorithm while maintaining comparable search quality to other works in the literature. To test for the proposed implementation's improvement in computational speed, the CEC 2010 large-scale global optimization benchmark functions are used due to the diversity they offer in terms of complexity, separability and modality. Results of the proposed distributed implementation suggest that a speedup of up to x3.36 is possible on large-scale global optimization benchmarks using the Apache Spark platform.

Automated summary

The cooperative coevolution framework improves the quality and computational speed of metaheuristic algorithms for solving continuous large-scale global optimization problems by dividing the problem into subcomponents. This work proposes a distributed implementation of the framework on the Apache Spark platform, utilizing its features to enhance computational speed while maintaining comparable search quality. The proposed implementation achieves a speedup of up to x3.36 on large-scale global optimization benchmarks using Apache Spark.