DEFEG: Deep Ensemble with Weighted Feature Generation

With the significant breakthrough of Deep Neural Networks in recent years, multi-layer architecture has influenced other sub-fields of machine learning including ensemble learning. In 2017, Zhou and Feng introduced a deep random forest called gcForest that involves several layers of Random Forest-based classifiers. Although gcForest has outperformed several benchmark algorithms on specific datasets in terms of classification accuracy and model complexity, its input features do not ensure better performance when going deeply through layer-by-layer architecture. We address this limitation by introducing a deep ensemble model with a novel feature generation module. Unlike gcForest where the original features are concatenated to the outputs of classifiers to generate the input features for the subsequent layer, we integrate weights on the classifiers' outputs as augmented features to grow the deep model. The usage of weights in the feature generation process can adjust the input data of each layer, leading the better results for the deep model. We encode the weights using variable-length encoding and develop a variable-length Particle Swarm Optimization method to search for the optimal values of the weights by maximizing the classification accuracy on the validation data. Experiments on a number of UCI datasets confirm the benefit of the proposed method compared to some well-known benchmark algorithms. (c) 2023 The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc- nd/4.0/).

Automated summary

With the breakthrough of Deep Neural Networks, multi-layer architecture has influenced ensemble learning. In 2017, Zhou and Feng introduced a deep random forest called gcForest. However, its input features do not ensure better performance in layer-by-layer architecture. To address this, we propose a novel deep ensemble model with a feature generation module. We integrate weights on classifiers' outputs and encode them using variable-length encoding and optimize their values using a Particle Swarm Optimization method. Experimental results on UCI datasets show the superiority of the proposed method over benchmark algorithms.