Multiple-instance ensemble for construction of deep heterogeneous committees for high-dimensional low-sample-size data

Deep ensemble learning, where we combine knowledge learned from multiple individual neural networks, has been widely adopted to improve the performance of neural networks in deep learning. This field can be encompassed by committee learning, which includes the construction of neural network cascades. This study focuses on the high-dimensional low-sample-size (HDLS) domain and introduces multiple instance ensemble (MIE) as a novel stacking method for ensembles and cascades. In this study, our proposed approach reformulates the ensemble learning process as a multiple-instance learning problem. We utilise the multiple-instance learning solution of pooling operations to associate feature representations of base neural networks into joint representations as a method of stacking. This study explores various attention mechanisms and proposes two novel committee learning strategies with MIE. In addition, we utilise the capability of MIE to generate pseudo-base neural networks to provide a proof-of-concept for a growing'' neural network cascade that is unbounded by the number of base neural networks. We have shown that our approach provides (1) a class of alternative ensemble methods that performs comparably with various stacking ensemble methods and (2) a novel method for the generation of high-performing growing'' cascades. The approach has also been verified across multiple HDLS datasets, achieving high performance for binary classification tasks in the low-sample size regime. (c) 2023 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).

Automated summary

Deep ensemble learning, specifically multiple instance ensemble (MIE), is introduced as a novel stacking method for improving the performance of neural networks. By reformulating the ensemble learning process as a multiple-instance learning problem, MIE associates feature representations of base neural networks into joint representations using pooling operations. This study explores attention mechanisms and proposes new committee learning strategies with MIE. The capability of MIE to generate pseudo-base neural networks enables the creation of growing cascades. Experimental results on multiple HDLS datasets demonstrate the high performance of the proposed approach in low-sample size regime for binary classification tasks.