Hierarchical-Bayesian-Based Sparse Stochastic Configuration Networks for Construction of Prediction Intervals

To address the architecture complexity and ill-posed problems of neural networks when dealing with high-dimensional data, this article presents a Bayesian-learning-based sparse stochastic configuration network (SCN) (BSSCN). The BSSCN inherits the basic idea of training an SCN in the Bayesian framework but replaces the common Gaussian distribution with a Laplace one as the prior distribution of the output weights of SCN. Meanwhile, a lower bound of the Laplace sparse prior distribution using a two-level hierarchical prior is adopted based on which an approximate Gaussian posterior with sparse property is obtained. It leads to the facilitation of training the BSSCN, and the analytical solution for output weights of BSSCN can be obtained. Furthermore, the hyperparameter estimation process is derived by maximizing the corresponding lower bound of the marginal likelihood function based on the expectation-maximization algorithm. In addition, considering the uncertainties caused by both noises in the real-world data and model mismatch, a bootstrap ensemble strategy using BSSCN is designed to construct the prediction intervals (PIs) of the target variables. The experimental results on three benchmark data sets and two real-world high-dimensional data sets demonstrate the effectiveness of the proposed method in terms of both prediction accuracy and quality of the constructed PIs.

Automated summary

This article proposes a Bayesian-learning-based sparse stochastic configuration network (BSSCN) to address the architecture complexity and ill-posed problems of neural networks when dealing with high-dimensional data. By using a Laplace distribution as the prior distribution of the output weights of SCN, a sparse approximate Gaussian posterior is obtained, facilitating the training process and providing analytical solutions for output weights. The proposed method also includes a hyperparameter estimation process and a bootstrap ensemble strategy for constructing prediction intervals (PIs), which have shown effectiveness in experiments.