Correlated product of experts for sparse Gaussian process regression

Gaussian processes (GPs) are an important tool in machine learning and statistics. However, off-the-shelf GP inference procedures are limited to datasets with several thousand data points because of their cubic computational complexity. For this reason, many sparse GPs techniques have been developed over the past years. In this paper, we focus on GP regression tasks and propose a new approach based on aggregating predictions from several local and correlated experts. Thereby, the degree of correlation between the experts can vary between independent up to fully correlated experts. The individual predictions of the experts are aggregated taking into account their correlation resulting in consistent uncertainty estimates. Our method recovers independent Product of Experts, sparse GP and full GP in the limiting cases. The presented framework can deal with a general kernel function and multiple variables, and has a time and space complexity which is linear in the number of experts and data samples, which makes our approach highly scalable. We demonstrate superior performance, in a time vs. accuracy sense, of our proposed method against state-of-the-art GP approximations for synthetic as well as several real-world datasets with deterministic and stochastic optimization.

Automated summary

Gaussian processes (GPs) are widely used in machine learning and statistics, but their computational complexity limits their applicability to datasets with only a few thousand data points. To overcome this limitation, we propose a new approach based on aggregating predictions from multiple local and correlated experts, which can provide consistent uncertainty estimates. Our method can handle various kernel functions and multiple variables, and has linear time and space complexity, making it highly scalable. Experimental results on synthetic and real-world datasets demonstrate the superior performance of our approach compared to state-of-the-art GP approximations in terms of both time and accuracy.