INFINITE-DIMENSIONAL GRADIENT-BASED DESCENT FOR ALPHA-DIVERGENCE MINIMISATION

This paper introduces the (alpha, Gamma)-descent, an iterative algorithm which operates on measures and performs alpha-divergence minimisation in a Bayesian framework. This gradient-based procedure extends the commonly-used variational approximation by adding a prior on the variational parameters in the form of a measure. We prove that for a rich family of functions Gamma, this algorithm leads at each step to a systematic decrease in the alpha-divergence and derive convergence results. Our framework recovers the Entropic Mirror Descent algorithm and provides an alternative algorithm that we call the Power Descent. Moreover, in its stochastic formulation, the (alpha, Gamma)-descent allows to optimise the mixture weights of any given mixture model without any information on the underlying distribution of the variational parameters. This renders our method compatible with many choices of parameters updates and applicable to a wide range of Machine Learning tasks. We demonstrate empirically on both toy and real-world examples the benefit of using the Power Descent and going beyond the Entropic Mirror Descent framework, which fails as the dimension grows.

Automated summary

This paper introduces an (alpha, Gamma)-descent algorithm for alpha-divergence minimisation in a Bayesian framework, extending the variational approximation method and systematically decreasing the alpha-divergence at each step. The algorithm recovers the Entropic Mirror Descent and offers the Power Descent as an alternative, while also being able to optimize mixture model weights without information on the underlying distribution of the variational parameters. Empirical results show the benefits of the Power Descent over the Entropic Mirror Descent as dimensions grow.