Accelerated Optimization in the PDE Framework: Formulations for the Manifold of Diffeomorphisms

We consider the problem of optimization of cost functionals on the infinite dimensional manifold of diffeomorphisms. We present a new class of optimization methods, valid for any optimization problem setup on the space of diffeomorphisms by generalizing Nesterov accelerated optimization to the manifold of diffeomorphisms. While our framework is general for infinite dimensional manifolds, we specifically treat the case of diffeomorphisms, motivated by optical flow problems in computer vision. This is accomplished by building on a recent variational approach to a general class of accelerated optimization methods by Wibisono, Wilson, and Jordan [Proc. Natl. Acad. Sci. USA, 113 (2016), pp. E7351-E7358], which applies in finite dimensions. We generalize that approach to infinite dimensional manifolds. We derive the surprisingly simple continuum evolution equations, which are partial differential equations, for accelerated gradient descent, and relate them to simple mechanical principles from fluid mechanics. Our approach has natural connections to the optimal mass transport problem. This is because one can think of our approach as an evolution of an infinite number of particles endowed with mass (represented with a mass density) that moves in an energy landscape. The mass evolves with the optimization variable and endows the particles with dynamics. This is different from the finite dimensional case where only a single particle moves and hence the dynamics does not depend on the mass. We derive the theory, compute the PDEs for accelerated optimization, and illustrate the behavior of these new accelerated optimization schemes.

Automated summary

This research considers the optimization of cost functionals on the infinite dimensional manifold of diffeomorphisms. It presents a new class of optimization methods that generalize Nesterov accelerated optimization to the manifold of diffeomorphisms. By establishing connections with simple mechanical principles from fluid mechanics, it derives surprisingly simple continuum evolution equations for accelerated gradient descent. The approach has natural connections to the optimal mass transport problem.