Journal
COMMUNICATIONS PHYSICS
Volume 3, Issue 1, Pages -Publisher
NATURE PUBLISHING GROUP
DOI: 10.1038/s42005-020-0373-8
Keywords
-
Categories
Funding
- DARPA [HR00111990069]
- DoD-SMART fellowship
- Center for Memory and Recording Research at the University of California, San Diego
Ask authors/readers for more resources
Restricted Boltzmann machines (RBMs) are a powerful class of generative models, but their training requires computing a gradient that, unlike supervised backpropagation on typical loss functions, is notoriously difficult even to approximate. Here, we show that properly combining standard gradient updates with an off-gradient direction, constructed from samples of the RBM ground state (mode), improves training dramatically over traditional gradient methods. This approach, which we call 'mode-assisted training', promotes faster training and stability, in addition to lower converged relative entropy (KL divergence). We demonstrate its efficacy on synthetic datasets where we can compute KL divergences exactly, as well as on a larger machine learning standard (MNIST). The proposed mode-assisted training can be applied in conjunction with any given gradient method, and is easily extended to more general energy-based neural network structures such as deep, convolutional and unrestricted Boltzmann machines. While widely adopted, contrastive divergence methods for Restricted Boltzmann Machines typically result in poor representations of the data distribution. Here, the authors propose an unsupervised training where gradient-descent is combined with the Machine's mode samples, significantly improving the final model quality.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available