Balanced Gradient Training of Feed Forward Networks

We show that there are infinitely many valid scaled gradients which can be used to train a neural network. A novel training method is proposed that finds the best scaled gradients in each training iteration. The method's implementation uses first order derivatives which makes it scalable and suitable for deep learning and big data. In simulations, the proposed method has similar or less testing error than conjugate gradient and Levenberg Marquardt. The method reaches the final network utilizing fewer multiplies than the other two algorithms. It also works better than conjugate gradient in convolutional neural networks.

Automated summary

The study demonstrates the existence of infinitely many valid scaled gradients for neural network training, proposing a novel training method that outperforms conjugate gradient and Levenberg Marquardt, and is scalable for deep learning and big data. It achieves similar or lower testing error than the other two algorithms and requires fewer multiplies to reach the final network. Additionally, it performs better than conjugate gradient in convolutional neural networks.