Block-cyclic stochastic coordinate descent for deep neural networks

We present a stochastic first-order optimization algorithm, named block-cyclic stochastic coordinate descent (BCSC), that adds a cyclic constraint to stochastic block-coordinate descent in the selection of both data and parameters. It uses different subsets of the data to update different subsets of the parameters, thus limiting the detrimental effect of outliers in the training set. Empirical tests in image classification benchmark datasets show that BCSC outperforms state-of-the-art optimization methods in generalization leading to higher accuracy within the same number of update iterations. The improvements are consistent across different architectures and datasets, and can be combined with other training techniques and regularizations. (C) 2021 Elsevier Ltd. All rights reserved.

Automated summary

BCSC is a stochastic first-order optimization algorithm that adds a cyclic constraint to the selection of data and parameters, resulting in higher accuracy in image classification. It effectively limits the impact of outliers in the training set and provides better generalization performance within the same number of update iterations.