期刊
INFORMATION SCIENCES
卷 558, 期 -, 页码 157-173出版社
ELSEVIER SCIENCE INC
DOI: 10.1016/j.ins.2020.12.075
关键词
Stochastic optimization; Mini batches; Recursive iteration; Variance reduction
资金
- China Postdoctoral Science Foundation [2019M663238]
SARAH method's performance relies on the choice of step size sequence, leading to the proposal of MB-SARAH-RBB method, which is proven to linearly converge in expectation for strongly convex objective functions and has better gradient complexity. Numerical experiments show the superiority of the proposed methods.
StochAstic Recursive grAdient algoritHm (SARAH), originally proposed for convex optimization and also proven to be effective for general nonconvex optimization, has received great attention because of its simple recursive framework for updating stochastic gradient estimates. The performance of SARAH significantly depends on the choice of the step size sequence. However, SARAH and its variants often manually select a best-tuned step size, which is time consuming in practice. Motivated by this gap, we propose a variant of the Barzilai-Borwein (BB) method, referred to as the Random Barzilai-Borwein (RBB) method, to determine the step size for SARAH in the mini-batch setting, leading to a new SARAH method: MB-SARAH-RBB. We prove that MB-SARAH-RBB converges linearly in expectation for strongly convex objective functions. Moreover, we analyze the gradient complexity of MB-SARAH-RBB and show that it is better than the original method. To further confirm the efficacy of the RBB method, we propose the MB-SARAH+-RBB method, by incorporating it into the MB-SARAH + method. Numerical experiments on standard data sets indicate that our proposed methods outperform or match state-of-the-art algorithms. (C) 2021 Elsevier Inc. All rights reserved.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据