Riemannian gradient methods for stochastic composition problems

Review Mathematics, Applied

DISTRIBUTIONALLY ROBUST OPTIMIZATION: A REVIEW ON THEORY AND APPLICATIONS

Fengming Lin et al.

Summary: This paper surveys the primary research on the theory and applications of distributionally robust optimization (DRO). It reviews the modeling power and computational attractiveness of DRO approaches, summarizes efficient solution methods, performance guarantees, and convergence analysis. Additionally, the paper illustrates applications of DRO in machine learning and operations research, and discusses future research directions.

NUMERICAL ALGEBRA CONTROL AND OPTIMIZATION (2022)

添加到收藏夹

Article Computer Science, Artificial Intelligence

Orthogonal Deep Neural Networks

Shuai Li et al.

Summary: This paper introduces the algorithms of Orthogonal Deep Neural Networks (OrthDNNs) to improve generalization performance by connecting with recent interest in spectrally regularized deep learning methods. Theoretical analyses and experiments demonstrate that OrthDNNs can achieve local isometric properties on practical data distributions, leading to better optimization of network weights. Proposed algorithms, including strict and approximate OrthDNNs, along with the SVB and BBN methods, show effective and efficient performance in benchmark image classification tasks.

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE (2021)

添加到收藏夹

Article Engineering, Electrical & Electronic

Solving Stochastic Compositional Optimization is Nearly as Easy as Solving Stochastic Optimization

Tianyi Chen et al.

Summary: Stochastic compositional optimization generalizes classic stochastic optimization for minimizing compositions of functions, with applications in reinforcement learning and meta learning. The new Stochastically Corrected Stochastic Compositional gradient method (SCSC) ensures convergence at the same rate as traditional methods and can be accelerated with SGD techniques. Applying Adam to SCSC achieves state-of-the-art performance in stochastic compositional optimization, tested in model-agnostic meta-learning tasks.

IEEE TRANSACTIONS ON SIGNAL PROCESSING (2021)

添加到收藏夹

Article Computer Science, Artificial Intelligence