Error controlled actor-critic

The approximation inaccuracy of the value function in reinforcement learning (RL) algo-rithms unavoidably leads to an overestimation phenomenon, which has negative effects on the convergence of the algorithms. To limit the negative effects of the approximation error, we propose error controlled actor-critic (ECAC) which ensures the approximation error is limited within the value function. We present an investigation of how approxima-tion inaccuracy can impair the optimization process of actor-critic approaches. In addition, we derive an upper bound for the approximation error of the Q function approximator and discover that the error can be reduced by limiting the KL-divergence between every two consecutive policies during policy training. Experiments on a variety of continuous control tasks demonstrate that the proposed actor-critic approach decreases approximation error and outperforms previous model-free RL algorithms by a significant margin.(c) 2022 Elsevier Inc. All rights reserved.

Automated summary

This article discusses the negative effects of the approximation inaccuracy in reinforcement learning algorithms on algorithm convergence. To mitigate the impact of approximation error, the error controlled actor-critic (ECAC) approach is proposed, which limits the approximation error within the value function. Experimental results demonstrate that the proposed actor-critic approach reduces approximation error and outperforms previous model-free RL algorithms.