期刊
2020 25TH INTERNATIONAL CONFERENCE ON PATTERN RECOGNITION (ICPR)
卷 -, 期 -, 页码 347-353出版社
IEEE COMPUTER SOC
DOI: 10.1109/ICPR48806.2021.9413027
关键词
-
类别
资金
- SHARCNET
- Compute Canada
- Philip Beesley Architect, Inc.
- SSHRC Partnership Grant in collaboration
Multi-step methods in reinforcement learning have been shown to improve performance, but the specific contributions to performance enhancement are still not well understood. This study analyzes the effect of multi-step methods on alleviating the overestimation problem in deep reinforcement learning, proposing two improved methods, MDDPG and MMDDPG, based on Deep Deterministic Policy Gradient.
Multi-step (also called n-step) methods in Reinforcement Learning (RL) have been shown to be more efficient than the 1-step method due to faster propagation of the reward signal, both theoretically and empirically, in tasks exploiting tabular representation of the value-function. Recently, research in Deep Reinforcement Learning (DRL) also shows that multi-step methods improve learning speed and final performance in applications where the value-function and policy are represented with deep neural networks. However, there is a lack of understanding about what is actually contributing to the boost of performance. In this work, we analyze the effect of multi-step methods on alleviating the overestimation problem in DRL, where multi-step experiences are sampled from a replay buffer. Specifically building on top of Deep Deterministic Policy Gradient (DDPG), we propose Multistep DDPG (MDDPG), where different step sizes are manually set, and a variant called Mixed Multi-step DDPG (MMDDPG) where an average over different multi-step backups is used as an update target for the Q-value function. Empirically, we show that both MDDPG and MMDDPG are significantly less affected by the overestimation problem than DDPG with 1-step backup, which consequently results in better final performance and learning speed. We also discuss the advantages and disadvantages of different ways to do multi-step expansion in order to reduce approximation error, and expose the tradeoff between overestimation and underestimation that underlies offline multi-step methods. Finally, we compare the computational resource needs of MDDPG and MMDDPG with those of Twin Delayed Deep Deterministic Policy Gradient (TD3), a state-of-the-art algorithm proposed to address overestimation in actor-critic methods, since they show comparable final performance and learning speed.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据