期刊
JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION
卷 -, 期 -, 页码 -出版社
TAYLOR & FRANCIS INC
DOI: 10.1080/01621459.2022.2096620
关键词
Asymptotic normality; Multiplier bootstrap; Reinforcement learning; Statistical inference; Stochastic approximation
资金
- National Science Foundation [2048075, 2008827, 2015568, 1934931]
- Simons Institute
- Amazon
- J.P. Morgan
- Two Sigma
- ONR [N00014-18-1-2759]
- National Science Foundation (NSF - SCALE MoDL) [2134209]
- Direct For Mathematical & Physical Scien
- Division Of Mathematical Sciences [2134209] Funding Source: National Science Foundation
The recent emergence of reinforcement learning has led to a demand for robust statistical inference methods. Existing methods for inference in online learning are not applicable in RL, this article explores the use of the online bootstrap method in RL policy evaluation and demonstrates its effectiveness through experiments.
The recent emergence of reinforcement learning (RL) has created a demand for robust statistical inference methods for the parameter estimates computed using these algorithms. Existing methods for inference in online learning are restricted to settings involving independently sampled observations, while inference methods in RL have so far been limited to the batch setting. The bootstrap is a flexible and efficient approach for statistical inference in online learning algorithms, but its efficacy in settings involving Markov noise, such as RL, has yet to be explored. In this article, we study the use of the online bootstrap method for inference in RL policy evaluation. In particular, we focus on the temporal difference (TD) learning and Gradient TD (GTD) learning algorithms, which are themselves special instances of linear stochastic approximation under Markov noise. The method is shown to be distributionally consistent for statistical inference in policy evaluation, and numerical experiments are included to demonstrate the effectiveness of this algorithm across a range of real RL environments. Supplementary materials for this article are available online.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据