Model-Free λ-Policy Iteration for Discrete-Time Linear Quadratic Regulation

This article presents a model-free lambda-policy iteration (lambda-PI) for the discrete-time linear quadratic regulation (LQR) problem. To solve the algebraic Riccati equation arising from solving the LQR in an iterative manner, we define two novel matrix operators, named the weighted Bellman operator and the composite Bellman operator. Then, the lambda-PI algorithm is first designed as a recursion with the weighted Bellman operator, and its equivalent formulation as a fixed-point iteration with the composite Bellman operator is shown. The contraction and monotonic properties of the composite Bellman operator guarantee the convergence of the lambda-PI algorithm. In contrast to the PI algorithm, the lambda-PI does not require an admissible initial policy, and the convergence rate outperforms the value iteration (VI) algorithm. Model-free extension of the lambda-PI algorithm is developed using the off-policy reinforcement learning technique. It is also shown that the off-policy variants of the lambda-PI algorithm are robust against the probing noise. Finally, simulation examples are conducted to validate the efficacy of the lambda-PI algorithm.

Automated summary

This article presents a model-free lambda-policy iteration algorithm for the discrete-time linear quadratic regulation problem. It introduces the weighted Bellman operator and the composite Bellman operator to solve the algebraic Riccati equation. Compared to the PI algorithm, the lambda-PI algorithm does not require an initial policy and has a faster convergence rate. The model-free extension of the lambda-PI algorithm is developed using the off-policy reinforcement learning technique.