Continuous-time reinforcement learning for robust control under worst-case uncertainty

Reinforcement learning (RL) is an effective method to design a robust controller for unknown nonlinear systems. Uncertainty in the worst case requires a large state-action space. Hence, it is natural to use continuous-time RL methods rather than the discretisation of the spaces. In this paper, we propose a novel continuous-time RL using neural network approximation. Our method uses worst-case uncertainty to train the continuous-time RL algorithm. The backward Euler approximation is used to approximate the time derivative of the value function. Compared with the actor-critic (AC) algorithm, our method finds the robust control policy in the presence of worst-case uncertainty by taking into account the applied actions. It is shown that the AC algorithm finds the robust controller in less episodes, but its robustness is less than the results presented by our approach. The convergence of the proposed algorithm is analysed using the contraction property and differential equation techniques. The experiments show that our approach is more robust than the model-based LQR method and the well-known AC method.

Automated summary

In this paper, a novel continuous-time reinforcement learning method using neural network approximation is proposed to train algorithms with worst-case uncertainty, resulting in a more robust control policy. Experimental results demonstrate that this approach is more robust than the model-based LQR method and the well-known AC method.