A Study of First-Passage Time Minimization via Q-Learning in Heated Gridworlds

Optimization of first-passage times is required in applications ranging from nanobots navigation to market trading. In such settings, one often encounters unevenly distributed noise levels across the environment. We extensively study how a learning agent fares in 1- and 2- dimensional heated gridworlds with an uneven temperature distribution. The results show certain bias effects in agents trained via simple tabular Q-learning, SARSA, Expected SARSA and Double Q-learning. Namely, the state-dependency of noise triggers convergence to suboptimal solutions and the respective policies follow them for practically long learning times. The high learning rate prevents exploration of regions with higher temperature, while the low enough rate increases the presence of agents in such regions. These biases of temporal-difference-based reinforcement learning methods may have implications for their application in real-world physical scenarios and agent design.

Automated summary

The study reveals bias effects in agents trained using different methods, leading to convergence to suboptimal solutions in practical learning scenarios. High learning rates may inhibit exploration of certain regions, while low rates can increase agent presence in those regions, potentially impacting the application of reinforcement learning methods in real-world scenarios.