Quantifying Reinforcement-Learning Agent's Autonomy, Reliance on Memory and Internalisation of the Environment

Intuitively, the level of autonomy of an agent is related to the degree to which the agent's goals and behaviour are decoupled from the immediate control by the environment. Here, we capitalise on a recent information-theoretic formulation of autonomy and introduce an algorithm for calculating autonomy in a limiting process of time step approaching infinity. We tackle the question of how the autonomy level of an agent changes during training. In particular, in this work, we use the partial information decomposition (PID) framework to monitor the levels of autonomy and environment internalisation of reinforcement-learning (RL) agents. We performed experiments on two environments: a grid world, in which the agent has to collect food, and a repeating-pattern environment, in which the agent has to learn to imitate a sequence of actions by memorising the sequence. PID also allows us to answer how much the agent relies on its internal memory (versus how much it relies on the observations) when transitioning to its next internal state. The experiments show that specific terms of PID strongly correlate with the obtained reward and with the agent's behaviour against perturbations in the observations.

Automated summary

In this study, we introduce an algorithm for computing the level of autonomy of an agent using an information-theoretic formulation. We use the partial information decomposition framework to monitor the autonomy level and environment internalization of reinforcement learning agents. Our experiments show strong correlations between specific PID terms and the obtained reward, as well as the agent's behavior in response to perturbations in the observations.