Prefrontal solution to the bias-variance tradeoff during reinforcement learning

Evidence that the brain combines different value learning strategies to minimize prediction error is accumulating. However, the tradeoff between bias and variance error, which imposes different constraints on each learning strategy's performance, poses a challenge for value learning. While this tradeoff specifies the requirements for optimal learning, little has been known about how the brain deals with this issue. Here, we hypothesize that the brain adaptively resolves the bias-variance tradeoff during reinforcement learning. Our theory suggests that the solution necessitates baseline correction for prediction error, which offsets the adverse effects of irreducible error on value learning. We show behavioral evidence of adaptive control using a Markov decision task with context changes. The prediction error baseline seemingly signals context changes to improve adaptability. Critically, we identify multiplexed representations of prediction error baseline within the ventrolateral and ventromedial prefrontal cortex, key brain regions known to guide model based and model-free reinforcement learning.

Automated summary

The brain has been found to adaptively resolve the tradeoff between bias and variance during reinforcement learning, requiring baseline correction for prediction error to offset the adverse effects of irreducible error on value learning. Behavioral evidence of adaptive control has been shown in a Markov decision task with context changes, suggesting that the prediction error baseline signals context changes to improve adaptability. Multiplexed representations of prediction error baseline within specific brain regions have been identified, indicating their role in guiding model based and model-free reinforcement learning.