Midbrain dopamine activity during reinforcement learning reflects bias-variance tradeoff

Abstract

To maximize reward in a context-changing environment, a reinforcement learning (RL) agent should quickly recognize the changes before adaptation (Fig 1A), which can be facilitated by bias error signaling. Previous studies have shown that, when an animal knows that the environment switches from one familiar setting to another, dopamine reward prediction error (RPE) coding instantly adapts to the change [1]. However, environmental changes are often hidden and have to be inferred. Little is known about dopamine activity while an animal tries to figure out the current context corresponds to which environmental setting it previously learned. We hypothesized that dopamine signals variance (or RPE) in a stable environment, but encodes bias error while the animal in a changing environment is trying to figure out which bias fits the current environmental setting. To test this hypothesis, we analyzed dopamine spike trains recorded while rats were performing a probabilistic two-arm bandit reversal learning task. Previous studies have suggested that animals extensively trained in a reversal task learns the task structure, and use the knowledge to bias animals’ choices [2]. We found that extensively trained rats tend to choose a rewarded option consistently but occasionally tested the other option. Dopamine activity during the brief testing resembled bias error. These findings suggest that dopamine activity not only encodes RPE but also signals the bias error in a changing environment. In the broader context, the results also imply that dopamine has the capacity to signal information necessary for adaptation in a volatile environment.