Memory augmented deep reinforcement learning in non-markovian environments

Abstract

Artificial neural networks currently provide the best performance in the field of reinforcement learning. Here, a new deep reinforcement learning framework, called the deep recurrent external memory Q-Network (DReEMQN) is proposed. It consists of a deep neural network and an external memory matrix. The neural network acts as a function approximator for the deep Q-learning algorithm. The external memory is manipulated by a recurrent layer from with in the network, creating a true integrative framework for both learning and remembering what was learnt. The external memory is able to remember longer sequences of the observed states and actions taken by the reinforcement learning agent, and thus this framework is aimed at functioning well in real world environments where the environment state description is limited and long term dependencies on previous actions and states are needed to be remembered. DReEMQN is then tested on partially observable grid world environments of multiple sizes and is compared with a Deep Recurrent Q-Network, which does not have an external memory. The results obtained from the said experiments confirmed that the external memory integration in a deep reinforcement learning algorithm aids the algorithm to perform better in environments where observations are limited and an internal representation of the unobserved environment states and the agent’s past actions is needed.