Sample-efficient deep reinforcement learning via episodic backward update

Abstract

We propose Episodic Backward Update – a new algorithm to boost the performance of a deep reinforcement learning agent by a fast reward propagation. In contrast to the conventional use of the experience replay with uniform random sampling, our agent samples a whole episode and successively propagates the value of a state to its previous states. Our computationally efficient recursive algorithm allows sparse and delayed rewards to propagate efficiently through all transitions of a sampled episode. We evaluate our algorithm on 2D MNIST maze environment and 49 games of the Atari 2600 environment, and show that our method improves sample efficiency with a competitive amount of computational cost.