Addressing double sampling issue by learning dynamics model

Abstract

With the recent advancements in deep neural networks, reinforcement learning has demonstrated remarkable performance in various fields such as games, language models, and robotics. However, currently prevalent reinforcement learning algorithms employ the target network to address the double sampling issue, which necessitates an additional Q-network and delays the update. In this thesis, we tackle the aforementioned problem by training the dynamics model instead of using the target network, aiming to resolve the double sampling issue. Specifically, our approach modified deep Q-network by sampling another independent next state from the learned dynamics model and introducing a new loss function that takes into account the double sampling issue. With the proposed method, we aim to optimize the Q-network through a more precise gradient closer to the true gradient of mean squared Bellman error. In experiments, the proposed algorithm robustly achieved higher undiscounted returns and predicted action-values more stably compared to deep Q-network.