Value-instilled diffusion for offline goal-conditioned reinforcement learning

Abstract

This paper discusses an approach to offline Goal-Conditioned Reinforcement Learning(GCRL) using a diffusion model. GCRL is a problem of learning policies that depend on given goals, and it is important to extract useful information from sparse reward signals. To address this, the proposed methodology introduces a planning-based approach using a conditional diffusion model to generate trajectories for taking actions. Specifically, a goal relabeling method is proposed to overcome sparse rewards, and a conditional diffusion model is tasked to generate trajectories that satisfy certain action value level. The proposed methodology demonstrates high performance in various GCRL experimental environments, as demonstrated through qualitative and quantitative evaluations.