Performance enhancement in multigoal reinforcement learning using hindsight experience replay

Abstract

Recent advances in Artificial Intelligence (AI), especially in the area of deep reinforcement learning (RL), have been responsible for breakthrough results in robotics. For a specific type of RL, known as multigoal RL, the agent learns to achieve multiple different goals with a goal-conditioned policy. The goal-conditioned policy is trained to effectively generalize its behavior for multiple goals. At the beginning of training, the agent is still not capable of performing the task successfully and is mostly taking random exploratory actions over the action space. When the goal space is large and rewards are sparse, the exploration phase leads to a very low proportion of successful experiences in the training batches. To this end, hindsight experience replay (HER) increases sampling efficiency by converting unsuccessful episodes into successful episodes substituting the original goal by the goal achieved at the end of the episode. In this thesis, the framework that combines a deep RL algorithm with HER to solve multigoal RL problems is investigated and methods to enhance the final success rate and convergence speed are proposed. Proposed methods are combined with HER for experiments in robotic control tasks to demonstrate enhanced performance when compared to the original framework and other performance enhancement methods.