Exploratory networks using binary rewards for multigoal reinforcement learning

Abstract

In multigoal reinforcement learning, an agent learns a policy to achieve multiple goals by interacting with an environment. With a sparse binary reward and a large state space, success cases do not appear frequently, which slows down the learning speed and increases the learning difficulty. To solve these problems, researches have been on reward design and efficient exploration and experience sampling. However, traditional reward designs may include a developer's bias and efficient exploration is still challenging. In this paper, a method to improve exploration efficiency while minimizing the developer's bias by using exploratory networks using binary rewards. The binary rewards used for exploratory networks involve minimal the developer's bias as a reward that makes the agent take actions that have the possibility of success, but since the main network is isolated and only depends on the goal reward, sparse binary reward, the final behavior of the main network only achieves the goal without involving the developer's bias. The proposed method can increase the exploration efficiency through the same effect as using a combined reward which has multiple terms in the reward function while maintaining the advantages of sparse binary rewards in which the developer's bias is not involved. In this paper, the proposed method is experimented to compare the performance of agent with and without the proposed method in Push task, PickAndPlace task, and Slide task, in which a robot moves an object and sends it to the target point. In three experiments, the cases with the proposed method show a higher success rate or marginally improved success rate compared to the cases without the proposed method.