Robust imitation learning

Abstract

RL (Reinforcement Learning) aims to produce a policy that can perform a given task well. The recent studies of RL show remarkable results in various fields such as robot control simulation and games, however, these studies fail to deploy their approaches into the real world. The main obstacles to deploying RL into the real world are well-designing dense reward issue and non-robust policy issue. RL heavily relies on well-designed dense reward and produces non-robust policy that dramatically deteriorates when it faces with environment which has unseen environment dynamics during training. Imitation RL is one among approaches to address well-designing dense reward issues. However, this Imitation RL also suffers from non-robust policy issue. In order to solve such issues, we propose a novel algorithm, called RAME, producing a robust policy through Generative Adversarial Imitation Learning while the agent can access multiple environment. The policy via our algorithm is trained in multiple training environments with expert trajectories in their own training environment. Discriminators and state discriminators make appropriate reward used to train the policy while taking into account the relation between given state-action pair and training environments. It is able to train with any training environments, not only well-selected training environments. The results of these experiments show that it is enable to produce more robust policy than the existing Imitation RL approaches. This implicates that the policy via RAME can adapt to multiple environments without relying on dense rewards.