Algorithms for offline imitation learning with supplementary demonstrations

Abstract

We consider offline imitation learning (IL), which aims to mimic the expert’s behavior from its demonstration without further interaction with the environment. One of the main challenges in offline IL is to deal with the narrow support of the data distribution exhibited by the expert demonstrations that cover only a small fraction of the state and the action spaces. In this thesis, we address the two offline IL problems: imitation learning from (1) state-action demonstrations by experts, and (2) state-only demonstrations by experts. First, we consider the problem of learning from demonstrations (LfD), in which the agent aims to mimic the expert’s behavior from the state-action demonstrations by experts. Compared with the recent LfD algorithms that adopt adversarial minimax training objectives, we substantially stabilize overall learning process by reducing minimax optimization to a direct convex optimization in a principled manner. Next, we consider the problem of learning from observation (LfO), in which the agent aims to mimic the expert’s behavior from the state-only demonstrations by experts. We introduce an algorithm that solves a single convex minimization problem, which minimizes the divergence between the two state-transition distributions induced by the expert and the agent policy.