Algorithms for safe reinforcement learning

Abstract

Standard reinforcement learning (RL) aims to learn a reward-maximizing policy through online interaction with the MDP environment. However, in many real-world domains, naive application of RL may be problematic especially when some behaviors of the agent can cause irrecoverable damage to the agent itself or its surroundings. Therefore, for RL to be applied to practical problems, we should consider the notion of safety in the process of policy learning and execution. In this thesis, we address safety in RL from two perspectives: (1) safety via offline learning, and (2) safety via constraints. First, we consider the offline RL problem where the agent optimizes the policy solely from the pre-collected experiences, whose learning process is essentially safe in that it does not involve taking actions sampled from the unoptimized policy in the real environment. We present two different offline RL algorithms using gradient-based hyperparameter optimization and using stationary distribution correction estimation. Second, we consider the constrained MDP (CMDP), which provides a framework to encode safety specifications through cost constraints. We present a scalable solution method for CMDPs based on the Monte-Carlo tree search. Lastly, we consider the offline constrained RL problem that lies in the intersection of the two safety considerations. We introduce an efficient offline constrained RL algorithm that aims to compute a cost-conservative policy for actual constraint satisfaction by constraining the cost upper bound.