Self-imitation learning algorithms for goal-oriented dialogues

Abstract

Reinforcement learning (RL) aims to learn a policy that maximizes reward through interaction with the environment. Task-oriented dialogues could be naturally formulated by RL problems. However, when we consider applying the standard RL algorithm to real-world task-oriented dialogues, there are three main challenges to consider: (1) task-oriented dialogues assume offline learning that the agent optimizes the policy from the only previously collected dataset without online environment interaction, (2) standard policy-gradient-based RL methods easily fail and generate responses diverging from human language, and (3) optimizing the task-oriented dialogue agent is very challenging because of the enormous action space of natural language actions. In this thesis, to address these challenges, we present three different RL algorithms based on self-imitation learning that the agent learns the policy to imitate the agent’s own past good decisions generated by itself. First, we present a model-based offline RL algorithm that combines RNN-based dialogue generation and MCTS-based Bayesian planning. Secondly, we present a Monte-Carlo planning algorithm that combines Monte-Carlo tree search with language-driven exploration, then introduce RL algorithm built on this planning algorithm. Lastly, we present a model-free offline RL algorithm that is built upon GPT-2 with fine-tuning the language model through behavior cloning of critic-guided self-generated dialogues.