Algorithm and application of imitation learning and reinforcement learning for sequential decision making problems with multiple agents

Abstract

Systems, in systems engineering, are characterized by a large number of interrelated elements to achieve predefined objectives. Therefore, the general purpose of the sequential decision problem defined in the engineering system is to operate the system according to the predefined objectives through decision making based on the recognized information at each decision epoch. Because a system consists of many components that are related to each other, many decision problems in a system often need to be treated as sequential decision-making problems with multiple agents. In this dissertation, we study various sequential decision-making problems with multiple agents, using the emergency medical service system as a major application domain. Markov Decision Processes (MDP), decentralized-Partially Observable MDP (dec-POMDP), stochastic game (SG) models are mainly used, and multi-agent reinforcement learning (MARL) and imitation learning algorithms are used to solve difficult problems to solve. A typical problem we used is a selective patient admission problem at an ED after a mass-casualty incident. In Chapter 2, we formulate and analyze the MDP model for the selective patient admission problem by focusing on a single ED. The structural properties of the optimal policy of MDP model are reviewed and we identify the variation of an optimal policy according to the characteristics of the input functions that represent the external factors affecting decision making. We propose the solution method for partially-observable multi-agent problems in disaster response operations in Chapter 3. A Dec-POMDP model is suitable for the sequential decision-making problems in disaster response because it assumes the situation where multiple decision-makers choose actions based on partial information. We propose a solution method for dec-POMDP problems in disaster response by combining MARL and behavior cloning (BC) technique of imitation learning. The proposed solution method uses reference policies from the previous research on disaster response through the imitation learning method. We utilize the domain knowledge about the problem through BC to pretrain policy network and value network which will be used in reinforcement learning. As a case of using the proposed solution method, we generalize the mathematical model for the selective patient admission problem to the dec-POMDP model. The proposed solution method significantly reduces the computation time than the MARL algorithm which does not use pretraining and can obtain a near optimal dec-POMDP policy in which performance is close to the upper bound value of a problem. Besides, we find through various numerical experiments that the proposed method is still effective in inherently partially observable environments and the cases when decisions at the prehospital phase effects on the performance of selective patient admission strategy. In Chapter 4, we propose a method to improve a cooperative MARL algorithm using the imitation learning method. This method is using the reference policy obtained from the decision environment with more information than the situation assumed in a dec-POMDP problem to find a solution to a dec-POMDP problem. It collects the demonstrations from the solution of an multi-agent MDP (MMDP) or multi-agent POMDP (MPOMDP) model to mix these demonstrations when training a policy network in an MARL algorithm. We discover that the baseline MARL algorithm can obtain a better dec-POMDP policy when we mix demonstrations from a solution of a centralized model through the experiments in benchmark dec-POMDP problems. A comparison test shows that the method of mixing demonstration is more effective than the another method of using demonstrations to improve an MARL algorithm. We also find that investing a computational budget to learn a centralized policy in the earlier training steps is effective when a reference centralized policy is not provided.