Learning multiple roles of heterogeneous multi-agent system with centralized team training

Abstract

In multi-agent reinforcement learning, getting cooperative behavior among agents is one of the most important issues. In addition, in the field of heterogeneous multi-agent reinforcement learning with different types of homogeneous agents, it is crucial to obtain cooperative behavior between different groups of agents, and also important to perform roles of same groups of agents. Learning joint-action set of value using centralized training is an attractive way to getting cooperative behavior in multi-agent reinforcement area. However, this training method have limitations in the heterogeneous reinforcement learning field and require additional work. Our solution is Two Branch Heterogeneous Centralized Training (TBHCT), a novel heterogeneous multi-agent reinforcement learning method that can learn multiple roles with centralized training for cooperative behavior. During training, we divide the training process into two branches, branch one is training the same types of agents with role rewards and branch two is training all the heterogeneous agents with total reward. Our results in 5 versus 5 robot soccer game system with simulated physics for heterogeneous soccer robots setting show that TBHCT can learn cooperative soccer strategies of one goalkeeper, two defenders, and two forwards. Also, the soccer robots trained with the TBHCT algorithm achieve a wining rate of 80% or more in all matches of 90 minutes.