Off-policy multi-agent policy optimization with agent-wise advantage estimation

Abstract

In many multi-agent environments, agents must coordinate their actions with partial information to cooperate or compete. To overcome miscoordination, the framework appeared that uses central informationn to estimate the global value function. Naturally, adopting the policy gradient method to multi-agent reinforcement learning has been actively studied. However, many of these studies do not deal with credit assignment or only in an implicit way. There have been recent attempts to explicitly design rewards, but they have some weaknesses. In this paper, we investigate reward shaping and credit assignment in multi-agent systems with a theoretical understanding of the trade-off between variance and bias. Also, we study off-policy correction for multi-agent systems. From these, we propose a multi-agent off-policy optimization algorithm based on a new advantage estimator with off-policy correction. The algorithm is capable of off-policy estimation while enabling the control bias and the credit assignment. Empirical evaluations on the StarCraft II benchmark and multi-agent MuJoCo environments demonstrate that our method outperforms recent algorithms.