Estimating the level of inference using an order-mimic agent

Abstract

Multi-agent reinforcement learning (RL) considers problems of learning policies and predicting values through interactions with multiple opponents. To make the solutions feasible, one assumes single-type opponents. However, this may not hold in most real-world situations. Interactions with a mixture of different types of agents make it extremely hard to learn. This study examines the hypothesis that when the potential types of agents are unknown, the level of agent inference can act as a proxy for characterizing the opponents. We present a computational framework to estimate the level of agent's inference using a deterministic and stochastic order-mimic agent. We then propose a calibration method for unbiased estimation, which offsets the adverse effect of order-mimic agents on the environment's order estimation. Finally, to generalize the method to a wide range of contexts, we proposed iterative inference level estimation. We demonstrate the feasibility of the proposed method in computer simulations with agents mimicking agents' behavior with various inference levels. Our framework can estimate the learning capacity of various algorithms and humans; therefore it can be used to design high-level inference models that can effectively handle the complexity of multi-agent learning problems.