A Distributed Q-Learning Algorithm for Multi-Agent Team Coordination

Q-learning is an effective model-free reinforcement learning algorithm. However, Q-learning is centralized and competent only for single agent learning but not multi-agent learning because in later case the size of state-action space is huge and will grow exponentially with the number of agents increasing. In the paper we present a distributed Q-learning algorithm to solving this problem. In our algorithm, the tasks of learning optimal action policy are distributed to each agent in team but not a central agent. In order to reduce the size of action-state space of multi-agent team we introduce a state-action space sharing strategy of agent team, through which one agent in team can use the states already explored by other agents before and need not take time to explore these states again. Additionally, our algorithm has the ability to allocate sub-goals dynamically among agents according to environment changing, which can make agent team coordinate more efficiently. Experiments show the efficiency of our algorithm when it is applied to the benchmark problem of predator-prey pursuit game, also called pursuit game, in which a team of predators coordinate to capture a prey.