Socially augmented hierarchical reinforcement learning for reducing complexity in cooperative multi-agent systems

This paper addresses the inherent complexity in coordinating learned behavioral strategies of multiple agents working towards a common goal. Because of the interactions among the agents, a primary challenge of policy learning is escalating computational complexity with increasing number of agents and the size of the task space (including action choices and world states). We employ an approach that incorporates social constructs based on analogs from biological systems of high functioning mammals in order to constrain state-action choices in reinforcement learning. Additionally, we use state-space abstraction and a hierarchical learning structure to improve learning efficiency. Theoretical results bound the reduction in computational complexity due to state abstraction, hierarchical learning, and socially-constrained action selection in learning problems that can be described as decentralized Markov decision processes. Simulation results show that these theoretical bounds hold and that satisficing multi-agent coordination policies emerge, which reduce task completion time, computational cost, and memory resources compared to learning with no social knowledge.