Hierarchical sub-task decomposition for reinforcement learning of multi-robot delivery mission

In applying reinforcement learning (RL) to multi-robot control, the size of the learning state space easily explodes because the state space has a high dimension. Hierarchical reinforcement learning (HRL) is one of the most practical approaches to solve the problem; however, automatically decomposing a plain MDP state space into sub-spaces has not been studied thoroughly enough to be applied to practical robotics problems. We propose a method that automatically forms hierarchical sub-tasks for multi-robot delivery missions. The method executes sub-task decomposition and the learning process in a step-by-step manner, by widening the robot´s range of movements around the load and gradually decreasing the domain of the load position. The method automatically detects the state in which cooperative motion among the robots is needed for them to accomplish the mission. The performance of the method is demonstrated by simulations.