Connect-Based Subgoal Discovery for Options in Hierarchical Reinforcement Learning

This paper presents a new method which uses a connect-based thought to automatically discover subgoals in a dynamic environment. We argue that the states which are not only visited frequently in the whole space but also have a relative high visiting frequency in their neighboring regions performing a critical role in connecting the neighbor states are subgoals, and then propose a novel algorithm for identifying them by considering one state who has a high sum of the relative exceeding frequencies from all its neighbor states as a subgoal. Most earlier methods actually discover a frequent visited region and then filter and select the critical states in this region, but our method directly discovers the critical states of a frequent visited region in the space without generating excessive useless candidates. Combining the global and local perspectives is a key property of our algorithm and the one that differentiates it from most of existing works in this area. Experiments show that this simple and robust approach detects subgoals quickly and correctly.