Deep Belief Network for Modeling Hierarchical Reinforcement Learning Policies

Intelligent agents over their lifetime face multiple tasks that require simultaneous modeling and control of complex, initially unknown environments, observed via incomplete and uncertain observations. In such scenarios, policy learning is subject to the curse of dimensionality, leading to scaling problems for traditional Reinforcement Learning (RL). To address this, the agent has to efficiently acquire and reuse latent knowledge. One way is through Hierarchical Reinforcement Learning (HRL), which embellishes RL with a hierarchical, model-based approach to state, reward and policy representation. This paper presents a novel learning approach for HRL based on Conditional Restricted Boltzmann Machines (CRBMs). The proposed model provides a uniform means to simultaneously learn policies and associated abstract state features, and allows learning and executing hierarchical skills within a consistent, uniform network structure. In this model, learning is performed incrementally from basic grounded features to complex abstract policies based on automatically extracted latent states and rewards.