A unifying framework for HAMs-family HRL methods

In the HRL (hierarchical reinforcement learning) field, there are three main methods such as HAMs (hierarchical abstract machines), options, MAXQ. These methods all rely on the theory of SMDPs. While the SMDP framework allows us to directly model the high-level actions that take varying amounts of time, it provides little in the way of concrete representational guidance, which is critical from a computational and analytical point of view. In particular, the SMDP framework does not specify how the overall task can be decomposed into a collection of subtasks, which is important for us to do state abstraction and subtask sharing for individual subtask or module. In addition, we also want to choose between hierarchical optimality and recursive optimality for a given hierarchy on our problem. This paper introduces a unifying framework for HAMs-family methods. Based on this framework, we can define HAMs or sub- HAM homomorphism for state abstraction and can also freely select alternative policy optimality.