HCMDP: A hierarchical solution to Constrained Markov Decision Processes

Constrained Markov Decision Processes offer a principled way to tackle sequential decision problems with multiple objectives. Although they could be very valuable in numerous robotic applications, to date their use has been quite limited. One of the reasons is that their solution requires to solve constrained linear programs with a large number of variables and this is computationally demanding, especially when considering dynamic environments. In this paper we propose a hierarchical approach to solve large CMDPs. States are clustered into macro states and relevant parameters like transition probabilities and costs are extracted with a Monte Carlo approach. Macro states are created with the objective of grouping together states with similar costs while preserving feasibility. We illustrate the value of our findings in a path planning scenario where the robot moves through an environment characterized by different risk levels. Our approach largely outperforms the non-hierarchical method and we also show how it prevails over methods based on fixed partitioning strategies.