Efficient solution of Markov decision problems with multiscale representations

Many problems in sequential decision making and stochastic control naturally enjoy strong multiscale structure: sub-tasks are often assembled together to accomplish complex goals. However, systematically inferring and leveraging hierarchical structure has remained a longstanding challenge. We describe a fast multiscale procedure for repeatedly compressing or homogenizing Markov decision processes (MDPs), wherein a hierarchy of sub-problems at different scales is automatically determined. Coarsened MDPs are themselves independent, deterministic MDPs, and may be solved using any method. The multiscale representation delivered by the algorithm decouples sub-tasks from each other and improves conditioning. These advantages lead to potentially significant computational savings when solving a problem, as well as immediate transfer learning opportunities across related tasks.