Reinforcement Learning with Inertial Exploration

In the Q-learning framework, the exploration of large environment is influenced by the time credit assignment problem. In this context, abstraction techniques may be used. Thus, multi-step actions (MSA) Q-learning has been proposed to take advantage of the fact that few action switches are usually required in optimal policies. In this article, we propose the concept of inertial exploration, we apply a log-selection of the scales to MSA Q-learning and we go further by proposing a dynamic time scale approach. We demonstrate that the same improvement in learning speed can be achieved without the full scales set. This improvement is shown on the mountain car problem and on a more realistic application of vehicle control.