Anticipative reinforcement learning

This paper introduces anticipative reinforcement learning (ARL), a method that addresses the problem of the breakdown of value based algorithms for problems with small time steps and continuous action and state spaces when the algorithms are implemented with neural networks. In ARL, an agent is made of three components; the actor, the critic and the model (the model is as in Dyna but we use it differently). The main originality of ARL lies in the action selection process; the agent builds a set of candidate actions that includes the action recommended by the actor plus some random actions. Once the set of candidate actions is built, the candidate actions are ranked by considering what would happen if these actions were taken and followed by a sequence of actions using only the current policy (anticipation using iteratively the model with a finite look-ahead). We demonstrate the benefits of looking ahead with experiments on a Khepera robot.