Applying statistical generalization to determine search direction for reinforcement learning of movement primitives

In this paper we present a new methodology for robot learning that combines ideas from statistical generalization and reinforcement learning. First we apply statistical generalization to compute an approximation for the optimal control policy as defined by training movements that solve the given task in a number of specific situations. This way we obtain a manifold of movements, which dimensionality is usually much smaller than the dimensionality of a full space of movement primitives. Next we refine the policy by means of reinforcement learning on the approximating manifold, which results in a learning problem constrained to the low dimensional manifold. We show that in some situations, learning on the low dimensional manifold can be implemented as an error learning algorithm. We apply golden section search to refine the control policy. Furthermore, we propose a reinforcement learning algorithm with an extended parameter set, which combines learning in constrained domain with learning in full space of parametric movement primitives, which makes it possible to explore actions outside of the initial approximating manifold. The proposed approach was tested for learning of pouring action both in simulation and on a real robot.