Reinforcement learning with supervision by a stable controller

Reinforcement learning (RL) methods provide a means for solving optimal control problems when accurate models are unavailable. For many such problems, however, RL alone is impractical and the associated learning problem must be structured somehow to take advantage of prior knowledge. In this paper we examine the use of such knowledge in the form of a stable controller that generates control inputs in parallel with an RL system. The controller acts as a supervisor that not only teaches the RL system about favorable control actions but also protects the learning system from risky behavior. We demonstrate the approach with a simulated robotic arm and a real seven-DOF manipulator.