Tutor learning using linear constraints in approximate dynamic programming

In adaptive control, agents interacting with Markov Decision Processes typically face two types of setups. In the first setup, the environment´s model is known and dynamic programming and related methods are used to obtain the optimal control. In the second setup, the environment´s model is unknown and reinforcement learning methods are used. In this work we investigate a new setup that is a mix of the two mentioned setups: only part of the environment´s model is known and additional information regarding the environment is provided by a tutor. We formalize this problem using linear function approximation in order to overcome the “curse of dimensionality” phenomenon. In addition, using the Envelope Theorem, we show how one can tune the approximation basis in order to get a locally optimal results. Finally, the suggested methods are demonstrated in simulations.