An analysis of gradient-based policy iteration

Recently, a system theoretic framework for learning and optimization has been developed that shows how many approximate dynamic programming paradigms such as perturbation analysis, Markov decision processes, and reinforcement learning are very closely related. Using this system theoretic framework, a new optimization technique called gradient-based policy iteration (GBPI) has been developed. In this paper, we show how GBPI iteration can be extended to partially observable Markov decision processes (POMDPs). We also develop the value iteration analogue of GBPI and show that this new version of value iteration, extended to POMDPs, not only theoretically acts like value iteration but also does so numerically.