Single sample path based optimization of Markov systems: examples and algorithms

Motivated by the needs of online optimization of real world engineering systems, we study the single sample path based algorithms for Markov decision problems (MDPs). We give a simple example to explain the advantages of the sample path based approach over the traditional computation based approach: matrix inversion is not required; some transition probabilities do not have to be known; it may save storage space; and it gives the flexibility of iterating the actions for a subset of the state space in each iteration. The effect of the estimation errors and the convergence property of the sample path based approach are studied. Finally, we propose a “fast” algorithm which updates the policy whenever the system reaches a particular set of states; the algorithm converges to the true optimal policy with probability one under some conditions