A sensitivity-based construction approach to sample-path variance minimization of Markov decision processes

We study the limiting average variance along the sample path as the secondary criterion for Markov decision processes, with the long-run average performance as the primary criterion. By applying the sensitivity-based approach, we intuitively construct the difference formula for the sample-path variance under different policies. Thereby, a sufficient condition for the sample-path variance optimality can be easily derived. This work extends the sensitivity-based construction approach to the Markov decision processes with the nonstandard performance criterion. Compared with the pure mathematical verification, the sensitivity-based construction approach shows more intuition and provides insights on the sample-path structure of Markov decision processes.