Constructing Performance Sensitivities with Sample Paths in Continuous-time Markov Systems

Sensitivity analysis plays an important role in performance optimization of stochastic systems. It provides a unified view to different areas such as perturbation analysis, Markov decision processes, and reinforcement learning. Furthermore, with the sample path based construction of sensitivity this approach leads to some new research directions such as the event-based optimization approach (Cao, 2005). The previous results are on discrete-time Markov chains (Cao, 2004) and in this paper, we extend the sample path based construction approach to continuous-time Markov processes. The complexity involved is that in continuous-time Markov processes the transition rate also changes in addition to the changes in the transition probability matrix