Event-Based Optimization of Markov Systems

Recent research indicates that Markov decision processes (MDPs) and perturbation analysis (PA) based optimization can be derived easily from two fundamental performance sensitivity formulas. With this sensitivity point of view, an event-based optimization approach, including event-based sensitivity analysis and event-based policy iteration, was proposed via an example by X. R. Cao (Discrete Event Dyn. Syst.: Theory Appl., vol. 15, pp. 169-197, 2005). This approach utilizes the special feature of a system and illustrates how the potentials can be aggregated using the special feature. The approach applies to many practical problems that do not fit well the standard MDP formulation. This note provides a mathematical formulation and proves the main results for this approach.