Singularly perturbed Markov control problem: limiting average cost

Consideration is given to a singularly perturbed Markov decision process (MDP) with the limiting average cost criterion. It is assumed that the underlying process is composed of n separate irreducible processes, and that the small perturbation is such that it `unites´ these processes into a single irreducible process. This structure corresponds to the Markov chains admitting strong and weak interactions, a situation that arises in many applications. The underlying control problem for the singularly perturbed MDP, the so-called limit Markov control problem (limit MCP) is formulated, and the relationship between the two is investigated. In particular, it is shown that an optimal solution to the perturbed MDP can be approximated by an optimal solution of the limit MCP for sufficiently small perturbation. It is demonstrated that the limit MCP is equivalent to a suitably constructed nonlinear program in the space of long-run state-action frequencies. It is shown that the suboptimal strategy for the singularly perturbed MDP can be considered as the outcome of a certain two-level optimization problem exhibiting the aggregation and disaggregation properties of the limit MCP