Optimal index policies for MDPs with a constraint

The search for optimal policies in Markov decision processes (MDPs) is considered. Many controlled queuing systems poses simple index-type optimal policies, when discounted, average, or finite-time cost criteria are considered. For constrained optimization problems, the index structure of the optimal policies is in general not preserved. The authors provide a framework under which the solution of the constrained optimization problem uses the same index policies as the nonconstrained problem. The method is applicable to the discrete-time Klimov system, which is shown to be equivalent to the open bandit problem