Adaptive Markov and semi-Markov decision processes: a weak contrast function approach

The authors consider the control of dynamic systems modeled as a family of either Markov or semi-Markov processes, parameterized by an unknown parameter α which takes values in a given finite set A . The true parameter α⁰, which represents the real system, belongs to A. The problem is to devise a control strategy that despite the initial ignorance of α⁰ can still achieve the minimum long-run average cost. The authors first introduce a class of adaptive controllers for Markov chains that meet the challenge, under fairly relaxed conditions. These algorithms are based on the concept of weak contrast functions, an extension of P.L. Mandi´s (1974) contrast functions. A class of adaptive controllers for semi-Markov processes is formulated based on the weak contrast function concept. The authors show that these controllers achieve the minimum long-run average cost and illustrate their application to the adaptive multilayer control of Markov chains