Adaptive control of Markov chains, I: Finite parameter set

Consider a controlled Markov chain whose transition probabilities depend upon an unknown parameter α taking values in finite set . To each α is associated a prespecified stationary control law . The adaptive control law selects at each time the control action indicated by where αtis the maximum likelihood estimate of α. It is shown that αtconverges to a parameter α^*such that the "closed-loop" transition probabilities corresponding to α^*and are the same as those corresponding to α⁰and where α⁰is the true parameter. The situation when α⁰does not belong to the model set is briefly discussed.