Adaptation of stochastic automata in nonstationary environments

This paper considers the effect of environmental nonstationarities on the performance of a stochastic automaton when it is used in the synthesis of an adaptive controller. The stochastic automaton considered has a variable structure in that its state probabilities are continuously altered according to a reinforcement scheme in response to penalties received from the environment. The automaton adapts by reducing the average penalty. Periodic perturbations of penalty strengths are used as "test signals" to derive analytic expressions describing the "tracking" behavior of the automaton operating under a linear reinforcement scheme. The pertinent parameters governing the adaptive behavior are discussed in detail. Digital simulation studies are presented for 2-state and 10-state cases. The results agree with the theoretical analysis.