On expediency and convergence in variable structure automata

A variable structure stochastic automaton responds to the penalties from a random environment by changing its state probability distribution through a reinforcement scheme. This paper discusses the efficiency of learning for a 2-state automaton in terms of expediency and convergence under two types of nonlinear reinforcement schemes, one based on penalty probabilities and the other on penalty strengths. The stability of the asymptotic expected values of the state probability is discussed in detail. The conditions for achieving optimal, and expedient behavior of the automaton are derived. Convergence is discussed in the light of variance analysis. Learning curves can be obtained by solving nonlinear difference equations relating successive expected values, while for the linear case an analytic expression is derived. Finally transformation of penalty strengths to improve asymptotic state probability separation is considered.