Continuous-time tracking algorithms involving two-time-scale Markov chains

This work is concerned with least-mean-squares (LMS) algorithms in continuous time for tracking a time-varying parameter process. A distinctive feature is that the true parameter process is changing at a fast pace driven by a finite-state Markov chain. The states of the Markov chain are divisible into a number of groups. Within each group, the transitions take place rapidly; among different groups, the transitions are infrequent. Introducing a small parameter into the generator of the Markov chain leads to a two-time-scale formulation. The tracking objective is difficult to achieve. Nevertheless, a limit result is derived yielding algorithms for limit systems. Moreover, the rates of variation of the tracking error sequence are analyzed. Under simple conditions, it is shown that a scaled sequence of the tracking errors converges weakly to a switching diffusion. In addition, a numerical example is provided and an adaptive step-size algorithm developed.