Estimation of non-stationary Markov Chain transition models

Many decision systems rely on a precisely known Markov Chain model to guarantee optimal performance, and this paper considers the online estimation of unknown, non-stationary Markov Chain transition models with perfect state observation. In using a prior Dirichlet distribution on the uncertain rows, we derive a mean-variance equivalent of the maximum a posteriori (MAP) estimator. This recursive mean-variance estimator extends previous methods that recompute the moments at each time step using observed transition counts. It is shown that this mean-variance estimator responds slowly to changes in transition models (especially switching models) and a modification that uses ideas of pseudonoise addition from classical filtering is used to speed up the response of the estimator. This new, discounted mean-variance estimator has the intuitive interpretation of fading previous observations and provides a link to fading techniques used in Hidden Markov Model estimation. Our new estimation techniques is both faster and has reduced error than alternative estimation techniques, such as finite memory estimators.