Penalized maximum likelihood methods for finite memory estimators of infinite memory processes

Stationary ergodic processes with finite alphabets are estimated by finite memory processes from a sample, an n-length realization of the process. Both the transition probabilities and the memory depth of the estimator process are estimated from the sample using penalized maximum likelihood (PML). Under some assumptions on the continuity rate and the assumption of non-nullness, a rate of convergence in d̅-distance is obtained, with explicit constants. The results show an optimality of the PML Markov order estimator for not necessarily finite memory processes. Moreover, the notion of consistent Markov order estimation is generalized for infinite memory processes using the concept of oracle order estimates, and generalized consistency of the PML Markov order estimator is presented.