Generalization and maximum likelihood from small data sets

A technique is described which can be used to prevent overtraining and encourage generalization in training under a maximum likelihood criterion. Applications to Boltzmann machines and hidden Markov models (HMMs) are discussed. While the confidence constraint may slow the training algorithm, in general it should involve very little additional calculation. The results presented for HMMs are for training under a maximum likelihood criterion based on the marginal distribution. Similar modifications can be made to the segmental K-means and N-best algorithms