Exploiting syntactic, semantic and lexical regularities in language modeling via directed Markov random fields

We present a directed Markov random field (MRF) model, that combines n-gram models, probabilistic context free grammars (PC FGs) and probabilistic latent semantic analysis (PLSA), for the purpose of statistical language modeling. The composite directed MRF model has a potentially exponential number of loops and becomes a context sensitive grammar, nevertheless we are able to estimate its parameters in cubic time using an efficient modified ME method, the generalized inside-outside algorithm, which extends the inside-outside algorithm to incorporate the effects of the n-gram and PLSA language models.