Distant bigram language modelling using maximum entropy

Applies the maximum entropy approach to so-called distant bigram language modelling. In addition to the usual unigram and bigram dependencies, we use distant bigram dependencies, where the immediate predecessor word of the word position under consideration is skipped. We analyze the computational complexity of the resulting training algorithm, i.e. the generalized iterative scaling (GIS) algorithm, and study the details of its implementation. We describe a method for handling unseen events in the maximum entropy approach; this is achieved by discounting the frequencies of observed events. We study the effect of this discounting operation on the convergence of the GIS algorithm. We give experimental perplexity results for a corpus from the Wall Street Journal (WSJ) task. By using the maximum entropy approach and the distant bigram dependencies, we are able to reduce the perplexity from 205.4 for our best conventional bigram model to 169.5