Syntactic heads in statistical language modeling

The use of syntactic structure in general and heads of syntactic constituents in particular has recently been shown to be beneficial for statistical language modeling. The paper provides an insightful analysis of this role of syntactic structure. It is shown that the predictive power of syntactic heads is mostly complementary to the predictive power of N-grams: they help in positions where an intervening phrase or clause separates the heads from the word being predicted, making the N-gram a poor predictor. Furthermore, a significant portion of this predictive power comes in the form of a more sophisticated back-off effect via the syntactic categories (nonterminal tags) of the heads. Finally, it is shown that using the categories of the syntactic heads is better than using the categories (part-of-speech tags) of the two preceding words, confirming that it is the syntactic analysis and not just the improved back-off strategy which leads to improvements over N-gram models. Experimental results for perplexity and word error rate are presented on the Switchboard corpus to support this analysis