Unsupervised language model adaptation using latent Dirichlet allocation and dynamic marginals

In this paper, we introduce an unsupervised language model adaptation approach using latent Dirichlet allocation (LDA) and dynamic marginals: locally estimated (smoothed) unigram probabilities from in-domain text data. In LDA analysis, topic clusters are formed by using a hard-clustering method assigning one topic to one document based on the maximum number of words chosen from a topic for that document. The n-grams of the topic generated by hard-clustering are used to compute the mixture weights of the component topic models. Instead of using all the words of the training vocabulary, selected words are used for LDA analysis, which are chosen by incorporating some information retrieval techniques. We adapted the LDA adapted topic model by minimizing the Kullback-Leibler (KL) divergence between the final adapted model and the LDA adapted topic model subject to a constraint that the marginalized unigram probability distribution of the final adapted model is equal to the dynamic marginals. We have compared our approach with the conventional adapted model obtained by minimizing the KL divergence between the background model and the adapted model using the above constraint. We have seen that our approach gives significant perplexity and word error rate (WER) reductions over the traditional approach.