Deep belief network based CRF for spoken language understanding

The key task in spoken language understanding research is the semantic tagging of sequences. Deep belief networks have recently shown great performance in word-labeling tasks while conditional random field has been a successful approach to model probabilities of sequences in a global fashion. In contrast to CRFs, DBNs are optimized based on a tag-by-tag likelihood in a locally normalized way and may suffer from the label bias problem. In this paper, we combine the DBN and CRF by employing the CRF model on top hidden layer of the DBN. This DBN-CRF architecture can explicitly model the dependencies of the output labels with transition features, and can be trained with a global sequence-level objective function. Experiments on ATIS corpus show that the new model outperforms CRFs and DBNs by 4.9% and 3.8% respectively. After effectively pre-training with additional unlabeled data, the results can be state-of-the-art, compared to the recent RNN-CRF model.