Comparison of depth-based features for lipreading

We examine the effect of depth information captured by Microsoft Kinect on the task of visual speech recognition. We propose depth-based active appearance model (AAM) features and show improved results over discrete cosine transform (DCT). The visual and depth features are evaluated on a database of 54 speakers each uttering 50 isolated words. In order to exploit the speech dynamics, the features are enhanced by a simplified one-stage variant of hierarchical linear discriminant analysis (Hi-LDA). In the experiments, we consider feature fusion via combined video-depth active appearance model as a form of early integration, and compare it to traditional multi-stream hidden Markov Model as a form of decision fusion. We also perform experiments on audio-visual recognition in noisy environments and show improved results of incorporating depth information over both traditional audio-video fusion and utilization of speech enhancement algorithms.