A multi-stream bimodal continuous speech recognition system using datasieve based features

This paper presents an audio visual bimodal continuous speech recognition system. The visual feature extraction of the mouth movements uses the number of granules obtained by applying a datasieve. Multi-stream HMMs are introduced for combining audio and visual modalities using time synchronous audio visual features. Experimental results show that the recognition system provided by this paper is suitable for continuous speech recognition tasks in noisy environments, and the datasieve based visual features outperform the conventional DCT and DWT features.